# Robots.txt Generator (`automation-lab/robots-txt-generator`) Actor Generate valid robots.txt files from structured rules. Apply presets (block AI bots, SEO-friendly), add custom per-bot rules, sitemaps, and crawl-delay. Zero-proxy, instant output. - **URL**: https://apify.com/automation-lab/robots-txt-generator.md - **Developed by:** [Stas Persiianenko](https://apify.com/automation-lab) (community) - **Categories:** SEO tools, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Robots.txt Generator Generate valid, production-ready **robots.txt files** from structured rules β€” no manual editing, no syntax errors. Define per-bot allow/disallow paths, crawl delays, sitemap URLs, and apply one-click presets like "Block AI Crawlers" or "SEO-Friendly". Zero-proxy, instant output. > **Zero proxy. Zero scraping. Pure computation.** This actor runs entirely on Apify compute β€” no bandwidth costs, no rate limits, no CAPTCHAs. --- ### What does it do? The Robots.txt Generator takes structured JSON rules and transforms them into a properly formatted `robots.txt` file that you can deploy directly to your website root. It: - πŸš€ Applies one-click **presets**: Block All, Allow All, Block AI Crawlers, SEO-Friendly - πŸ“‹ Accepts **custom per-bot rules** (user-agent, allow paths, disallow paths, crawl-delay) - πŸ—ΊοΈ Appends **sitemap URLs** and optional **Host directive** - βœ… **Validates rules** for conflicts, duplicates, invalid paths, and missing wildcards - πŸ’Ύ Saves output to **dataset** (JSON) and **key-value store** (`robots.txt`, `text/plain`) for direct download - ⚑ Runs in **under 5 seconds** β€” no network requests required --- ### Who is it for? #### πŸ§‘β€πŸ’» Web developers and DevOps engineers You manage multiple sites and need to generate consistent, validated robots.txt files without error-prone manual editing. Use this actor as part of your deployment pipeline β€” run it via API, download the output, and deploy automatically. #### πŸ“Š SEO professionals and consultants You're optimizing client sites and need to quickly generate robots.txt files that follow SEO best practices: allow search engine crawlers, block scrapers and AI training bots, and include sitemaps β€” all in one step. #### πŸ€– AI/LLM product owners You want to protect your content from AI training crawlers (GPTBot, CCBot, anthropic-ai, etc.) without blocking legitimate search engines. The "Block AI Crawlers" preset handles this automatically with an up-to-date list of known AI bots. #### 🏒 Enterprise teams managing many properties You're maintaining a set of standard robots.txt policies across a portfolio of websites and need a programmatic, repeatable way to generate them from structured configuration. --- ### Why use it? | Problem | Without this actor | With Robots.txt Generator | |---|---|---| | Syntax errors | Easy to make by hand | Automatically formatted | | Conflicting rules | Hard to spot manually | Validated with warnings | | AI bot list | Research bots yourself | 16 AI crawlers pre-loaded | | Sitemap inclusion | Remember to add manually | Just list URLs | | Consistency | Copy-paste across sites | Programmatic, repeatable | | Integration | Manual file upload | API-driven, downloadable | --- ### How much does it cost to generate a robots.txt file? This actor uses **Pay-Per-Event pricing** β€” you only pay for what you generate. | Event | FREE tier | BRONZE | SILVER | GOLD | PLATINUM | DIAMOND | |---|---|---|---|---|---|---| | Actor start | $0.005 | $0.00475 | $0.00425 | $0.00375 | $0.003 | $0.0025 | | Per robots.txt generated | $0.001 | $0.0009 | $0.0008 | $0.00065 | $0.0005 | $0.0004 | **Typical cost per run: $0.006** (start + 1 file on FREE tier). > πŸ’‘ **Free plan estimate:** Apify's free tier gives you $5/month in credit. That's enough for **~800 robots.txt files per month** at FREE tier pricing. This actor is one of the most affordable on the Store β€” zero proxy costs means there's no bandwidth markup. --- ### How to use it #### Step 1: Choose a preset (optional) Start with a preset that matches your use case: - **Allow all** β€” open your site to all crawlers - **Block all** β€” protect a staging or private site - **Block AI crawlers** β€” block GPTBot, CCBot, anthropic-ai, and 13 other AI training bots - **SEO-friendly** β€” allow search engines, block scrapers and AI bots, disallow admin areas #### Step 2: Add custom rules (optional) Append your own per-bot rules on top of the preset. Specify: ```json { "userAgent": "Googlebot", "allow": ["/"], "disallow": ["/admin/", "/private/"], "crawlDelay": 1, "comment": "Googlebot: full access except admin" } ```` #### Step 3: Add sitemaps and host (optional) ```json { "sitemaps": ["https://example.com/sitemap.xml"], "host": "example.com" } ``` #### Step 4: Run and download The actor saves your `robots.txt` to: - **Dataset** β€” JSON record with content, warnings, and metadata - **Key-value store** β€” raw `text/plain` file (key: `robots.txt`) for direct download *** ### Input parameters | Parameter | Type | Default | Description | |---|---|---|---| | `preset` | string | `seo-friendly` | Quick preset: `none`, `allow-all`, `block-all`, `block-ai-crawlers`, `seo-friendly` | | `rules` | array | `[]` | Custom user-agent rule blocks (user-agent, allow, disallow, crawl-delay, comment) | | `sitemaps` | array | `[]` | Sitemap URLs to append as `Sitemap:` directives | | `host` | string | `""` | Optional `Host:` directive (used by Yandex) | | `includeTimestamp` | boolean | `true` | Add generation timestamp comment | | `includeGeneratorComment` | boolean | `true` | Add Apify generator comment | | `validateRules` | boolean | `true` | Check for conflicts, duplicates, invalid paths | #### Rule block schema Each item in the `rules` array: ```json { "userAgent": "Googlebot", "allow": ["/public/"], "disallow": ["/admin/", "/login/"], "crawlDelay": 2, "comment": "Optional comment above this block" } ``` | Field | Type | Required | Description | |---|---|---|---| | `userAgent` | string or string\[] | Yes | Bot name(s), e.g. `"*"`, `"Googlebot"`, `["GPTBot", "CCBot"]` | | `allow` | string\[] | No | Paths the bot CAN access | | `disallow` | string\[] | No | Paths the bot CANNOT access | | `crawlDelay` | number | No | Seconds between requests (not supported by Googlebot) | | `comment` | string | No | Comment placed above the block | *** ### Output fields | Field | Type | Description | |---|---|---| | `robotsTxt` | string | The complete robots.txt file content | | `warnings` | string\[] | Validation warnings (conflicts, duplicates, etc.) | | `ruleBlockCount` | number | Total User-agent blocks in the file | | `sitemapCount` | number | Total Sitemap: directives | | `lineCount` | number | Total lines in the file | | `generatedAt` | string | ISO 8601 timestamp | | `preset` | string | Preset that was applied | | `success` | boolean | Whether generation succeeded | | `error` | string | Error message (if failed) | *** ### Example output #### Input ```json { "preset": "block-ai-crawlers", "sitemaps": ["https://mysite.com/sitemap.xml"], "includeTimestamp": true } ``` #### Output robots.txt ``` ## robots.txt generated by Apify Robots.txt Generator ## https://apify.com/automation-lab/robots-txt-generator ## Generated: 2026-04-09T10:00:00.000Z ## Preset: Block AI Crawlers User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: CCBot Disallow: / User-agent: anthropic-ai Disallow: / ## ... (13 more AI bots) ## Allow all other bots User-agent: * Allow: / Sitemap: https://mysite.com/sitemap.xml ``` *** ### Tips and best practices #### 🎯 Rule ordering matters Bots use the **first matching** User-agent block. If you have both a specific bot rule (`Googlebot`) and a wildcard rule (`*`), Googlebot will use its specific block and ignore the wildcard. Add specific bot blocks BEFORE the wildcard block. #### πŸ“Œ The wildcard block is your default Always include a `User-agent: *` block as your default rule. Without it, bots that aren't explicitly listed have no rules β€” this can cause unpredictable crawling behavior. #### ⏱️ Crawl-delay is not universal `Crawl-delay` is supported by Bingbot, Yandex, and many lesser-known bots, but **Googlebot ignores it**. To slow down Googlebot, use Google Search Console's crawl rate settings instead. #### 🚫 Don't block CSS and JS from Googlebot Blocking `/css/`, `/js/`, or `/assets/` prevents Googlebot from rendering your pages correctly, which can hurt rankings. Use the SEO-friendly preset which leaves these open. #### βœ… Validate before deploying The actor's built-in validator catches common mistakes: overlapping allow/disallow patterns, duplicate user-agents, paths not starting with `/`, and missing wildcard blocks. #### πŸ”„ Regenerate when your site structure changes Update and re-run whenever you add admin panels, API endpoints, or new content sections that should be crawled differently. *** ### Presets reference #### Allow All ``` User-agent: * Allow: / ``` Use for: Public sites, CDNs, open documentation. #### Block All ``` User-agent: * Disallow: / ``` Use for: Staging environments, private intranets, sites under construction. #### Block AI Crawlers Blocks all known AI training crawlers: GPTBot (OpenAI), CCBot (Common Crawl), anthropic-ai (Anthropic), Claude-Web, Google-Extended (Gemini training), cohere-ai, PerplexityBot, Bytespider (ByteDance), Diffbot, FacebookBot, ImagesiftBot, Omgilibot, YouBot, Applebot-Extended, ChatGPT-User, PetalBot. All other bots (including Googlebot, Bingbot) are allowed. Use for: Publishers protecting training data rights, paywalled content sites. #### SEO-Friendly - Allows all search engine bots with `Allow: /` - Blocks 10 bad/spam bots (AhrefsBot, SemrushBot, MJ12bot, etc.) - Blocks 16 AI crawlers - Disallows common admin paths (`/admin/`, `/wp-admin/`, `/login/`, `/cart/`, `/checkout/`) - Applies `Crawl-delay: 1` to all bots Use for: Production websites, e-commerce sites, blogs, SaaS marketing pages. *** ### Integrations #### πŸ”— Zapier / Make automation Trigger this actor on a schedule and automatically deploy the output to your web server: 1. Set up a scheduled Apify run (or cron trigger) 2. In Zapier/Make, trigger when the run completes 3. Fetch the `robots.txt` from the key-value store URL 4. Upload to your server via FTP, S3, or API #### ☁️ AWS S3 deployment pipeline Use the Apify API to run this actor as part of your CI/CD pipeline, then upload the output directly to S3: ```bash RUN_OUTPUT=$(apify call automation-lab/robots-txt-generator --input-file=robots-config.json --output=dataset) aws s3 cp robots.txt s3://mybucket/robots.txt --content-type text/plain ``` #### 🌐 Cloudflare Workers / CDN edge Generate your `robots.txt` via this actor and serve it directly from a KV store URL in Cloudflare Workers, enabling instant updates without redeployment. #### πŸ“Š Spreadsheet-driven multi-site management Store your robots.txt configuration in Google Sheets (one row per site). Use Make.com to read each row and trigger this actor, then deploy each generated file to the corresponding site. *** ### API usage #### Node.js ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_API_TOKEN' }); const run = await client.actor('automation-lab/robots-txt-generator').call({ preset: 'seo-friendly', rules: [ { userAgent: 'Googlebot', allow: ['/'], disallow: ['/admin/'], }, ], sitemaps: ['https://example.com/sitemap.xml'], }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items[0].robotsTxt); ``` #### Python ```python from apify_client import ApifyClient client = ApifyClient("YOUR_API_TOKEN") run = client.actor("automation-lab/robots-txt-generator").call(run_input={ "preset": "block-ai-crawlers", "sitemaps": ["https://example.com/sitemap.xml"], "validateRules": True, }) items = client.dataset(run["defaultDatasetId"]).list_items().items print(items[0]["robotsTxt"]) ``` #### cURL ```bash ## Start the actor curl -X POST "https://api.apify.com/v2/acts/automation-lab~robots-txt-generator/runs?token=YOUR_API_TOKEN" \ -H "Content-Type: application/json" \ -d '{ "preset": "seo-friendly", "sitemaps": ["https://example.com/sitemap.xml"] }' ## Get results (replace DATASET_ID with run output) curl "https://api.apify.com/v2/datasets/DATASET_ID/items?token=YOUR_API_TOKEN" ``` *** ### MCP β€” use with Claude, Cursor, and VS Code You can use this actor as an **MCP tool** in Claude Desktop, Claude Code, Cursor, and VS Code to generate robots.txt files directly from AI chat. #### Claude Code (terminal) ```bash claude mcp add --transport http apify "https://mcp.apify.com?tools=automation-lab/robots-txt-generator" ``` #### Claude Desktop / Cursor / VS Code Add to your MCP config file: ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com?tools=automation-lab/robots-txt-generator", "headers": { "Authorization": "Bearer YOUR_APIFY_TOKEN" } } } } ``` #### Example prompts > "Generate a robots.txt that blocks GPTBot and CCBot but allows Googlebot and Bingbot. Add my sitemap at https://example.com/sitemap.xml." > "Create a robots.txt using the SEO-friendly preset for a WordPress site, but also disallow /wp-content/uploads/." > "Generate a robots.txt that blocks all bots from /staging/ and /draft/ paths, but allows Googlebot everywhere else." *** ### Legal and compliance Robots.txt is a voluntary protocol (the [Robots Exclusion Standard](https://en.wikipedia.org/wiki/Robots_exclusion_standard)). Compliant bots like Googlebot and Bingbot honor these directives. However, some scrapers and AI crawlers may ignore your `robots.txt`. For legally binding protection against crawling, you must also: - Add Terms of Service that explicitly prohibit scraping - Implement technical access controls (rate limiting, authentication) - Consider emerging AI scraping liability developments in your jurisdiction This actor generates `robots.txt` files that comply with the RFC 9309 specification. The SEO-friendly and block-AI-crawlers presets include directives targeting bots known to the developer as of April 2026 β€” the list may not be exhaustive. *** ### FAQ **Q: What's the difference between `Allow` and `Disallow`?** `Disallow` tells bots not to crawl a path. `Allow` explicitly permits access to a path that would otherwise be blocked by a broader `Disallow` rule. When paths conflict, the more specific (longer) rule wins in Googlebot β€” other bots may vary. **Q: Does Googlebot respect `Crawl-delay`?** No. Googlebot ignores the `Crawl-delay` directive. To control Googlebot's crawl rate, use [Google Search Console's crawl rate settings](https://search.google.com/search-console). Other bots (Bingbot, Yandex) do respect `Crawl-delay`. **Q: How do I block a bot from only some pages?** Create a rule block for that specific bot with `Disallow:` paths, BEFORE the `User-agent: *` block. Bots use the first matching rule block. **Q: Why am I getting "User-agent appears in 2 rule blocks" warnings?** When using a preset AND custom rules that include the same user-agent (e.g., you add a `*` rule on top of the SEO-friendly preset which already has one), the bot will use the FIRST matching block and ignore the second. Either merge your rules into one block or remove the duplicate. **Q: Will my custom rules override the preset?** Custom rules are APPENDED after preset blocks. Since bots use the first matching block, if your custom block has the same user-agent as a preset block, the preset block wins. To override a preset rule, set `preset: "none"` and write all rules yourself. **Q: The robots.txt is saved in the key-value store β€” how do I download it?** Go to your Apify run, open the Key-Value Store, find the key `robots.txt`, and click "Download" or use the direct URL. The content-type is `text/plain` so browsers display it correctly. **Q: Can I run this actor on a schedule to keep my robots.txt up to date?** Yes β€” use Apify Schedules (free with any plan) to run this actor automatically. Combine with a Zapier/Make integration to auto-deploy the output to your server. *** ### Related actors - [Broken Link Checker](https://apify.com/automation-lab/broken-link-checker) β€” Find broken links on your site after updating robots.txt - [Canonical URL Checker](https://apify.com/automation-lab/canonical-url-checker) β€” Verify canonical tags across your site - [Duplicate Content Checker](https://apify.com/automation-lab/duplicate-content-checker) β€” Detect duplicate pages before tuning crawl rules - [DNS Propagation Checker](https://apify.com/automation-lab/dns-propagation-checker) β€” Verify DNS settings alongside your SEO config - [Ads.txt Checker](https://apify.com/automation-lab/ads-txt-checker) β€” Validate your ads.txt file for publisher compliance # Actor input Schema ## `preset` (type: `string`): Apply a common preset to quickly generate a standard robots.txt. Choose 'custom' to define your own rules. Presets can be combined with additional rules below. ## `rules` (type: `array`): List of user-agent rule blocks. Each block defines which bot(s) it applies to, and what paths they can or cannot access. Leave empty to use only a preset. ## `sitemaps` (type: `array`): Full URLs to your XML sitemaps. Each will be added as a Sitemap: directive at the end of the file. ## `host` (type: `string`): Optional Host: directive specifying the canonical domain (used by some Russian search engines like Yandex). Example: 'example.com'. ## `includeTimestamp` (type: `boolean`): Add a comment at the top of the file with the generation date/time. ## `includeGeneratorComment` (type: `boolean`): Add a comment identifying this file was generated by Apify Robots.txt Generator. ## `validateRules` (type: `boolean`): Check for common mistakes: conflicting allow/disallow patterns, missing wildcards, duplicate user-agents, and more. ## Actor input object example ```json { "preset": "seo-friendly", "rules": [ { "userAgent": "*", "allow": [ "/public/", "/blog/" ], "disallow": [ "/admin/", "/private/", "/api/" ], "crawlDelay": 2, "comment": "Default rules for all bots" }, { "userAgent": "Googlebot", "allow": [ "/" ], "disallow": [], "comment": "Give Googlebot full access" } ], "sitemaps": [ "https://example.com/sitemap.xml" ], "includeTimestamp": true, "includeGeneratorComment": true, "validateRules": true } ``` # Actor output Schema ## `overview` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "preset": "seo-friendly", "rules": [ { "userAgent": "*", "allow": [ "/public/", "/blog/" ], "disallow": [ "/admin/", "/private/", "/api/" ], "crawlDelay": 2, "comment": "Default rules for all bots" }, { "userAgent": "Googlebot", "allow": [ "/" ], "disallow": [], "comment": "Give Googlebot full access" } ], "sitemaps": [ "https://example.com/sitemap.xml" ], "host": "", "includeTimestamp": true, "includeGeneratorComment": true, "validateRules": true }; // Run the Actor and wait for it to finish const run = await client.actor("automation-lab/robots-txt-generator").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "preset": "seo-friendly", "rules": [ { "userAgent": "*", "allow": [ "/public/", "/blog/", ], "disallow": [ "/admin/", "/private/", "/api/", ], "crawlDelay": 2, "comment": "Default rules for all bots", }, { "userAgent": "Googlebot", "allow": ["/"], "disallow": [], "comment": "Give Googlebot full access", }, ], "sitemaps": ["https://example.com/sitemap.xml"], "host": "", "includeTimestamp": True, "includeGeneratorComment": True, "validateRules": True, } # Run the Actor and wait for it to finish run = client.actor("automation-lab/robots-txt-generator").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "preset": "seo-friendly", "rules": [ { "userAgent": "*", "allow": [ "/public/", "/blog/" ], "disallow": [ "/admin/", "/private/", "/api/" ], "crawlDelay": 2, "comment": "Default rules for all bots" }, { "userAgent": "Googlebot", "allow": [ "/" ], "disallow": [], "comment": "Give Googlebot full access" } ], "sitemaps": [ "https://example.com/sitemap.xml" ], "host": "", "includeTimestamp": true, "includeGeneratorComment": true, "validateRules": true }' | apify call automation-lab/robots-txt-generator --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=automation-lab/robots-txt-generator", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Robots.txt Generator", "description": "Generate valid robots.txt files from structured rules. Apply presets (block AI bots, SEO-friendly), add custom per-bot rules, sitemaps, and crawl-delay. Zero-proxy, instant output.", "version": "0.1", "x-build-id": "SwGfYqiWbgCVud9n5" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/automation-lab~robots-txt-generator/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-automation-lab-robots-txt-generator", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/automation-lab~robots-txt-generator/runs": { "post": { "operationId": "runs-sync-automation-lab-robots-txt-generator", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/automation-lab~robots-txt-generator/run-sync": { "post": { "operationId": "run-sync-automation-lab-robots-txt-generator", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "preset": { "title": "⚑ Quick preset", "enum": [ "none", "allow-all", "block-all", "block-ai-crawlers", "seo-friendly" ], "type": "string", "description": "Apply a common preset to quickly generate a standard robots.txt. Choose 'custom' to define your own rules. Presets can be combined with additional rules below.", "default": "seo-friendly" }, "rules": { "title": "πŸ“‹ User-agent rules", "type": "array", "description": "List of user-agent rule blocks. Each block defines which bot(s) it applies to, and what paths they can or cannot access. Leave empty to use only a preset." }, "sitemaps": { "title": "πŸ—ΊοΈ Sitemap URLs", "type": "array", "description": "Full URLs to your XML sitemaps. Each will be added as a Sitemap: directive at the end of the file." }, "host": { "title": "🌐 Host directive", "type": "string", "description": "Optional Host: directive specifying the canonical domain (used by some Russian search engines like Yandex). Example: 'example.com'." }, "includeTimestamp": { "title": "πŸ• Include generation timestamp", "type": "boolean", "description": "Add a comment at the top of the file with the generation date/time.", "default": true }, "includeGeneratorComment": { "title": "πŸ’¬ Include generator comment", "type": "boolean", "description": "Add a comment identifying this file was generated by Apify Robots.txt Generator.", "default": true }, "validateRules": { "title": "βœ… Validate rules", "type": "boolean", "description": "Check for common mistakes: conflicting allow/disallow patterns, missing wildcards, duplicate user-agents, and more.", "default": true } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```