# Robots.txt Generator (`maximedupre/robots-txt-generator`) Actor Generate deployable robots.txt files from presets, custom bot rules, sitemap URLs, and host directives. Create one file or batch files for multiple sites, then export raw text plus validation data. - **URL**: https://apify.com/maximedupre/robots-txt-generator.md - **Developed by:** [Maxime DuprΓ©](https://apify.com/maximedupre) (community) - **Categories:** Developer tools, Marketing - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.55 / 1,000 robots.txt files This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### πŸ€– Robots.txt generator for SEO and crawler control Robots.txt Generator creates deployable `robots.txt` files from presets, custom bot rules, sitemap URLs, and host directives. Use this robots txt generator when you need a clean file for SEO work, AI crawler blocking, staging privacy, WordPress, e-commerce, or repeatable multi-site setup. The Actor follows the public [`robots.txt` protocol](https://www.rfc-editor.org/rfc/rfc9309.html) format and returns both the raw text file and structured validation data. You can generate one file, batch multiple sites in a single run, save the raw files from the key-value store, and export the dataset as JSON, CSV, Excel, XML, RSS, or HTML. ### βš™οΈ What this Actor does - Generates ready-to-deploy `robots.txt` text. - Starts from practical presets such as allow all, block all, SEO-friendly, WordPress SEO, e-commerce SEO, staging private, block AI crawlers, and AI training opt-out. - Adds custom `User-agent`, `Allow`, `Disallow`, `Crawl-delay`, `Sitemap`, and `Host` lines. - Supports multiple generated files in one run through the `sites` batch input. - Saves each raw `robots.txt` file to the key-value store. - Emits one dataset row per generated file with counts, warnings, validation details, file metadata, and a SHA-256 hash. - Lets you append custom rules to a preset or override the preset rules entirely. This Actor is a generator, not a crawler. It does not visit your website, deploy the file, audit live crawler behavior, or guarantee that every bot will obey the file. ### πŸ“₯ Input Use the top-level fields for a single file, or use `sites` when you want separate files for multiple websites. ```json { "preset": "block-ai-crawlers", "sitemaps": ["https://example.com/sitemap.xml"], "host": "example.com", "rules": [ { "userAgents": ["Googlebot"], "allow": ["/"], "disallow": ["/private/"], "comment": "Keep search access while hiding private paths" } ], "presetConflictMode": "append", "validateRules": true, "includeTimestamp": true, "includeGeneratorComment": true } ```` For batch generation, add items to `sites`. Each site can define its own label, preset, sitemap URLs, host directive, file name, and rule blocks. ### πŸ“€ Output Each dataset item represents one generated `robots.txt` file. Rows can include: - `robotsTxt` - the generated file contents. - `preset` - the preset used for that file. - `siteLabel` and `inputIndex` - batch context when provided. - `fileName`, `contentType`, `byteSize`, `sha256`, `kvStoreKey`, and `kvStoreUrl` - file export metadata. - `host` and `sitemaps` - directives included in the file. - `ruleBlockCount`, `sitemapCount`, and `lineCount` - quick file-size and structure counts. - `warnings`, `validation`, and `validationSummary` - rule issues and suggestions when validation is enabled. - `ruleSummary` - per-rule counts for user agents, allowed paths, blocked paths, and crawl delay. - `botListVersion` and `generatedAt` - generation metadata. ```json { "success": true, "preset": "block-ai-crawlers", "fileName": "robots.txt", "kvStoreKey": "robots-txt/robots.txt", "host": "example.com", "sitemaps": ["https://example.com/sitemap.xml"], "ruleBlockCount": 2, "sitemapCount": 1, "lineCount": 18, "warnings": [], "validationSummary": { "errors": 0, "warnings": 0 }, "robotsTxt": "# Generated robots.txt..." } ``` Open `kvStoreUrl` or fetch `kvStoreKey` from the run's key-value store when you want the raw `.txt` file. ### πŸ’³ Pricing This Actor uses pay-per-event pricing. You are charged once for each generated `robots.txt` file that is saved as an output row. There is no separate Actor-start charge in the local pricing artifact. Use one generated file for a quick first run, then switch to the `sites` batch input when you are ready to generate files for multiple domains. ### ⚠️ Limits and caveats - `robots.txt` is a voluntary crawler directive file. Well-behaved crawlers respect it, but it is not an access-control system. - The Actor does not upload the file to your website. You still need to deploy it at the root path, such as `https://example.com/robots.txt`. - The Actor does not crawl your website, verify live sitemap URLs, or check whether a deployed file is currently reachable. - AI crawler presets are based on the maintained bot list included with this Actor at generation time. - Validation catches common formatting and rule issues, but final SEO policy choices remain yours. ### ❓ FAQ #### 🧭 Can I use this as a robots.txt API? Yes. Run the Actor through the Apify API, pass your preset and rule fields, then read the dataset row or key-value-store file from the finished run. #### 🧱 Can it block AI crawlers? It can generate rule blocks for known AI crawlers and AI training bots. The output is still a `robots.txt` file, so enforcement depends on each crawler respecting the protocol. #### πŸ—ΊοΈ Can I include sitemap URLs? Yes. Add one or more sitemap URLs and the Actor writes `Sitemap:` lines into the generated file. #### 🏒 Can I generate files for multiple sites? Yes. Use the `sites` array to create one output file per site with separate labels, presets, sitemap URLs, host directives, custom rules, and file names. ### πŸ“ Changelog - 0.1: Initial release. ### πŸ†˜ Support For issues, questions, or feature requests, [file a ticket](https://console.apify.com/actors/maximedupre~robots-txt-generator/issues) and I'll fix or implement it in less than 24h 🫑 ### πŸ”— Other actors - [Website URL Crawler β†—](https://apify.com/maximedupre/website-url-crawler) - Build rendered website link maps for SEO audits and migrations. - [Font Detector β†—](https://apify.com/maximedupre/font-detector) - Audit fonts, font files, and typography metadata from public pages. - [SEMrush Free Website Stats Scraper β†—](https://apify.com/maximedupre/semrush-free-website-stats-scraper) - Collect public SEMrush website metrics for domains and URLs. - [Ahrefs Free Website Stats Scraper β†—](https://apify.com/maximedupre/ahrefs-free-website-stats-scraper) - Collect public Ahrefs website stats for SEO research. - [Email MX Verifier β†—](https://apify.com/maximedupre/email-mx-verifier) - Check email syntax, MX records, disposable domains, and delivery risk. **Made with ❀️ by Maxime DuprΓ©** # Actor input Schema ## `preset` (type: `string`): Start with a common robots.txt policy, then add custom rules below. ## `sitemaps` (type: `array`): Add full sitemap URLs to include as Sitemap directives. ## `host` (type: `string`): Add the canonical host when you want a Host directive in the file. ## `rules` (type: `array`): Add per-bot Allow, Disallow, Crawl-delay, and comment rules. ## `presetConflictMode` (type: `string`): Choose how custom rules behave when they use the same bot name as a preset rule. ## `sites` (type: `array`): Generate several robots.txt files in one run. Each site can set its own label, preset, rules, sitemaps, and host. ## `validateRules` (type: `boolean`): Return warnings for duplicate bots, invalid paths, conflicting directives, and missing wildcard rules. ## `includeTimestamp` (type: `boolean`): Add a generated-at comment to the robots.txt content. ## `includeGeneratorComment` (type: `boolean`): Add a short comment that identifies the file as generated by this Actor. ## Actor input object example ```json { "preset": "seo-friendly", "sitemaps": [ "https://example.com/sitemap.xml" ], "host": "example.com", "rules": [ { "userAgents": [ "*" ], "allow": [ "/" ], "disallow": [ "/admin/", "/private/" ], "crawlDelay": 1, "comment": "Default rules for all bots" }, { "userAgents": [ "Googlebot" ], "allow": [ "/" ], "disallow": [], "comment": "Give Googlebot full access" } ], "presetConflictMode": "append", "sites": [ { "label": "Marketing site", "preset": "seo-friendly", "sitemaps": [ "https://example.com/sitemap.xml" ], "host": "example.com" }, { "label": "Staging site", "preset": "staging-private", "sitemaps": [], "host": "staging.example.com" } ], "validateRules": true, "includeTimestamp": true, "includeGeneratorComment": true } ``` # Actor output Schema ## `results` (type: `string`): Open one dataset row per generated robots.txt file. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "preset": "seo-friendly", "sitemaps": [ "https://example.com/sitemap.xml" ], "host": "example.com", "rules": [ { "userAgents": [ "*" ], "allow": [ "/" ], "disallow": [ "/admin/", "/private/" ], "crawlDelay": 1, "comment": "Default rules for all bots" }, { "userAgents": [ "Googlebot" ], "allow": [ "/" ], "disallow": [], "comment": "Give Googlebot full access" } ], "presetConflictMode": "append", "sites": [ { "label": "Marketing site", "preset": "seo-friendly", "sitemaps": [ "https://example.com/sitemap.xml" ], "host": "example.com" }, { "label": "Staging site", "preset": "staging-private", "sitemaps": [], "host": "staging.example.com" } ], "validateRules": true, "includeTimestamp": true, "includeGeneratorComment": true }; // Run the Actor and wait for it to finish const run = await client.actor("maximedupre/robots-txt-generator").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "preset": "seo-friendly", "sitemaps": ["https://example.com/sitemap.xml"], "host": "example.com", "rules": [ { "userAgents": ["*"], "allow": ["/"], "disallow": [ "/admin/", "/private/", ], "crawlDelay": 1, "comment": "Default rules for all bots", }, { "userAgents": ["Googlebot"], "allow": ["/"], "disallow": [], "comment": "Give Googlebot full access", }, ], "presetConflictMode": "append", "sites": [ { "label": "Marketing site", "preset": "seo-friendly", "sitemaps": ["https://example.com/sitemap.xml"], "host": "example.com", }, { "label": "Staging site", "preset": "staging-private", "sitemaps": [], "host": "staging.example.com", }, ], "validateRules": True, "includeTimestamp": True, "includeGeneratorComment": True, } # Run the Actor and wait for it to finish run = client.actor("maximedupre/robots-txt-generator").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "preset": "seo-friendly", "sitemaps": [ "https://example.com/sitemap.xml" ], "host": "example.com", "rules": [ { "userAgents": [ "*" ], "allow": [ "/" ], "disallow": [ "/admin/", "/private/" ], "crawlDelay": 1, "comment": "Default rules for all bots" }, { "userAgents": [ "Googlebot" ], "allow": [ "/" ], "disallow": [], "comment": "Give Googlebot full access" } ], "presetConflictMode": "append", "sites": [ { "label": "Marketing site", "preset": "seo-friendly", "sitemaps": [ "https://example.com/sitemap.xml" ], "host": "example.com" }, { "label": "Staging site", "preset": "staging-private", "sitemaps": [], "host": "staging.example.com" } ], "validateRules": true, "includeTimestamp": true, "includeGeneratorComment": true }' | apify call maximedupre/robots-txt-generator --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=maximedupre/robots-txt-generator", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Robots.txt Generator", "description": "Generate deployable robots.txt files from presets, custom bot rules, sitemap URLs, and host directives. Create one file or batch files for multiple sites, then export raw text plus validation data.", "version": "0.1", "x-build-id": "3GwztC4I9IzwwwSxI" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/maximedupre~robots-txt-generator/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-maximedupre-robots-txt-generator", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/maximedupre~robots-txt-generator/runs": { "post": { "operationId": "runs-sync-maximedupre-robots-txt-generator", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/maximedupre~robots-txt-generator/run-sync": { "post": { "operationId": "run-sync-maximedupre-robots-txt-generator", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "preset": { "title": "Preset", "enum": [ "none", "allow-all", "block-all", "block-ai-crawlers", "seo-friendly", "wordpress-seo", "ecommerce-seo", "staging-private", "ai-training-opt-out" ], "type": "string", "description": "Start with a common robots.txt policy, then add custom rules below.", "default": "seo-friendly" }, "sitemaps": { "title": "Sitemap URLs", "uniqueItems": true, "type": "array", "description": "Add full sitemap URLs to include as Sitemap directives.", "items": { "type": "string", "minLength": 1 } }, "host": { "title": "Host directive", "type": "string", "description": "Add the canonical host when you want a Host directive in the file." }, "rules": { "title": "Custom bot rules", "type": "array", "description": "Add per-bot Allow, Disallow, Crawl-delay, and comment rules.", "items": { "type": "object", "required": [ "userAgents" ], "properties": { "userAgents": { "title": "Bot names", "description": "Crawler user-agent names for this rule block.", "type": "array", "items": { "type": "string", "minLength": 1 }, "minItems": 1 }, "allow": { "title": "Allow paths", "description": "Paths these bots can crawl.", "type": "array", "items": { "type": "string" } }, "disallow": { "title": "Disallow paths", "description": "Paths these bots should skip.", "type": "array", "items": { "type": "string" } }, "crawlDelay": { "title": "Crawl delay", "description": "Seconds these bots should wait between requests.", "type": "number", "minimum": 0 }, "comment": { "title": "Rule comment", "description": "Comment written above this rule block.", "type": "string" } } } }, "presetConflictMode": { "title": "Preset conflict handling", "enum": [ "append", "override" ], "type": "string", "description": "Choose how custom rules behave when they use the same bot name as a preset rule.", "default": "append" }, "sites": { "title": "Batch site configs", "type": "array", "description": "Generate several robots.txt files in one run. Each site can set its own label, preset, rules, sitemaps, and host.", "items": { "type": "object", "properties": { "label": { "title": "Site label", "description": "Short name shown with this generated file.", "type": "string" }, "preset": { "title": "Site preset", "description": "Preset for this site.", "type": "string" }, "sitemaps": { "title": "Site sitemap URLs", "description": "Sitemap URLs for this site.", "type": "array", "items": { "type": "string" } }, "host": { "title": "Site host", "description": "Host directive for this site.", "type": "string" }, "rules": { "title": "Site custom rules", "description": "Custom bot rules for this site.", "type": "array", "items": { "type": "object" } }, "fileName": { "title": "File name", "description": "Output file name for this site.", "type": "string" } } } }, "validateRules": { "title": "Validate rules", "type": "boolean", "description": "Return warnings for duplicate bots, invalid paths, conflicting directives, and missing wildcard rules.", "default": true }, "includeTimestamp": { "title": "Include generated timestamp", "type": "boolean", "description": "Add a generated-at comment to the robots.txt content.", "default": true }, "includeGeneratorComment": { "title": "Include generator comment", "type": "boolean", "description": "Add a short comment that identifies the file as generated by this Actor.", "default": true } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```