# Robots Sitemap Analyzer - SEO Crawl Rules (`benthepythondev/robots-sitemap-analyzer`) Actor Analyze robots.txt files and discover sitemap URLs, user-agent groups, allow rules, disallow rules and crawl-delay directives. - **URL**: https://apify.com/benthepythondev/robots-sitemap-analyzer.md - **Developed by:** [ben](https://apify.com/benthepythondev) (community) - **Categories:** SEO tools, Automation, Business - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Robots Sitemap Analyzer - SEO Crawl Rules Analyze `robots.txt` files at scale and extract the crawl rules and sitemap URLs that search engines, SEO tools and crawlers use before visiting a website. Paste domains or URLs and the actor returns a structured row for each target with status code, final robots URL, sitemap links, user-agent groups, allow rules, disallow rules and crawl-delay directives. ### What is the Robots Sitemap Analyzer? The `robots.txt` file is a small text file with a big impact. It tells crawlers which parts of a website are allowed, which paths are blocked, and where XML sitemaps can be found. For SEO audits, migration checks, technical monitoring and crawler setup, this file is often the first place to inspect. This actor turns robots files into structured data. Instead of opening files manually or writing one-off scripts, you can run a clean Apify job across many domains and export the results to JSON, CSV, Excel or your own API workflow. It is fast, browserless and inexpensive to run. ### Common use cases Use this actor to audit client websites, monitor whether a production deploy accidentally blocked important paths, discover sitemap URLs for further crawling, compare crawl rules across competitors, check crawl-delay directives before running a crawler, or build a list of sitemap indexes for SEO discovery. Agencies can run it before technical SEO audits. Data teams can use it as the first step in a web discovery pipeline. Developers can add it to release checks to catch broken robots files before search visibility is affected. ### Input Provide domains, hostnames or full URLs. The actor normalizes each target and fetches `/robots.txt`. ```json { "domains": [ "apify.com", "https://example.com/blog" ] } ```` ### Output Each dataset item represents one analyzed website. ```json { "input": "apify.com", "domain": "apify.com", "robots_url": "https://apify.com/robots.txt", "final_url": "https://apify.com/robots.txt", "status_code": 200, "exists": true, "sitemap_urls": ["https://apify.com/sitemap.xml"], "sitemap_count": 1, "user_agent_groups": [ { "user_agent": "*", "allow": [], "disallow": ["/private"], "crawl_delay": null, "other": [] } ], "user_agent_count": 1, "disallow_count": 1, "allow_count": 0 } ``` ### Output fields - `input` - the original domain or URL. - `domain` - normalized domain from the fetched robots URL. - `robots_url` - the robots URL the actor attempted. - `final_url` - final URL after redirects. - `status_code` - HTTP status code. - `exists` - true when a non-empty robots file is found. - `sitemap_urls` - sitemap declarations found in the file. - `sitemap_count` - number of unique sitemap URLs. - `user_agent_groups` - parsed crawl rules grouped by user agent. - `disallow_count` and `allow_count` - quick summary counts. ### Why use this actor? Manual robots checks do not scale. This actor gives you repeatable structured results, easy scheduling, exports, API access and webhooks. You can run a weekly audit across your own websites, trigger alerts when sitemap declarations disappear, or feed discovered sitemap URLs into a sitemap extraction actor. ### Example workflows For an SEO audit, run the actor across every client domain, export the results, and quickly spot missing robots files, blocked sections, crawl-delay rules and sitemap declarations. For migration monitoring, run it before and after a launch to make sure the staging block did not accidentally ship to production. For crawler setup, use the discovered sitemap URLs as inputs for a sitemap extraction run. For agencies, this actor is a lightweight first-pass audit tool. It helps answer practical questions quickly: does the site expose a sitemap, are important folders blocked, are there different rules for Googlebot, and did the robots file redirect correctly from HTTP to HTTPS? Those answers are useful before you spend time on a deeper crawl. ### Integration ideas Schedule the actor weekly and compare `sitemap_urls`, `disallow_count` and `status_code` against the previous run. Send changes to Slack, email, Make, Zapier or n8n. Store the output in a database if you manage many websites. Use Apify webhooks to trigger a sitemap scraping workflow whenever a new sitemap appears. The output is intentionally compact: one row per target domain. That makes it easy to join with CRM records, client lists, domain inventories or uptime monitoring tables. ### Notes and limits The actor parses common robots directives and preserves less common directives in the `other` array. It does not attempt to decide whether every URL on a site is crawlable. Robots semantics can be crawler-specific, especially for conflicting `allow` and `disallow` paths. Use the parsed data as a reliable audit input, then apply your crawler's exact matching rules if needed. ### Best practices Run the actor against canonical production domains, not random deep URLs. Review both the `status_code` and `exists` fields; a 404 robots file is not always an error, but it means no explicit sitemap declaration was available there. If you find sitemap URLs, pass them into a sitemap extraction actor to continue discovery. For high-value domains, monitor robots changes on a schedule because a single mistaken `Disallow: /` can remove important pages from search results. ### FAQ Does it crawl the website? No. It only fetches and parses `/robots.txt`. This keeps it fast and safe. Does it extract sitemap URLs? Yes. Every `Sitemap:` directive is returned as a structured array. Can I combine it with a sitemap scraper? Yes. Use this actor to discover sitemap URLs, then pass them to a sitemap URL extractor. Can it check many domains? Yes. Add multiple domains in the input and export the combined dataset. ### You might also like - Sitemap URL Extractor - collect URLs from XML sitemaps. - Website Contact Extractor - extract contacts from websites. - RSS Feed Scraper - monitor feeds and content updates. ### Keywords robots.txt analyzer, robots scraper, sitemap finder, sitemap extractor, SEO crawl rules, technical SEO audit, crawl audit, robots API, sitemap API, Apify SEO tools. # Actor input Schema ## `domains` (type: `array`): Domains or website URLs to analyze. The actor fetches /robots.txt for each target. ## Actor input object example ```json { "domains": [ "apify.com" ] } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "apify.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("benthepythondev/robots-sitemap-analyzer").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": ["apify.com"] } # Run the Actor and wait for it to finish run = client.actor("benthepythondev/robots-sitemap-analyzer").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "apify.com" ] }' | apify call benthepythondev/robots-sitemap-analyzer --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=benthepythondev/robots-sitemap-analyzer", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Robots Sitemap Analyzer - SEO Crawl Rules", "description": "Analyze robots.txt files and discover sitemap URLs, user-agent groups, allow rules, disallow rules and crawl-delay directives.", "version": "1.0", "x-build-id": "UgmtGf90VlVicwAMU" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/benthepythondev~robots-sitemap-analyzer/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-benthepythondev-robots-sitemap-analyzer", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/benthepythondev~robots-sitemap-analyzer/runs": { "post": { "operationId": "runs-sync-benthepythondev-robots-sitemap-analyzer", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/benthepythondev~robots-sitemap-analyzer/run-sync": { "post": { "operationId": "run-sync-benthepythondev-robots-sitemap-analyzer", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "domains": { "title": "Domains or URLs", "type": "array", "description": "Domains or website URLs to analyze. The actor fetches /robots.txt for each target.", "default": [ "apify.com" ], "items": { "type": "string" } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```