# Robots.txt & Sitemap Analyzer πŸ•·οΈ (`perryay/robots-txt-sitemap-analyzer`) Actor Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output. - **URL**: https://apify.com/perryay/robots-txt-sitemap-analyzer.md - **Developed by:** [Perry AY](https://apify.com/perryay) (community) - **Categories:** Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.005 / actor start This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Robots.txt & Sitemap Analyzer πŸ€– **Fetch, parse, and analyze robots.txt and sitemap.xml for any domain β€” rules, URL testing, sitemap extraction, and batch analysis.** A misconfigured `robots.txt` can block search engines from indexing your content, expose private directories to bots, or waste crawl budget on irrelevant pages. An outdated or missing sitemap means search engines may never discover your important pages. Manually checking these files across multiple domains is tedious, error-prone, and doesn't scale. **Robots.txt & Sitemap Analyzer** automates the entire workflow: fetches `robots.txt` and `sitemap.xml` from one or more domains, parses every directive with correct precedence rules, tests specific URLs against the extracted rules, and extracts all URLs from sitemaps β€” including sitemap indexes with recursive child resolution, gzip-compressed files, and plain-text sitemaps. Results are returned as structured JSON for SEO audits, pre-launch crawl verification, and security reconnaissance. --- ### πŸ‘€ Who Is This For? | Persona | Need | How This Actor Helps | |---------|------|---------------------| | **SEO Specialist** | Audit crawl configuration across client domains | Batch-analyze multiple domains in one run; check which paths are blocked; verify sitemap presence and completeness | | **Web Developer** | Verify robots.txt before site launch | Test specific URLs (e.g., `/admin`, `/api`, staging paths) against live rules before going live | | **DevOps / SRE** | Infrastructure monitoring and compliance | Automate periodic robots.txt and sitemap checks across server fleets; detect configuration drift | | **Security Analyst** | Discover information disclosure risks | Identify exposed disallowed paths, unreferenced admin areas, and non-standard sitemap locations | | **Content Strategist** | Validate sitemap coverage | Confirm all important pages are listed in sitemaps; detect orphan pages not present in any sitemap | | **Digital Agency** | Client onboarding audit | Generate comprehensive crawl configuration reports for new clients during technical SEO audits | --- ### ✨ Features - **Full robots.txt parsing** β€” Extracts User-agent, Disallow, Allow, and Sitemap directives with correct precedence rules (most-specific-path-wins, Allow-overrides-Disallow at equal specificity). Handles comments, blank lines, malformed entries, and case-insensitive directives. - **URL compliance testing** β€” Test any list of URL paths against the extracted robots.txt rules. Each result shows whether the URL is allowed or blocked and which rule matched. Relative paths are automatically resolved against the domain. - **Sitemap discovery & extraction** β€” Detects sitemaps declared in `robots.txt` via the `Sitemap` directive. Falls back to common paths (`/sitemap.xml`, `/sitemap_index.xml`, `/sitemaps/sitemap.xml`) when no sitemap is declared. - **Sitemap index resolution** β€” Automatically detects sitemap index files, fetches each child sitemap, and merges all URLs into a single result set. Supports up to 50 child sitemaps per index. - **Multi-format sitemap support** β€” Parses standard XML sitemaps, gzip-compressed (`.gz`) sitemaps, and plain-text sitemaps (one URL per line). Correctly handles namespace-prefixed XML. - **Batch mode** β€” Analyze up to 20 domains in a single run. Each domain is processed independently with individual error handling β€” a failure on one domain never blocks the others. - **Structured output** β€” Every result is delivered as a clean JSON object with dedicated `robotsTxt`, `sitemap`, and `summary` sections, ready for downstream processing, dashboard ingestion, or data pipeline integration. - **Parallel fetching** β€” Uses `asyncio.gather` with `return_exceptions=True` for concurrent domain processing. Transient network errors trigger automatic retries with exponential backoff (up to 3 attempts). - **Premium charge events** β€” Transparent usage-based billing with distinct events for robots.txt analysis, sitemap extraction, and batch processing. Only successful analyses are charged. --- ### πŸš€ Quick Start #### Single Domain β€” Analyze robots.txt Only ```json { "domain": "example.com", "analyzeRobotsTxt": true, "analyzeSitemap": false } ```` **Response:** ```json { "domain": "example.com", "robotsTxt": { "exists": true, "fetchedAt": "2026-07-24T10:00:00Z", "rawSize": 482, "rules": [ {"userAgent": "*", "path": "/admin", "disallow": true, "allow": false, "comment": ""}, {"userAgent": "*", "path": "/cgi-bin", "disallow": true, "allow": false, "comment": "Block CGI access"}, {"userAgent": "Googlebot", "path": "/public", "disallow": false, "allow": true, "comment": ""}, {"userAgent": "*", "path": "https://example.com/sitemap.xml", "disallow": false, "allow": false, "comment": ""} ], "sitemapUrls": ["https://example.com/sitemap.xml"], "testResults": [], "error": null }, "sitemap": null, "summary": { "httpStatus": true, "contentType": "text/plain", "totalUrlsInSitemap": 0, "ruleCount": 4, "testCount": 0 }, "error": null, "success": true } ``` #### Test Specific URLs Against robots.txt ```json { "domain": "example.com", "analyzeSitemap": false, "testUrls": ["/admin", "/public/page", "/wp-admin", "/images/logo.png"] } ``` **Response excerpt:** ```json { "domain": "example.com", "robotsTxt": { "exists": true, "testResults": [ {"url": "https://example.com/admin", "allowed": false, "matchedRule": "Disallow: /admin", "userAgent": "*"}, {"url": "https://example.com/public/page", "allowed": true, "matchedRule": "Default: allow", "userAgent": "*"}, {"url": "https://example.com/wp-admin", "allowed": true, "matchedRule": "Default: allow", "userAgent": "*"}, {"url": "https://example.com/images/logo.png", "allowed": true, "matchedRule": "Default: allow", "userAgent": "*"} ] } } ``` #### Single Domain β€” Full Analysis with Sitemap Extraction ```json { "domain": "example.com", "analyzeRobotsTxt": true, "analyzeSitemap": true, "extractSitemapUrls": true } ``` **Response excerpt:** ```json { "domain": "example.com", "robotsTxt": { "exists": true, "ruleCount": 4, "sitemapUrls": ["https://example.com/sitemap.xml"] }, "sitemap": { "exists": true, "fetchedAt": "2026-07-24T10:00:01Z", "type": "xml", "urlCount": 142, "urls": [ {"loc": "https://example.com/", "lastmod": "2026-07-23", "priority": "1.0", "changefreq": "daily"}, {"loc": "https://example.com/about", "lastmod": "2026-07-22", "priority": "0.8", "changefreq": "monthly"}, {"loc": "https://example.com/blog/post-1", "lastmod": "2026-07-20", "priority": "0.6", "changefreq": "weekly"} ], "isSitemapIndex": false, "childSitemaps": [] }, "summary": { "httpStatus": true, "contentType": "text/plain", "totalUrlsInSitemap": 142, "ruleCount": 4, "testCount": 0 }, "success": true } ``` #### Batch Mode β€” Multiple Domains ```json { "domains": ["example.com", "example.net", "example.org"], "analyzeRobotsTxt": true, "analyzeSitemap": true, "extractSitemapUrls": false, "batchMode": true } ``` #### Sitemap Index β€” Recursive Resolution When a sitemap is a sitemap index (pointing to multiple child sitemaps), the actor automatically resolves all children: ```json { "domain": "example.com", "analyzeRobotsTxt": false, "analyzeSitemap": true, "extractSitemapUrls": true } ``` **Response excerpt for sitemap index:** ```json { "sitemap": { "exists": true, "type": "xml", "urlCount": 1420, "isSitemapIndex": true, "childSitemaps": [ "https://example.com/sitemap-pages.xml", "https://example.com/sitemap-posts.xml", "https://example.com/sitemap-categories.xml", "https://example.com/sitemap-images.xml" ] } } ``` *** ### πŸ“₯ Input | Field | Type | Required | Default | Description | |-------|------|----------|---------|-------------| | `domain` | `string` | No\* | β€” | Single domain to analyze (e.g., `"example.com"`). Ignored when `batchMode` is true. | | `domains` | `array[string]` | No\* | β€” | List of domains for batch analysis. Requires `batchMode: true`. Max 20 domains. | | `analyzeRobotsTxt` | `boolean` | No | `true` | Fetch and parse `robots.txt`. When enabled, extracts all directives, sitemap declarations, and optionally tests URLs. | | `analyzeSitemap` | `boolean` | No | `true` | Fetch and parse sitemap. Discovers sitemaps from robots.txt declarations and common paths. | | `testUrls` | `array[string]` | No | β€” | List of URL paths to test against robots.txt rules. Relative paths are resolved against the domain. | | `extractSitemapUrls` | `boolean` | No | `true` | Extract all URLs from sitemap(s). When disabled, only reports metadata and URL count without the full URL list. | | `batchMode` | `boolean` | No | `false` | Enable batch processing for multiple domains. Provide domain list via the `domains` array. | *\* Either `domain` or `domains` must be provided.* #### Input Validation Rules - Domain names are automatically cleaned: protocol prefixes (`https://`) and URL paths are stripped. Only the hostname is used. - Duplicate domains are not removed β€” each entry is processed independently. - Invalid domain names (missing TLD, empty strings) are silently skipped. - URLs in `testUrls` that don't start with `http` are resolved as relative paths under `https:///`. *** ### πŸ“€ Output Results are pushed to the default dataset as individual items per domain. Each item contains: | Field | Type | Description | |-------|------|-------------| | `domain` | `string` | The input domain being analyzed. | | `robotsTxt` | `object \| null` | Robots.txt analysis result. `null` when `analyzeRobotsTxt` is `false`. | | `sitemap` | `object \| null` | Sitemap analysis result. `null` when `analyzeSitemap` is `false`. | | `summary` | `object \| null` | Summary statistics combining both analyses. | | `error` | `string \| null` | Error message if the domain could not be analyzed. | | `success` | `boolean` | `true` if at least one analysis (robots.txt or sitemap) completed successfully. | #### robotsTxt Object | Field | Type | Description | |-------|------|-------------| | `exists` | `boolean` | Whether robots.txt was successfully fetched. | | `fetchedAt` | `string` | ISO 8601 timestamp of when the file was fetched. | | `rawSize` | `integer` | Raw byte size of the fetched robots.txt content. | | `rules` | `array[object]` | Parsed robots.txt rules. Each rule has `userAgent`, `path`, `disallow`, `allow`, and `comment` fields. | | `sitemapUrls` | `array[string]` | Sitemap URLs declared via `Sitemap` directives in robots.txt. | | `testResults` | `array[object]` | Results of URL compliance testing. Each entry has `url`, `allowed`, `matchedRule`, and `userAgent`. | | `ruleCount` | `integer` | Total number of parsed rules. | | `error` | `string \| null` | Error message if robots.txt fetching/parsing failed. | #### sitemap Object | Field | Type | Description | |-------|------|-------------| | `exists` | `boolean` | Whether a sitemap was successfully found and parsed. | | `fetchedAt` | `string` | ISO 8601 timestamp of when the sitemap was fetched. | | `type` | `string` | Sitemap format: `"xml"` (standard XML), `"gz"` (gzip-compressed), or `"txt"` (plain-text). | | `urlCount` | `integer` | Total number of URLs extracted from the sitemap. | | `urls` | `array[object]` | Extracted URLs (only when `extractSitemapUrls` is `true`). Each entry has `loc`, `lastmod`, `priority`, and `changefreq`. Capped at 10,000 URLs. | | `isSitemapIndex` | `boolean` | Whether the sitemap is a sitemap index file. | | `childSitemaps` | `array[string]` | URLs of child sitemaps (only when `isSitemapIndex` is `true`). | | `error` | `string \| null` | Error message if sitemap fetching/parsing failed. | #### Summary Object | Field | Type | Description | |-------|------|-------------| | `httpStatus` | `boolean` | Whether any HTTP resource was successfully fetched for this domain. | | `contentType` | `string` | Content type indicator. | | `totalUrlsInSitemap` | `integer` | Total URL count across all discovered sitemaps. | | `ruleCount` | `integer` | Total number of robots.txt rules parsed. | | `testCount` | `integer` | Number of URL compliance tests performed. | *** ### 🎯 Use Cases #### 1. SEO Crawl Budget Optimization Analyze how search engine bots see your site. Identify blocked resources (CSS, JS, images) that may be harming rendering and indexing. Ensure important pages are accessible and less important areas are properly disallowed. ```json { "domain": "example.com", "testUrls": [ "/css/style.css", "/js/app.js", "/images/product.jpg", "/admin/dashboard", "/api/internal" ] } ``` #### 2. Pre-Launch Verification Before pushing a new site or major update, verify that `robots.txt` correctly allows search engines to crawl your new content while blocking staging or development areas. #### 3. Sitemap Coverage Audit Verify that all important pages are included in your sitemap. Compare sitemap URL count against known page inventory to detect orphan pages β€” content that exists on your site but isn't referenced in any sitemap. #### 4. Multi-Domain Portfolio Management Agencies managing dozens of client domains can batch-analyze all sites in a single run, checking for missing `robots.txt`, outdated sitemaps, or misconfigured crawl rules across the portfolio. #### 5. Security Reconnaissance Identify potentially sensitive paths that are disallowed in `robots.txt` (which attackers use as a discovery source). Detect unreferenced admin panels, internal APIs, and staging environments that shouldn't be publicly listed. #### 6. Content Migration Validation After migrating to a new CMS or domain structure, validate that the new `robots.txt` and sitemaps are correctly configured before going live. Automated checking prevents post-migration indexing disasters. #### 7. Competitor SEO Research Analyze how competitors structure their crawl rules and sitemaps. Understand which sections they prioritize in sitemaps and which areas they block from search engines. #### 8. CI/CD Pipeline Integration Embed the actor into your CI/CD pipeline to automatically validate `robots.txt` and sitemap changes before deployment. Fail the build if critical paths are accidentally blocked or sitemaps become invalid. *** ### ❓ Frequently Asked Questions **Q1: What happens if a domain doesn't have a robots.txt file?** The actor reports `robotsTxt.exists: false` and an error message `"robots.txt not found (HTTP 404)"`. The sitemap analysis still runs independently. **Q2: Can I analyze a domain that only uses HTTP (not HTTPS)?** The actor defaults to HTTPS. If a domain only serves over HTTP, the fetch will fail with a connection error. There is no HTTP fallback currently. **Q3: How are relative test URLs resolved?** Test URLs that don't start with `http` are resolved as `https:///`. For example, `testUrls: ["/admin"]` becomes `https://example.com/admin`. **Q4: What's the maximum number of URLs extracted from a sitemap?** The actor caps extraction at 10,000 URLs per domain. Sitemap indexes with child sitemaps are resolved recursively, and the total combined URLs are subject to the same cap. **Q5: How does the robots.txt rule matching work?** The actor implements Google's robots.txt parsing rules: the most specific path wins (longest match), and at equal specificity, an `Allow` directive overrides a `Disallow` directive. If no rule matches, the URL is allowed by default. **Q6: Does the actor follow sitemap index redirects?** Yes, sitemap URLs are fetched with HTTP redirect following enabled (up to 5 redirects). This handles common cases like `http://` to `https://` redirects. **Q7: Are gzip-compressed sitemaps supported?** Yes. Sitemaps served with `Content-Type: application/x-gzip` or `Content-Encoding: gzip`, as well as raw gzip byte streams (magic bytes `\x1f\x8b`), are automatically decompressed and parsed. **Q8: What happens when a sitemap is too large?** If a sitemap (or the combined output of a sitemap index) exceeds 10,000 URLs, the actor caps the output at 10,000 entries. The `urlCount` field in the sitemap object shows the actual parsed count before capping. **Q9: Can I test the same URLs against multiple user-agents?** Currently, URL testing uses the wildcard (`*`) user-agent by default. Multi-user-agent testing is not supported in this version. **Q10: What happens if a domain fails during batch processing?** Each domain is processed independently with its own error handling. A DNS failure, timeout, or invalid response on one domain does not affect the processing of other domains in the batch. **Q11: How are sitemaps discovered when no Sitemap directive exists in robots.txt?** The actor falls back to probing common sitemap paths: `/sitemap.xml`, `/sitemap_index.xml`, and `/sitemaps/sitemap.xml`. The first path that returns HTTP 200 is used. **Q12: Can I extract all sitemap URLs without fetching individual entries?** Yes. Set `extractSitemapUrls: false` to get sitemap metadata (type, URL count, whether it's an index, child sitemap locations) without downloading the full URL list. This is useful for quick checks that don't need the complete URL inventory. **Q13: Does the actor handle internationalized domain names (IDNs)?** Domains should be provided in their Punycode/ASCII form (e.g., `xn--example-123.com`). The actor does not perform IDN-to-Punycode conversion. **Q14: How are malformed robots.txt entries handled?** The parser gracefully skips unrecognized or malformed directives. It handles case-insensitive field names, trailing whitespace, inline comments, and empty directives without crashing. **Q15: Can I run this actor on a schedule?** Yes. Like all Apify actors, you can schedule recurring runs via the Apify Console scheduler or API. This is useful for periodic SEO audits and crawl configuration monitoring. *** ### πŸ’° Usage & Billing This actor uses a pay-per-event pricing model. You are charged only for successfully processed items β€” failed domains do not incur charges. #### Event Summary | Event Name | Price (USD) | Trigger | |------------|-------------|---------| | `apify-actor-start` | $0.005 | Every actor run, charged once at start | | `robots-analysis` | $0.01 | Per domain where robots.txt was successfully fetched and parsed | | `sitemap-extract` | $0.01 | Per domain where sitemap was successfully discovered and parsed | | `batch-analyze` | $0.005 | One-time surcharge when batch mode processes multiple domains (>1) | #### Examples | Scenario | Total Cost | |----------|-----------| | Single domain, both robots.txt and sitemap analyzed | $0.025 (start $0.005 + robots $0.01 + sitemap $0.01) | | Single domain, robots.txt only | $0.015 (start $0.005 + robots $0.01) | | Single domain, sitemap only | $0.015 (start $0.005 + sitemap $0.01) | | Batch of 5 domains, both analyses on all | $0.125 (start $0.005 + 5Γ—robots $0.05 + 5Γ—sitemap $0.05 + batch $0.005) | | Batch of 5 domains, 2 robots.txt failures, all 5 sitemaps succeed | $0.085 (start $0.005 + 3Γ—robots $0.03 + 5Γ—sitemap $0.05 + batch $0.005) | > **Charges are only applied for successfully processed items.** If a domain's `robots.txt` returns 404 or a network error, the robots-analysis charge is not applied for that domain. If no sitemap is found, the sitemap-extract charge is not applied. *** ### πŸ”— MCP Integration The Robots.txt & Sitemap Analyzer can be used as a tool inside any MCP-compatible AI client (Claude Desktop, Cursor, VS Code with Copilot, etc.) via the Apify Model Context Protocol (MCP) server. This allows AI agents to analyze domain crawl configurations through natural language. #### Quick Start The Apify MCP server wraps every published Apify actor as a tool β€” no additional npm packages or Python modules required. Your AI assistant discovers and invokes the actor just like any other MCP tool. #### Example Prompts > "Check the robots.txt for example.com and tell me which paths are blocked." > "Run a sitemap audit on example.com and example.net. List all URLs from both sitemaps and count how many pages each has." > "Test whether /admin, /api, and /wp-admin are allowed by example.com's robots.txt." > "Batch-analyze my top 5 client domains: example.com, example.net, example.org, example.edu, and example.io. I need both robots.txt rules and sitemap stats." #### Claude Desktop Configuration Add the following to your `claude_desktop_config.json`: ```json { "mcpServers": { "apify": { "url": "https://mcp.apify.com" } } } ``` On first connection, your browser opens to sign in to Apify and authorize access to your account. No API tokens are stored in the configuration. #### Bearer Token Alternative (Headless / CI/CD) For headless environments (Cursor, VS Code, automated pipelines) that cannot perform OAuth, use the Bearer token configuration: ```json { "mcpServers": { "apify": { "url": "https://mcp.apify.com", "headers": { "Authorization": "Bearer YOUR_APIFY_TOKEN" } } } } ``` Replace `YOUR_APIFY_TOKEN` with your Apify API token from [Apify Console > Integrations](https://console.apify.com/account/integrations). > **Note:** The MCP server uses OAuth by default for browser-equipped clients. The Bearer token method is a supported alternative for environments that cannot complete the OAuth flow. See the [Apify MCP documentation](https://docs.apify.com/integrations/mcp) for more details. *** ### πŸ€– Related Tools - [**Website Tech Stack Detector**](https://apify.com/perryay/website-tech-stack-detector) β€” Identify technologies powering any website, complementing crawl configuration analysis with infrastructure profiling. - [**SSL Certificate Inspector**](https://apify.com/perryay/ssl-cert-inspector) β€” Deep SSL/TLS certificate analysis including chain inspection, vulnerability scanning, and security ratings. - [**URL Health Checker**](https://apify.com/perryay/url-health) β€” Batch URL status checking with redirect tracking, complementing sitemap URL validation workflows. - [**DNS Records Inspector**](https://apify.com/perryay/dns-records-inspector-a-aaaa-mx-txt-ns-cname) β€” Retrieve DNS records (A, AAAA, MX, TXT, NS, CNAME) for any domain. - [**Domain Age & History Checker**](https://apify.com/perryay/domain-age-history-checker) β€” WHOIS enrichment and domain registration history tracking for comprehensive domain intelligence. # Actor input Schema ## `domain` (type: `string`): Single domain to analyze (e.g., example.com). If batchMode is enabled, use the 'domains' array instead. ## `domains` (type: `array`): List of domains for batch analysis. When provided with batchMode: true, all domains are processed in a single run. ## `analyzeRobotsTxt` (type: `boolean`): Fetch and parse robots.txt β€” extract crawl rules, sitemap directives, and test URL compliance. ## `analyzeSitemap` (type: `boolean`): Fetch and parse sitemap.xml β€” discover URLs, detect sitemap indexes, and extract child sitemap locations. ## `testUrls` (type: `array`): Optional list of URL paths to test against robots.txt rules. Each URL is checked for Allow/Disallow compliance. Relative paths are resolved against the domain. ## `extractSitemapUrls` (type: `boolean`): If true, extract all URLs from sitemap(s) including resolving sitemap indexes. If false, only report sitemap metadata and URL count. ## `batchMode` (type: `boolean`): Enable batch processing for multiple domains. When enabled, provide domains via the 'domains' array field. ## Actor input object example ```json { "domain": "example.com", "domains": [ "example.com" ], "analyzeRobotsTxt": true, "analyzeSitemap": true, "testUrls": [ "/admin", "/wp-admin" ], "extractSitemapUrls": true, "batchMode": false } ``` # Actor output Schema ## `results` (type: `string`): Per-domain analysis results in the default dataset # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domain": "example.com", "domains": [ "example.com" ], "testUrls": [ "/admin", "/wp-admin" ] }; // Run the Actor and wait for it to finish const run = await client.actor("perryay/robots-txt-sitemap-analyzer").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domain": "example.com", "domains": ["example.com"], "testUrls": [ "/admin", "/wp-admin", ], } # Run the Actor and wait for it to finish run = client.actor("perryay/robots-txt-sitemap-analyzer").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domain": "example.com", "domains": [ "example.com" ], "testUrls": [ "/admin", "/wp-admin" ] }' | apify call perryay/robots-txt-sitemap-analyzer --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=perryay/robots-txt-sitemap-analyzer", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Robots.txt & Sitemap Analyzer πŸ•·οΈ", "description": "Fetch, parse, and analyze robots.txt and sitemap.xml for any domain. Extract crawl directives, test URL compliance against robots.txt rules, and discover all URLs from sitemaps including nested sitemap indexes. Supports batch analysis with structured JSON output.", "version": "1.0", "x-build-id": "usPTP8FPEcqIJlTCJ" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/perryay~robots-txt-sitemap-analyzer/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-perryay-robots-txt-sitemap-analyzer", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/perryay~robots-txt-sitemap-analyzer/runs": { "post": { "operationId": "runs-sync-perryay-robots-txt-sitemap-analyzer", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/perryay~robots-txt-sitemap-analyzer/run-sync": { "post": { "operationId": "run-sync-perryay-robots-txt-sitemap-analyzer", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "domain": { "title": "Domain", "type": "string", "description": "Single domain to analyze (e.g., example.com). If batchMode is enabled, use the 'domains' array instead." }, "domains": { "title": "Domains (Batch)", "maxItems": 20, "type": "array", "description": "List of domains for batch analysis. When provided with batchMode: true, all domains are processed in a single run.", "items": { "type": "string" } }, "analyzeRobotsTxt": { "title": "Analyze robots.txt", "type": "boolean", "description": "Fetch and parse robots.txt β€” extract crawl rules, sitemap directives, and test URL compliance.", "default": true }, "analyzeSitemap": { "title": "Analyze sitemap", "type": "boolean", "description": "Fetch and parse sitemap.xml β€” discover URLs, detect sitemap indexes, and extract child sitemap locations.", "default": true }, "testUrls": { "title": "Test URLs", "type": "array", "description": "Optional list of URL paths to test against robots.txt rules. Each URL is checked for Allow/Disallow compliance. Relative paths are resolved against the domain.", "items": { "type": "string" } }, "extractSitemapUrls": { "title": "Extract sitemap URLs", "type": "boolean", "description": "If true, extract all URLs from sitemap(s) including resolving sitemap indexes. If false, only report sitemap metadata and URL count.", "default": true }, "batchMode": { "title": "Batch Mode", "type": "boolean", "description": "Enable batch processing for multiple domains. When enabled, provide domains via the 'domains' array field.", "default": false } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```