Robots.txt & Sitemap Analyzer 🤖

Fetch, parse, and analyze robots.txt and sitemap.xml for any domain — rules, URL testing, sitemap extraction, and batch analysis.

A misconfigured robots.txt can block search engines from indexing your content, expose private directories to bots, or waste crawl budget on irrelevant pages. An outdated or missing sitemap means search engines may never discover your important pages. Manually checking these files across multiple domains is tedious, error-prone, and doesn't scale.

Robots.txt & Sitemap Analyzer automates the entire workflow: fetches robots.txt and sitemap.xml from one or more domains, parses every directive with correct precedence rules, tests specific URLs against the extracted rules, and extracts all URLs from sitemaps — including sitemap indexes with recursive child resolution, gzip-compressed files, and plain-text sitemaps. Results are returned as structured JSON for SEO audits, pre-launch crawl verification, and security reconnaissance.

---

👤 Who Is This For?

Persona	Need	How This Actor Helps
SEO Specialist	Audit crawl configuration across client domains	Batch-analyze multiple domains in one run; check which paths are blocked; verify sitemap presence and completeness
Web Developer	Verify robots.txt before site launch	Test specific URLs (e.g., /admin, /api, staging paths) against live rules before going live
DevOps / SRE	Infrastructure monitoring and compliance	Automate periodic robots.txt and sitemap checks across server fleets; detect configuration drift
Security Analyst	Discover information disclosure risks	Identify exposed disallowed paths, unreferenced admin areas, and non-standard sitemap locations
Content Strategist	Validate sitemap coverage	Confirm all important pages are listed in sitemaps; detect orphan pages not present in any sitemap
Digital Agency	Client onboarding audit	Generate comprehensive crawl configuration reports for new clients during technical SEO audits

---

✨ Features

• Full robots.txt parsing — Extracts User-agent, Disallow, Allow, and Sitemap directives with correct precedence rules (most-specific-path-wins, Allow-overrides-Disallow at equal specificity). Handles comments, blank lines, malformed entries, and case-insensitive directives.

• URL compliance testing — Test any list of URL paths against the extracted robots.txt rules. Each result shows whether the URL is allowed or blocked and which rule matched. Relative paths are automatically resolved against the domain.

• Sitemap discovery & extraction — Detects sitemaps declared in robots.txt via the Sitemap directive. Falls back to common paths (/sitemap.xml, /sitemap_index.xml, /sitemaps/sitemap.xml) when no sitemap is declared.

• Sitemap index resolution — Automatically detects sitemap index files, fetches each child sitemap, and merges all URLs into a single result set. Supports up to 50 child sitemaps per index.

• Multi-format sitemap support — Parses standard XML sitemaps, gzip-compressed (.gz) sitemaps, and plain-text sitemaps (one URL per line). Correctly handles namespace-prefixed XML.

• Batch mode — Analyze up to 20 domains in a single run. Each domain is processed independently with individual error handling — a failure on one domain never blocks the others.

• Structured output — Every result is delivered as a clean JSON object with dedicated robotsTxt, sitemap, and summary sections, ready for downstream processing, dashboard ingestion, or data pipeline integration.

• Parallel fetching — Uses asyncio.gather with return_exceptions=True for concurrent domain processing. Transient network errors trigger automatic retries with exponential backoff (up to 3 attempts).

• Premium charge events — Transparent usage-based billing with distinct events for robots.txt analysis, sitemap extraction, and batch processing. Only successful analyses are charged.

---

🚀 Quick Start

Single Domain — Analyze robots.txt Only

{
  "domain": "example.com",
  "analyzeRobotsTxt": true,
  "analyzeSitemap": false
}

Response:

{
  "domain": "example.com",
  "robotsTxt": {
    "exists": true,
    "fetchedAt": "2026-07-24T10:00:00Z",
    "rawSize": 482,
    "rules": [
      {"userAgent": "*", "path": "/admin", "disallow": true, "allow": false, "comment": ""},
      {"userAgent": "*", "path": "/cgi-bin", "disallow": true, "allow": false, "comment": "Block CGI access"},
      {"userAgent": "Googlebot", "path": "/public", "disallow": false, "allow": true, "comment": ""},
      {"userAgent": "*", "path": "https://example.com/sitemap.xml", "disallow": false, "allow": false, "comment": ""}
    ],
    "sitemapUrls": ["https://example.com/sitemap.xml"],
    "testResults": [],
    "error": null
  },
  "sitemap": null,
  "summary": {
    "httpStatus": true,
    "contentType": "text/plain",
    "totalUrlsInSitemap": 0,
    "ruleCount": 4,
    "testCount": 0
  },
  "error": null,
  "success": true
}

Test Specific URLs Against robots.txt

{
  "domain": "example.com",
  "analyzeSitemap": false,
  "testUrls": ["/admin", "/public/page", "/wp-admin", "/images/logo.png"]
}

Response excerpt:

{
  "domain": "example.com",
  "robotsTxt": {
    "exists": true,
    "testResults": [
      {"url": "https://example.com/admin", "allowed": false, "matchedRule": "Disallow: /admin", "userAgent": "*"},
      {"url": "https://example.com/public/page", "allowed": true, "matchedRule": "Default: allow", "userAgent": "*"},
      {"url": "https://example.com/wp-admin", "allowed": true, "matchedRule": "Default: allow", "userAgent": "*"},
      {"url": "https://example.com/images/logo.png", "allowed": true, "matchedRule": "Default: allow", "userAgent": "*"}
    ]
  }
}

Single Domain — Full Analysis with Sitemap Extraction

{
  "domain": "example.com",
  "analyzeRobotsTxt": true,
  "analyzeSitemap": true,
  "extractSitemapUrls": true
}

Response excerpt:

{
  "domain": "example.com",
  "robotsTxt": {
    "exists": true,
    "ruleCount": 4,
    "sitemapUrls": ["https://example.com/sitemap.xml"]
  },
  "sitemap": {
    "exists": true,
    "fetchedAt": "2026-07-24T10:00:01Z",
    "type": "xml",
    "urlCount": 142,
    "urls": [
      {"loc": "https://example.com/", "lastmod": "2026-07-23", "priority": "1.0", "changefreq": "daily"},
      {"loc": "https://example.com/about", "lastmod": "2026-07-22", "priority": "0.8", "changefreq": "monthly"},
      {"loc": "https://example.com/blog/post-1", "lastmod": "2026-07-20", "priority": "0.6", "changefreq": "weekly"}
    ],
    "isSitemapIndex": false,
    "childSitemaps": []
  },
  "summary": {
    "httpStatus": true,
    "contentType": "text/plain",
    "totalUrlsInSitemap": 142,
    "ruleCount": 4,
    "testCount": 0
  },
  "success": true
}

Batch Mode — Multiple Domains

{
  "domains": ["example.com", "example.net", "example.org"],
  "analyzeRobotsTxt": true,
  "analyzeSitemap": true,
  "extractSitemapUrls": false,
  "batchMode": true
}

Sitemap Index — Recursive Resolution

When a sitemap is a sitemap index (pointing to multiple child sitemaps), the actor automatically resolves all children:

{
  "domain": "example.com",
  "analyzeRobotsTxt": false,
  "analyzeSitemap": true,
  "extractSitemapUrls": true
}

Response excerpt for sitemap index:

{
  "sitemap": {
    "exists": true,
    "type": "xml",
    "urlCount": 1420,
    "isSitemapIndex": true,
    "childSitemaps": [
      "https://example.com/sitemap-pages.xml",
      "https://example.com/sitemap-posts.xml",
      "https://example.com/sitemap-categories.xml",
      "https://example.com/sitemap-images.xml"
    ]
  }
}

---

📥 Input

Field	Type	Required	Default	Description
domain	string	No*	—	Single domain to analyze (e.g., "example.com"). Ignored when batchMode is true.
domains	array[string]	No*	—	List of domains for batch analysis. Requires batchMode: true. Max 20 domains.
analyzeRobotsTxt	boolean	No	true	Fetch and parse robots.txt. When enabled, extracts all directives, sitemap declarations, and optionally tests URLs.
analyzeSitemap	boolean	No	true	Fetch and parse sitemap. Discovers sitemaps from robots.txt declarations and common paths.
testUrls	array[string]	No	—	List of URL paths to test against robots.txt rules. Relative paths are resolved against the domain.
extractSitemapUrls	boolean	No	true	Extract all URLs from sitemap(s). When disabled, only reports metadata and URL count without the full URL list.
batchMode	boolean	No	false	Enable batch processing for multiple domains. Provide domain list via the domains array.

* Either domain or domains must be provided.

Input Validation Rules

• Domain names are automatically cleaned: protocol prefixes (https://) and URL paths are stripped. Only the hostname is used.
• Duplicate domains are not removed — each entry is processed independently.
• Invalid domain names (missing TLD, empty strings) are silently skipped.
• URLs in testUrls that don't start with http are resolved as relative paths under https://<domain>/.

---

📤 Output

Results are pushed to the default dataset as individual items per domain. Each item contains:

Field	Type	Description
domain	string	The input domain being analyzed.
robotsTxt	object | null	Robots.txt analysis result. null when analyzeRobotsTxt is false.
sitemap	object | null	Sitemap analysis result. null when analyzeSitemap is false.
summary	object | null	Summary statistics combining both analyses.
error	string | null	Error message if the domain could not be analyzed.
success	boolean	true if at least one analysis (robots.txt or sitemap) completed successfully.

robotsTxt Object

Field	Type	Description
exists	boolean	Whether robots.txt was successfully fetched.
fetchedAt	string	ISO 8601 timestamp of when the file was fetched.
rawSize	integer	Raw byte size of the fetched robots.txt content.
rules	array[object]	Parsed robots.txt rules. Each rule has userAgent, path, disallow, allow, and comment fields.
sitemapUrls	array[string]	Sitemap URLs declared via Sitemap directives in robots.txt.
testResults	array[object]	Results of URL compliance testing. Each entry has url, allowed, matchedRule, and userAgent.
ruleCount	integer	Total number of parsed rules.
error	string | null	Error message if robots.txt fetching/parsing failed.

sitemap Object

Field	Type	Description
exists	boolean	Whether a sitemap was successfully found and parsed.
fetchedAt	string	ISO 8601 timestamp of when the sitemap was fetched.
type	string	Sitemap format: "xml" (standard XML), "gz" (gzip-compressed), or "txt" (plain-text).
urlCount	integer	Total number of URLs extracted from the sitemap.
urls	array[object]	Extracted URLs (only when extractSitemapUrls is true). Each entry has loc, lastmod, priority, and changefreq. Capped at 10,000 URLs.
isSitemapIndex	boolean	Whether the sitemap is a sitemap index file.
childSitemaps	array[string]	URLs of child sitemaps (only when isSitemapIndex is true).
error	string | null	Error message if sitemap fetching/parsing failed.

Summary Object

Field	Type	Description
httpStatus	boolean	Whether any HTTP resource was successfully fetched for this domain.
contentType	string	Content type indicator.
totalUrlsInSitemap	integer	Total URL count across all discovered sitemaps.
ruleCount	integer	Total number of robots.txt rules parsed.
testCount	integer	Number of URL compliance tests performed.

---

🎯 Use Cases

1. SEO Crawl Budget Optimization

Analyze how search engine bots see your site. Identify blocked resources (CSS, JS, images) that may be harming rendering and indexing. Ensure important pages are accessible and less important areas are properly disallowed.

{
  "domain": "example.com",
  "testUrls": [
    "/css/style.css",
    "/js/app.js",
    "/images/product.jpg",
    "/admin/dashboard",
    "/api/internal"
  ]
}

2. Pre-Launch Verification

Before pushing a new site or major update, verify that robots.txt correctly allows search engines to crawl your new content while blocking staging or development areas.

3. Sitemap Coverage Audit

Verify that all important pages are included in your sitemap. Compare sitemap URL count against known page inventory to detect orphan pages — content that exists on your site but isn't referenced in any sitemap.

4. Multi-Domain Portfolio Management

Agencies managing dozens of client domains can batch-analyze all sites in a single run, checking for missing robots.txt, outdated sitemaps, or misconfigured crawl rules across the portfolio.

5. Security Reconnaissance

Identify potentially sensitive paths that are disallowed in robots.txt (which attackers use as a discovery source). Detect unreferenced admin panels, internal APIs, and staging environments that shouldn't be publicly listed.

6. Content Migration Validation

After migrating to a new CMS or domain structure, validate that the new robots.txt and sitemaps are correctly configured before going live. Automated checking prevents post-migration indexing disasters.

7. Competitor SEO Research

Analyze how competitors structure their crawl rules and sitemaps. Understand which sections they prioritize in sitemaps and which areas they block from search engines.

8. CI/CD Pipeline Integration

Embed the actor into your CI/CD pipeline to automatically validate robots.txt and sitemap changes before deployment. Fail the build if critical paths are accidentally blocked or sitemaps become invalid.

---

❓ Frequently Asked Questions

Q1: What happens if a domain doesn't have a robots.txt file? The actor reports robotsTxt.exists: false and an error message "robots.txt not found (HTTP 404)". The sitemap analysis still runs independently.

Q2: Can I analyze a domain that only uses HTTP (not HTTPS)? The actor defaults to HTTPS. If a domain only serves over HTTP, the fetch will fail with a connection error. There is no HTTP fallback currently.

Q3: How are relative test URLs resolved? Test URLs that don't start with http are resolved as https://<domain>/<path>. For example, testUrls: ["/admin"] becomes https://example.com/admin.

Q4: What's the maximum number of URLs extracted from a sitemap? The actor caps extraction at 10,000 URLs per domain. Sitemap indexes with child sitemaps are resolved recursively, and the total combined URLs are subject to the same cap.

Q5: How does the robots.txt rule matching work? The actor implements Google's robots.txt parsing rules: the most specific path wins (longest match), and at equal specificity, an Allow directive overrides a Disallow directive. If no rule matches, the URL is allowed by default.

Q6: Does the actor follow sitemap index redirects? Yes, sitemap URLs are fetched with HTTP redirect following enabled (up to 5 redirects). This handles common cases like http:// to https:// redirects.

Q7: Are gzip-compressed sitemaps supported? Yes. Sitemaps served with Content-Type: application/x-gzip or Content-Encoding: gzip, as well as raw gzip byte streams (magic bytes \x1f\x8b), are automatically decompressed and parsed.

Q8: What happens when a sitemap is too large? If a sitemap (or the combined output of a sitemap index) exceeds 10,000 URLs, the actor caps the output at 10,000 entries. The urlCount field in the sitemap object shows the actual parsed count before capping.

Q9: Can I test the same URLs against multiple user-agents? Currently, URL testing uses the wildcard (*) user-agent by default. Multi-user-agent testing is not supported in this version.

Q10: What happens if a domain fails during batch processing? Each domain is processed independently with its own error handling. A DNS failure, timeout, or invalid response on one domain does not affect the processing of other domains in the batch.

Q11: How are sitemaps discovered when no Sitemap directive exists in robots.txt? The actor falls back to probing common sitemap paths: /sitemap.xml, /sitemap_index.xml, and /sitemaps/sitemap.xml. The first path that returns HTTP 200 is used.

Q12: Can I extract all sitemap URLs without fetching individual entries? Yes. Set extractSitemapUrls: false to get sitemap metadata (type, URL count, whether it's an index, child sitemap locations) without downloading the full URL list. This is useful for quick checks that don't need the complete URL inventory.

Q13: Does the actor handle internationalized domain names (IDNs)? Domains should be provided in their Punycode/ASCII form (e.g., xn--example-123.com). The actor does not perform IDN-to-Punycode conversion.

Q14: How are malformed robots.txt entries handled? The parser gracefully skips unrecognized or malformed directives. It handles case-insensitive field names, trailing whitespace, inline comments, and empty directives without crashing.

Q15: Can I run this actor on a schedule? Yes. Like all Apify actors, you can schedule recurring runs via the Apify Console scheduler or API. This is useful for periodic SEO audits and crawl configuration monitoring.

---

💰 Usage & Billing

This actor uses a pay-per-event pricing model. You are charged only for successfully processed items — failed domains do not incur charges.

Event Summary

Event Name	Price (USD)	Trigger
apify-actor-start	$0.005	Every actor run, charged once at start
robots-analysis	$0.01	Per domain where robots.txt was successfully fetched and parsed
sitemap-extract	$0.01	Per domain where sitemap was successfully discovered and parsed
batch-analyze	$0.005	One-time surcharge when batch mode processes multiple domains (>1)

Examples

Scenario	Total Cost
Single domain, both robots.txt and sitemap analyzed	$0.025 (start $0.005 + robots $0.01 + sitemap $0.01)
Single domain, robots.txt only	$0.015 (start $0.005 + robots $0.01)
Single domain, sitemap only	$0.015 (start $0.005 + sitemap $0.01)
Batch of 5 domains, both analyses on all	$0.125 (start $0.005 + 5×robots $0.05 + 5×sitemap $0.05 + batch $0.005)
Batch of 5 domains, 2 robots.txt failures, all 5 sitemaps succeed	$0.085 (start $0.005 + 3×robots $0.03 + 5×sitemap $0.05 + batch $0.005)

Charges are only applied for successfully processed items. If a domain's robots.txt returns 404 or a network error, the robots-analysis charge is not applied for that domain. If no sitemap is found, the sitemap-extract charge is not applied.

---

🔗 MCP Integration

The Robots.txt & Sitemap Analyzer can be used as a tool inside any MCP-compatible AI client (Claude Desktop, Cursor, VS Code with Copilot, etc.) via the Apify Model Context Protocol (MCP) server. This allows AI agents to analyze domain crawl configurations through natural language.

Quick Start

The Apify MCP server wraps every published Apify actor as a tool — no additional npm packages or Python modules required. Your AI assistant discovers and invokes the actor just like any other MCP tool.

Example Prompts

"Check the robots.txt for example.com and tell me which paths are blocked."

"Run a sitemap audit on example.com and example.net. List all URLs from both sitemaps and count how many pages each has."

"Test whether /admin, /api, and /wp-admin are allowed by example.com's robots.txt."

"Batch-analyze my top 5 client domains: example.com, example.net, example.org, example.edu, and example.io. I need both robots.txt rules and sitemap stats."

Claude Desktop Configuration

Add the following to your claude_desktop_config.json:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com"
    }
  }
}

On first connection, your browser opens to sign in to Apify and authorize access to your account. No API tokens are stored in the configuration.

Bearer Token Alternative (Headless / CI/CD)

For headless environments (Cursor, VS Code, automated pipelines) that cannot perform OAuth, use the Bearer token configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com",
      "headers": {
        "Authorization": "Bearer YOUR_APIFY_TOKEN"
      }
    }
  }
}

Replace YOUR_APIFY_TOKEN with your Apify API token from Apify Console > Integrations.

Note: The MCP server uses OAuth by default for browser-equipped clients. The Bearer token method is a supported alternative for environments that cannot complete the OAuth flow. See the Apify MCP documentation for more details.

---

🤖 Related Tools

• Website Tech Stack Detector — Identify technologies powering any website, complementing crawl configuration analysis with infrastructure profiling.
• SSL Certificate Inspector — Deep SSL/TLS certificate analysis including chain inspection, vulnerability scanning, and security ratings.
• URL Health Checker — Batch URL status checking with redirect tracking, complementing sitemap URL validation workflows.
• DNS Records Inspector — Retrieve DNS records (A, AAAA, MX, TXT, NS, CNAME) for any domain.
• Domain Age & History Checker — WHOIS enrichment and domain registration history tracking for comprehensive domain intelligence.