Sitemap + robots.txt Analyzer

Find the crawl problems that quietly cost you traffic. Give it a site; it lints robots.txt, validates every XML sitemap (index files, .gz and nesting supported) and samples the listed URLs live to catch dead pages, redirects and indexing conflicts. HTTP only, no browser, fast and cheap. Built for SEO freelancers, agencies, site owners and AI agents.

What is checked

• robots.txt: missing / 5xx / HTML-instead-of-rules, >500 KiB, wrong content type, syntax errors, unknown directives, rules before any User-agent, patterns without a leading /, relative Sitemap: URLs, unsupported noindex: directive, Disallow: / for all crawlers, blocked CSS/JS, duplicate agent groups, missing Sitemap directive.
• Blocked important pages: home page blocked, and every sitemap URL tested against the robots rules (reported as a sitemap/robots conflict).
• Sitemaps: not found, HTTP errors, redirects, invalid XML, empty, >50,000 URLs, >50 MB, relative or cross-host <loc>, http URLs in https sitemaps, duplicates, over-long URLs.
• lastmod sanity: invalid W3C datetime, future dates, pre-1995 dates, missing everywhere, all identical.
• Live URL sample (evenly spread, default 30/site): 4xx/5xx, redirects (301/302 to a different URL), noindex via meta or X-Robots-Tag, canonical pointing elsewhere, unreachable.
• AI crawler access: allowed/blocked for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and others.

Input

Field	Default	Description
urls	required	Up to 100 sites or domains.
sitemapUrls	none	Extra sitemap URLs to analyze.
checkUrls	true	Live-check a sample of sitemap URLs.
maxUrlChecksPerSite	30	Sample size (max 500). Billed per URL.
robotsToken	Googlebot	Agent used for the blocked-pages test.
delayMs / concurrency	300 / 3	Politeness controls.

{ "urls": ["https://example.com"], "maxUrlChecksPerSite": 50 }

Output

"type": "issue" items (one per finding, sorted high to low; identical findings across many sitemap files are merged with occurrences) and one "type": "site-summary" per site with score, sitemap list, AI crawler access, status counts.

{ "type": "issue", "siteOrigin": "https://www.bbc.com", "scope": "sitemap", "severity": "high", "id": "sitemap-cross-host", "occurrences": 13,
  "message": "100 URL(s) are on a different host than the sitemap (search engines ignore them)." }
{ "type": "site-summary", "siteOrigin": "https://htmx.org", "score": 100, "sitemapUrlsTotal": 188, "urlsChecked": 15, "statusCounts": { "200": 15 } }

Score: 100 minus 20/8/3 per high/medium/low issue (heuristic).

Pricing (pay per event; set in Apify Console)

Event	When	Planned price
apify-actor-start	once per run	~$0.03
site-analyzed	each site	~$0.02
url-checked	each sitemap URL fetched live	~$0.002

Charging only happens on the Apify platform.

Politeness and safety

Identifies as SitemapRobotsAnalyzer; at most 2 requests in flight per host with delayMs spacing; URLs disallowed by robots.txt are not fetched (reported as conflicts); only http/https; localhost, internal hosts and private/reserved IPs refused (connect time and every redirect hop). Sitemap-listed URLs on other hosts are only flagged, never requested beyond the same SSRF checks. Public pages only.

Run locally

npm install && npm test
mkdir -p storage/key_value_stores/default
echo '{"urls":["https://example.com"]}' > storage/key_value_stores/default/INPUT.json
npm start     # results in storage/datasets/default/

Tested on htmx.org (188 URLs, clean), bbc.com (25 sitemap files, cross-host and http URLs, redirects, AI bots blocked), python.org, w3.org, sqlite.org, github.com (no sitemap). Docker: docker build -t sitemap-robots-analyzer . && docker run --user 996 -v <dir>:/usr/src/app/storage sitemap-robots-analyzer.