Sitemap & robots.txt Consistency Auditor

Find the URLs in your XML sitemaps that search engines can't or won't index: pages blocked by your own robots.txt, 404s, redirects, noindex pages, canonical mismatches, and stale or invalid lastmod dates. You get one row per URL plus a per-site summary.

Use it on sites you own or manage: before a launch, after a migration, or on a schedule to catch regressions.

What it checks

robots.txt

• Reads robots.txt as specified by RFC 9309. A 4xx response means "no restrictions"; a 5xx or unreachable robots.txt means "assume everything is disallowed", so nothing else is fetched.
• Lists the Sitemap: lines and honours Crawl-delay (up to 10 s).

Sitemaps

• Follows sitemap indexes (up to 3 levels deep).
• Reads XML sitemaps, gzipped .xml.gz, plain-text sitemaps and RSS/Atom feeds.
• Validates each file: invalid XML, more than 50,000 URLs, more than 50 MB, URLs on a different host, HTTP errors, redirects.

Each listed URL

• Is the URL allowed by robots.txt? Disallowed URLs are reported and never fetched.
• HTTP status, without following redirects, so you see the redirect target.
• noindex from the X-Robots-Tag header or a <meta name="robots"> tag.
• <link rel="canonical"> pointing somewhere other than the listed URL.
• lastmod that is invalid, in the future, or older than N days.
• Duplicates across sitemaps.

Input

Field	Default	Meaning
startUrls	—	Homepages (sitemaps are found via robots.txt, then /sitemap.xml and /sitemap_index.xml) or direct sitemap URLs
maxUrlsPerSite	500	Upper limit on URL rows per site. Sitemap discovery stops once this many URLs are collected
checkUrls	true	Fetch each allowed URL. Set to false to only list and validate sitemap entries
staleAfterDays	365	Flag lastmod older than this. 0 disables it
maxConcurrency	3	Parallel requests per site. A robots.txt Crawl-delay forces 1
maxSitemaps	100	Sitemap files read per site
requestTimeoutSecs	20	Per-request timeout

{ "startUrls": [{ "url": "https://example.com" }], "maxUrlsPerSite": 1000 }

Output

Dataset: one row per listed URL.

{
  "type": "url",
  "siteUrl": "https://example.com",
  "url": "https://example.com/old-page",
  "sitemapUrl": "https://example.com/sitemap-pages.xml",
  "lastmod": "2019-01-01",
  "changefreq": null,
  "priority": null,
  "hreflangAlternates": 0,
  "robotsAllowed": true,
  "statusCode": 301,
  "redirectTo": "https://example.com/new-page",
  "contentType": "text/html",
  "noindex": false,
  "canonical": null,
  "responseTimeMs": 182,
  "issues": ["redirect", "stale-lastmod"],
  "checkedAt": "2026-10-26T14:31:40.000Z"
}

Key-value store OUTPUT: one summary per site. It covers the robots.txt status, declared sitemaps, crawl delay, every sitemap file with its format, URL count and issues, plus total URLs listed, duplicates, issue counts, and whether the listing was complete.

Issue codes

Code	Meaning
blocked-by-robots	Listed in a sitemap but disallowed by robots.txt. A contradiction search engines report
not-found / http-error	404/410, or another 4xx/5xx
redirect	3xx. Sitemaps should list final URLs
noindex	The page asks not to be indexed but is listed in the sitemap
canonical-mismatch	The canonical tag points to a different URL
stale-lastmod / future-lastmod / invalid-lastmod	Problems with lastmod
duplicate-in-sitemaps	The same URL appears more than once
cross-host-url	The URL's host differs from the sitemap's host, and the sitemap isn't declared in robots.txt
timeout / network-error	No response after one retry. The row's error field has the detail
sitemap level: sitemap-http-error, sitemap-redirect, invalid-xml, not-a-sitemap, too-many-urls, sitemap-too-large, sitemap-blocked-by-robots, cross-host-urls, sitemap-nesting-too-deep
site level: robots-txt-missing, robots-txt-unreachable, robots-txt-redirect, no-sitemap-in-robots-txt, no-sitemap-found, max-sitemaps-reached, crawl-delay-too-high

How it behaves on your site

• It identifies itself as SitemapRobotsAuditor/1.0 and obeys robots.txt rules for that agent (or *).
• It sends at most maxConcurrency parallel requests and honours Crawl-delay.
• It downloads at most 256 KB of each HTML page, only to read the <head>. Non-HTML responses are not downloaded.
• It collects no personal data. It records URLs, status codes and tags.

Pricing

Pay per event: each URL row in the dataset is one url-audited event. Use maxUrlsPerSite and the run's maximum cost setting to cap spend. The run stops cleanly when the cap is reached.

Limitations

• JavaScript-rendered noindex or canonical tags are not detected; only the server HTML is read.
• When a site lists more URLs than maxUrlsPerSite, the first URLs found in sitemap order are audited. Duplicate detection covers only the sitemaps read.

Support

Open an issue on the Actor's Issues tab. Breakages are fixed within 48 hours.

This Actor is built and maintained by an AI agent working for its publisher. A human owns the account and is responsible for it.