Sitemap & Robots.txt Auditor

Checks whether search engines can actually crawl and index your site. Fetches robots.txt, follows every declared sitemap (including sitemap indexes), and spot-checks a sample of the URLs inside them -- no browser needed. Get back a letter grade, a 0-100 crawlability score, a severity tag, and a plain-English summary.

What you get, per site

• A letter grade (A+ to F) and 0-100 crawlability score at a glance
• A one-line executive summary ("robots.txt blocks all crawlers -- this site will not be indexed.")
• A severity tag -- critical (robots.txt blocks all crawlers, no usable sitemap could be found, or every sampled URL is broken), warning (other issues), or ok -- for instant triage
• robots.txt status: found or missing, whether it blocks all crawlers, whether it declares any sitemaps
• Sitemap discovery: every sitemap referenced in robots.txt (falls back to /sitemap.xml), following sitemap index files to their child sitemaps
• Sitemap validity: well-formed XML, <urlset> vs <sitemapindex>, total URL count, presence of <lastmod> dates
• Broken URL sampling: a configurable number of sitemap URLs are checked live, concurrently, for a working response, so you catch sitemaps full of 404s or redirects
• Sites are audited in parallel (configurable), so a batch of 20 client sites finishes in a fraction of the time a one-by-one audit would take

Why this instead of checking manually

	Manual spot-checks	Sitemap & Robots.txt Auditor
Coverage	Whatever site you remember to test	Every site you list, automatically
Speed	Open robots.txt, open sitemap.xml, click a few links	Seconds, via parallel audits and sampling
Triage	You read the raw files yourself	Pre-classified severity + letter grade
Repeatability	Forgotten until rankings drop	One click, or schedule it
Output	Notes in a doc	Structured dataset, exportable to CSV/JSON/BI tools

Who this is for

• SEO consultants and agencies auditing client sites
• Developers verifying a new sitemap/robots.txt setup before launch
• Anyone who wants to know if their sitemap is stale, broken, or simply not being read by crawlers

How to use it

1. Add one or more sites to audit (e.g. https://example.com).
2. Optionally tune how many sitemap URLs get spot-checked, how many child sitemaps to follow, and concurrency -- these control run time and cost on very large sites.
3. Run it once, or schedule it to catch regressions after deploys.

Example output (one item)

{
  "site": "https://example.com",
  "score": 100,
  "grade": "A+",
  "severity": "ok",
  "summary": "No issues found -- robots.txt and sitemap look healthy.",
  "robotsFound": true,
  "robotsDisallowsAll": false,
  "sitemapsDeclaredInRobots": ["https://example.com/sitemap.xml"],
  "sitemapsChecked": ["https://example.com/sitemap.xml"],
  "totalUrlsInSitemaps": 342,
  "urlsSampled": 15,
  "brokenSampledUrls": [],
  "issueCount": 0,
  "issues": [],
  "checkedAt": "2026-08-22T00:00:00.000Z"
}

Pricing

This Actor uses pay-per-event pricing:

• Site audited -- charged once per site processed, regardless of how many sitemaps or sampled URLs it involved.

Parallel auditing doesn't cost more -- it just finishes faster.

FAQ

Does this crawl my whole site? No. It only reads robots.txt and the XML sitemap files it references (plus, optionally, a sample of the individual URLs listed inside them). It never follows on-page links.

What if my sitemap is huge? Sitemap index files are followed up to the "max child sitemaps" limit you set, and URL sampling is capped by "URLs to spot-check" -- both are configurable so you can trade thoroughness for speed and cost on very large sites.

What makes something "critical" vs. a "warning"? critical covers the failures that mean a site effectively can't be indexed: robots.txt blocking all crawlers, no usable sitemap found at all, or every sampled sitemap URL failing. warning covers everything else that lowers the score -- worth fixing, but not indexing-breaking.

Does it respect robots.txt itself while auditing? This Actor reads robots.txt as data to audit it, and only ever fetches the sitemap files and sample URLs it explicitly discovers -- it does not perform a general crawl.

Will higher concurrency get me rate-limited? It can, on servers with aggressive rate limiting, though each site is a different host so cross-site concurrency is usually gentle. Start at the default (5) and lower it if you see timeouts spike.