Sitemap URL Extractor & Auditor (index-aware, gzip, status checks)

Turn any XML sitemap into a clean URL dataset — sitemap-index aware, gzip aware — with lastmod / changefreq / priority per URL, plus an optional robots-respecting HTTP status sample that finds broken pages and redirect chains hiding in the sitemap.

Built for the SEO / migration / QA workflows that start with "get me every URL":

• SEO agencies & consultants — full URL inventory + freshness (lastmod) + status codes per client site, on a schedule.
• Site migrations — extract the complete old-site URL set for redirect mapping; re-run after launch to catch 404s the sitemap still advertises.
• Content audits — the lastmod distribution shows exactly how stale a content estate is.
• Competitor content mapping — a competitor's sitemap is their published content inventory.

What it does

1. Fetches each sitemap URL you give it — follows sitemap indexes recursively (depth-capped), decompresses .xml.gz, and extracts every <url> with its metadata, up to your maxUrls cap.
2. Optionally status-checks a sample (HEAD with GET fallback, redirect detection, response-time) — honouring robots.txt Disallow rules for the checks.
3. Writes a per-run summary (RUN_SUMMARY): sitemaps read, URLs extracted, status breakdown, lastmod coverage.

Proven against real-world scale: the gov.uk sitemap index (35 child sitemaps, 25,000 URLs per child) walks cleanly.

Input

{
    "sitemapUrls": ["https://example.com/sitemap.xml"],
    "maxUrls": 5000,
    "checkStatus": true,
    "statusSampleSize": 200,
    "respectRobots": true,
    "concurrency": 5
}

Output (one row per URL)

{
    "url": "https://www.gov.uk/",
    "lastmod": "2026-07-01T04:45:47+00:00",
    "changefreq": null,
    "priority": "0.5",
    "sourceSitemap": "https://www.gov.uk/sitemaps/sitemap_1.xml",
    "statusCheck": { "status": 200, "redirectedTo": null, "responseMs": 17 }
}

Pricing (pay per event)

Event	Price
Actor start	$0.02 per run
Per 100 URLs extracted	$0.005
Per URL status-checked	$0.0005

A 10,000-URL extract with a 500-URL status sample costs about $0.77.

Fair use & crawling ethics

Sitemaps exist specifically for automated consumption — that is their entire purpose under the sitemaps.org protocol. Status checks are lightweight (HEAD-first, ranged GET fallback, low default concurrency) and respect robots.txt Disallow rules by default; disallowed paths are reported as skipped rather than checked. The actor identifies itself with a stable User-Agent.

Limitations (told straight)

• Extraction is capped by maxUrls and a sitemap-count safety cap; enormous estates should be pulled in slices (one child sitemap per run works well).
• RSS/Atom feeds-as-sitemaps and non-standard sitemap extensions (image/video/news tags) are passed through by URL only — the extension metadata isn't parsed in v1.
• Status sampling checks the first N URLs in extraction order (deterministic), not a random sample.