Sitemap Scraper — find the sitemap, follow it, and watch for new URLs

For content audits, migration URL inventories, competitor page counts and watching a site for new pages: give it a domain — stripe.com is enough. It reads robots.txt to find where that site actually keeps its sitemap, follows nested indexes, unpacks gzip, and returns one row per URL with the last-modified date, change frequency and priority the sitemap states.

Asking for /sitemap.xml fails on most real sites. Measured on 2026-09-07 against ten live sites: stripe.com answers 404 there (its robots.txt declares /sitemap/sitemap.xml); www.nytimes.com answers 403 (and declares 25 separate sitemaps); github.com answers 406 and declares none at all — which this Actor reports instead of guessing. www.shopify.com uses a different name (sitemaps_list.xml), www.bbc.com declares 39 sitemaps, and apify.com's is an index pointing at 12 more files.

You are charged per URL returned. A site that produced nothing — no sitemap, a sitemap that answered 403, or a run that could not read robots.txt — comes back as its own row with the reason and is not billed.

It can also watch the same domains and return only URLs it has not returned before, so a daily run on a site that published nothing costs nothing. No API key. Export as JSON, CSV or Excel.

Why finding it matters

Asking for /sitemap.xml fails on a lot of real sites. Measured on 2026-09-07 against ten live sites:

Site	/sitemap.xml	Where the sitemap actually is
stripe.com	404	robots.txt declares /sitemap/sitemap.xml
www.nytimes.com	403	robots.txt declares 25 separate sitemaps
www.shopify.com	200	robots.txt declares sitemaps_list.xml (a different name)
www.bbc.com	200	robots.txt declares 39 sitemaps
github.com	406	declares none — this Actor reports that instead of guessing
apify.com	200	an index pointing at 12 more files

So the order is: read robots.txt first, take every Sitemap: line, and only guess if there are none. Nine common locations are tried as a fallback, and a site with no sitemap is reported as no-sitemap with the reason — not returned as an empty result.

Monitoring mode: only what is new

Turn on Monitoring mode and the Actor remembers every URL it returned.

• The first run returns everything and remembers it.
• Every run after that returns only URLs that were not there before.
• A quiet day returns one row saying nothing is new, and nothing is charged.

That makes a daily watch on a competitor's site, a documentation site or a job board cheap: you pay for new pages, not for the whole sitemap every day.

What comes back

Field	What it is
site, url	The site you asked for, and the URL found in its sitemap
host, path, depth	Split out so you can group by section. depth is how many path segments deep
extension	pdf, html, … or null when the URL has none
lastmod, lastmodAt	Exactly as the sitemap wrote it, and the same value as an ISO instant
changefreq, priority	What the sitemap claims. null when it says nothing
sitemapUrl, sitemapDepth	Which file this URL came out of, and how many indexes deep it was
foundVia	robots.txt, common-path or given — how the sitemap was located
sitemapIsGzipped	Whether that file was served compressed
status, reason	ok, or why a site produced nothing

A field the sitemap does not state comes back as null. Nothing is filled in with a zero or a guessed date.

Filters

All of these run on URLs already fetched — nothing extra is downloaded to apply them.

• Path contains / does not contain — keep /blog/, drop /tag/
• File extensions — only pdf, only html
• Changed on or after / before — by the sitemap's own lastmod. ⚠️ A URL with no date is kept, because unknown is not the same as old
• Minimum priority, Change frequency, Keywords / Exclude keywords

Limits you control

Maximum URLs, Maximum sitemap files and Maximum index depth. Large sites split their sitemap into hundreds of files, so these keep a run bounded and predictable.

Pricing

Charged per URL returned. Sites that produced nothing — no sitemap, a sitemap that answered 403, a run where nothing is new in monitoring mode, or filters that matched nothing — come back with the reason and are not charged.

Notes

• gzip is detected by the file's own first two bytes, not by the .gz in the name, because some servers compress without saying so in the URL.
• The same sitemap file is never read twice in one run, and the same URL is never returned twice.
• Only http and https are accepted. Anything else is rejected rather than rewritten into a different host.