Sitemap Scraper: Every URL on Any Site

Give it a domain. Get back every URL that site publishes, with the date each one last changed.

{ "startUrls": ["apify.com"], "maxUrlsPerSite": 5000 }

You do not have to find the sitemap first. That is the part that actually takes time, and it is the part this Actor does for you.

Where the other sitemap tools stop

	Typical Store Actor	This one
Input	you paste the exact sitemap URL	a bare domain is enough
robots.txt	ignored	read first, it is the authoritative answer
Sitemap index files	first level only	followed to any depth
.xml.gz	fails or returns binary	unpacked
No sitemap at all	empty run, no explanation	one error row saying so, unbilled

That third row is the one that quietly costs people money elsewhere. A sitemap index is a sitemap of sitemaps; a site with 400,000 URLs typically publishes an index pointing at 40 files of 10,000 each. Parsing only the top level returns about 40 rows and looks like it worked.

What each row gives you

Field
url	the page URL
path	just the path, for filtering and grouping
lastmod	normalised to UTC ISO, sites write it five different ways
changefreq, priority	as declared, when present
discoveredVia	robots.txt or which path was guessed
sitemap	which sitemap file this URL came from

Pull only what changed

{ "startUrls": ["example.com"], "changedSince": "2026-07-01" }

Re-crawling a whole site to find the 30 pages that moved is the usual waste. changedSince filters on lastmod, so you get the 30.

Filter by section

{
  "startUrls": ["example.com"],
  "includePattern": "^/blog/",
  "excludePattern": "/(tag|author|page)/"
}

Both are regular expressions tested against the path.

Use cases

• SEO audits: full URL inventory, find orphan and stale pages
• Migration QA: diff the sitemap before and after a replatform
• Feeding a crawler: a clean seed list beats spidering blindly
• Content monitoring: schedule it and watch lastmod move
• AI/RAG pipelines: enumerate a documentation site before ingesting it

Pricing

Charged per URL returned. A site with no sitemap, a sitemap that 404s, or one that fails to parse produces an error row and costs $0.00: you are never billed for a lookup that gave you nothing.

Do you need an API key? No. Is there a free tier? Yes.

The two things people search for in this category, measured, are free and API key: sitemap scraper, sitemap scraper python.

So, plainly:

• No API key. Nothing to register for, nothing to rotate, no key to leak in a repo. The source is a public API published deliberately by its owner.
• No proxy setup. The source does not bot-wall datacenter addresses, so the default works.
• Pay per result, not per month. There is no subscription and no minimum. Rows that error or that your filters drop are not charged.
• Free to try. Run it with the prefilled input and see real rows before deciding anything.

Notes

• URLs come from what the site declares. A sitemap can be stale or incomplete; this reports it faithfully rather than pretending otherwise.
• changedSince drops URLs that carry no lastmod, because there is no honest way to decide whether they changed.
• Up to 200 sitemap files are fetched per site, which covers roughly two million URLs and stops a self-referencing sitemap from looping forever.

FAQ

Do I have to know the sitemap URL?

No, and that is the point. Give it a bare domain and it reads robots.txt first, which is the authoritative answer, then falls back to the usual paths. Every row carries discoveredVia so you can see which route found it.

What if the site has no sitemap?

You get one error row saying so, and it is not charged. An empty dataset with no explanation is the failure mode elsewhere.

Does it handle sitemap index files?

Yes, to any depth. This is the difference that matters: a sitemap index is a sitemap of sitemaps, and a site with 400,000 URLs typically publishes an index pointing at 40 files of 10,000. A tool that parses only the top level returns about 40 rows and looks like it worked.

Are gzipped sitemaps supported?

Yes. .xml.gz is unpacked. Large sites commonly serve only the compressed form.

How do I find just the pages that changed?

changedSince: "2026-07-01" filters on lastmod, so you get the pages that moved instead of re-crawling the whole site to find them. Note that URLs carrying no lastmod at all are dropped by that filter, because there is no honest way to decide whether they changed.

Can I limit it to one section of a site?

Yes. includePattern and excludePattern are regular expressions tested against the path: ^/blog/ to keep a section, /(tag|author|page)/ to drop pagination and taxonomy noise.

Is a sitemap the same as everything on the site?

No, and the distinction matters for an audit. These are the URLs the site declares. A sitemap can be stale, incomplete, or list pages that now 404. This reports it faithfully rather than implying it is a crawl.

How large a site can it handle?

Up to 200 sitemap files per site, which covers roughly two million URLs and also stops a self-referencing sitemap from looping forever.