Sitemap URL Extractor — every page URL from a website's sitemaps

Give it a domain, a homepage, or a direct link to a sitemap.xml, and get back the full list of the site's page URLs — one dataset row per URL, with the lastmod, changefreq and priority the sitemap declares.

It is built to finish the job on sites where a naive sitemap reader gives up:

• Finds the sitemap for you. Reads robots.txt, then tries the common locations (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, …).
• Follows sitemap-index files, to any depth. A site that splits its sitemap into 40 partitions behind two levels of index still comes back whole.
• Unpacks gzip. .xml.gz sitemaps, and sitemaps served with a gzip body, are decompressed automatically.
• Never fails the whole run for one bad file. Every sitemap gets a row in the report saying whether it was read, and why not if it wasn't.
• No browser. It is HTTP plus XML parsing, so it is fast and the run is cheap.

---

Output

With one row per URL (the default), each row is:

{
  "url": "https://example.com/blog/how-sitemaps-work",
  "lastmod": "2026-08-14",
  "changefreq": "monthly",
  "priority": "0.7",
  "sourceSitemap": "https://example.com/sitemap-posts.xml",
  "website": "https://example.com"
}

That is exactly the shape a crawler or a screenshot tool wants as input, so the dataset drops straight into the next step — for example Website Screenshot & PDF Generator or any crawler that takes a URL list.

Turn "One row per URL" off to get instead one row per website, with the URLs as an array and a per-sitemap breakdown.

The sitemap report

Unless you turn it off, the dataset ends with one row per sitemap file found:

website	sitemapUrl	discoveredVia	status	urlCount	note
https://example.com	/sitemap.xml	robots.txt	ok (index)	12	12 new child sitemap(s)
https://example.com	/sitemap-posts.xml	index: /sitemap.xml	ok	1 843
https://example.com	/sitemap-authors.xml	index: /sitemap.xml	error	0	HTTP 404

This is the row that tells you why a count is lower than you expected — a sitemap the index points to that 404s, a file that isn't XML, a timeout.

Options

Option	What it does
Websites or sitemap URLs	One per line. A bare domain, a homepage, or a direct sitemap link.
Max URLs per website	Stop after N URLs for one site (0 = up to the run limit).
Include only URLs matching / Exclude URLs matching	JavaScript regular expressions applied to each URL.
Only URLs changed on or after	A date. Keeps only URLs whose <lastmod> is on or after it — the rest, and URLs with no <lastmod>, are dropped.
One row per URL	On: a flat URL list. Off: one row per site with an array and a per-sitemap breakdown.
Max sitemap files per website	Safety limit on how many sitemap files to fetch for one site.
Proxy	Optional, for sites that block datacenter traffic or serve a region-specific sitemap.

Pricing

Pay per URL returned. A website with no readable sitemap costs nothing.

A run returns at most 25,000 URLs, so a single run can never surprise you. When a run stops at that limit — or at the maximum cost you allow for one run — it finishes normally and adds a report row saying so, with the sitemap files still left to read: point a new run straight at those, or narrow the result with Include only URLs matching.

Common uses

• Get all URLs from a website to seed a crawl, an archive, or a migration checklist.
• Convert a sitemap.xml to a plain URL list (JSON or CSV via the dataset export).
• Find the pages that changed since your last crawl, using the lastmod filter, and re-process only those.
• Feed a screenshot or PDF Actor one row per page.

Limits

• It reads XML sitemaps (<urlset> and <sitemapindex>). It does not parse HTML sitemap pages or discover URLs by crawling links — give it the sitemap.
• lastmod is reported as the sitemap writes it; it is not independently verified against the page.
• Sites that publish no sitemap return an empty result with a report row saying so. Some large sites (for example python.org) genuinely have none at a standard location.

Issues

Found a sitemap it can't read? Open an issue on the Actor's Issues tab with the URL and it will be looked at.