Sitemap URL Extractor

Get the full list of page URLs of a website from its XML sitemaps, in seconds, without crawling the site.

Give it a domain (example.com) and it finds the sitemaps by itself: first the Sitemap: lines of robots.txt, then the usual locations (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.txt). Or give it a sitemap URL directly.

What it does

• Reads sitemap indexes (nested up to the depth you choose), gzip sitemaps (.xml.gz) and plain-text sitemaps.
• Returns one row per unique URL, de-duplicated across all sitemaps.
• Keeps everything the sitemap says: lastmod, changefreq, priority, number of images and hreflang alternates.
• Include / exclude filters (regular expressions) so you only get, for example, /blog/ or /product/ pages, and only pay for those.
• Writes a run report (RUN_REPORT in the key-value store): which sitemap files were read, how many URLs each had, and any error (404, blocked, invalid file).

Good to know

• It only downloads sitemap files and robots.txt; it does not visit the pages, so it is fast and light on the target site.
• It respects robots.txt: if a sitemap file is disallowed, it is skipped and reported.
• It does not log in anywhere, does not use a browser and does not collect personal data. Private and internal addresses are refused.
• lastmod is only as good as the sitemap: many sites omit it, and then the field is null. This actor never invents dates.
• Limits per run: up to 500 sitemap files and 50 MB per file (the sitemaps.org protocol limit).

Input example

{ "startUrls": ["crawlee.dev"], "maxUrls": 5000, "includePatterns": ["/docs/"] }

Output example

{
  "url": "https://crawlee.dev/docs/quick-start",
  "lastmod": "2026-10-30T00:00:00.000Z",
  "changefreq": "weekly",
  "priority": 0.5,
  "imageCount": 0,
  "alternates": [],
  "sitemap": "https://crawlee.dev/sitemap.xml",
  "site": "https://crawlee.dev"
}

(Values above are illustrative.)

Use it from an AI agent or the API

The output is flat JSON, one URL per row, ready for an agent or a script: call the actor with startUrls and read the dataset items. Use maxUrls to cap the size (and the cost) of each call. The cap applies to the whole run, not per site: with several sites and a low cap, the first site can use it all, so run large sites separately.

Pricing

Pay per event: US$ 0.10 per 1,000 URLs returned (url-extracted), plus a tiny start fee of US$ 0.00005 per run. You pay only for URLs in the results: filtered-out URLs, duplicates and errors are free. You can set a maximum spend per run in Apify, and the actor stops cleanly when it is reached.

Feedback

Found a sitemap it cannot read? Open an issue with the sitemap URL and it will be looked at.