Sitemap Extractor: Get All URLs from XML Sitemaps
Pricing
from $0.10 / 1,000 results
List every page URL a website publishes in its XML sitemaps (auto-found via robots.txt; indexes and .gz too). Use for a site's full URL list or recently changed pages. Input: startUrls (site or sitemap), optional regex, lastmodSince. Output per URL: url, lastmod, changefreq, priority, sitemap.
A website (https://example.com: sitemaps are discovered from its robots.txt, then /sitemap.xml) or a direct sitemap URL (.xml, .xml.gz, .txt, RSS/Atom). Leave empty to run the small example (crawlee.dev). Default: ["https://crawlee.dev"].
[ "https://crawlee.dev"]Optional. Keep only URLs matching at least one regular expression, e.g. '/blog/' or '\.html$'. Case-insensitive.
Optional. Drop URLs matching any of these regular expressions, e.g. '/tag/' or '\?'.
Optional. Keep only URLs whose is on/after this date (2026-10-01) or within a period ('7 days'). Example: "2026-10-01".
Only used with 'Modified since'. Default: true.
Drop URLs on other hosts than the start URL (www. is ignored). Default: false.
Add the xhtml:link hreflang alternates listed for each URL. Default: true.
Default 1,000 keeps runs quick and cheap. Set 0 for no limit (large sites can have hundreds of thousands of URLs).
0 = no limit. You are only charged for URLs saved. Default: 0.
Safety cap on sitemap files read per site. Default: 500.
How deep to follow nested sitemap indexes. Default: 5.
Sites processed at once. Requests to one host are always sequential, at least 1 s apart (or robots.txt Crawl-delay). Default: 3.