Sitemap URL Extractor — All Page URLs, Lastmod & Hreflang

Get every page URL of a website from its XML sitemaps: robots.txt discovery, sitemap indexes, .gz, text and RSS sitemaps. Each URL comes with lastmod, changefreq, priority, images, videos, news and hreflang alternates. Filters, dedupe, change detection. $0.40 per 1,000 URLs.

Get all URLs of a website from its XML sitemaps, with all the metadata. This sitemap extractor (sitemap URL extractor / XML sitemap parser / sitemap scraper) takes a domain, a home page or a sitemap URL and returns one row per page URL. It finds the sitemaps for you (the Sitemap: lines in robots.txt, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and other common paths), follows sitemap indexes to any depth with a cycle guard, and reads gzip (.xml.gz), text (sitemap.txt) and RSS/Atom feeds used as sitemaps. Each URL comes with lastmod (ISO 8601), changefreq and priority, plus the extensions other extractors drop: images (image:image loc, title, caption), videos (title, thumbnail, content and player URLs, duration), Google News (title, publication, date) and hreflang alternates (xhtml:link language versions). Filter by URL pattern and by lastmod date ("modified in the last 7 days"; old child sitemaps are not even downloaded), remove duplicates across sitemaps, optionally check each URL's HTTP status (200/301/404), and turn on change detection to get only new and updated URLs on scheduled runs, with removed URLs listed in the summary. The run summary shows the sitemap tree with URL counts per sitemap, a lastmod histogram by month and any errors. Huge sitemaps (50 MB / 50,000+ URLs per file) are streamed, not loaded whole, and malformed XML is read leniently. Use it for SEO audits, site migrations, content inventories, feeding crawlers and RAG pipelines, monitoring competitors' new pages, and hreflang checks. $0.40 per 1,000 URLs, with no compute bill on top (Apify's own sitemap extractor is $0.50 per 1,000).

What data does it return?

Every run outputs clean, schema-validated rows like these:

url	lastmod	changefreq	priority	sitemapUrl	images	videos	alternates
https://www.ikea.com/ee/et/p/zebrasaev-laelamp-valge-plast-10522481/	2026-10-18T22:35:41.152Z	weekly	1	https://www.ikea.com/sitemaps/prod-et-EE_1.xml	2	0	0
https://crawlee.dev/python/blog		weekly	0.5	https://crawlee.dev/python/sitemap.xml	0	0	0
https://wordpress.org/embed-image/	2022-10-04T14:46:18.000Z			https://wordpress.org/image-sitemap-1.xml	1	0	0
https://wordpress.org/6-3-hero-animation-v2-mp4-compressed/	2023-08-10T02:07:05.000Z			https://wordpress.org/video-sitemap-1.xml	0	1	0

How much does it cost?

Transparent pay-per-event pricing — you only pay for what you get, and the actor always respects your run budget: set a maximum total charge and it stops cleanly at the cap instead of overrunning.

Event	Price	Charged	Worked example
url-result	$0.0004	per page URL delivered (one dataset row)	A 5,000-page site = $2.00; 50,000 URLs = $20; filtered-out URLs and duplicates are free
url-checked	$0.001	per URL whose live HTTP status is checked (only with 'Check each URL's HTTP status' on)	1,000 URLs with status checks = $0.40 + $1.00 = $1.40
apify-actor-start	$0.00005	per run per GB of memory (platform start fee)	One run at 512 MB = $0.000025

Input options

Field	Type	Required	Default	Description
urls	array	yes	—	Domains (example.com), home pages, or direct sitemap URLs (sitemap.xml, sitemap_index.xml, .xml.gz, sitemap.txt, RSS/Atom feeds). For a domain or page, the site's sitemaps are found automatically from robots.txt and the usual paths. A page URL stands for its whole site: use the URL filters to narrow it down.
discover	boolean	no	true	For domains and page URLs, read the Sitemap: lines in robots.txt, and when there are none try /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt and other common paths. Turn off to read only the exact URLs you give.
includeGlobs	array	no	[]	Keep only page URLs matching at least one of these patterns. * matches anything except /, ** matches anything, e.g. https://example.com/blog/** or **/products/*. Leave empty to keep all.
excludeGlobs	array	no	[]	Drop page URLs matching any of these patterns, e.g. **/tag/** or **?page=*. Filtered URLs are free.
lastmodAfter	string	no	—	Only URLs whose <lastmod> is after this point: a date like 2026-10-01, or a window like "7 days", "24 hours" or "3 months". URLs without a lastmod are left out when this is set. Child sitemaps an index marks as older are not even downloaded.
lastmodBefore	string	no	—	Only URLs whose <lastmod> is before this point (same formats as above). URLs without a lastmod are left out when this is set.
maxUrls	integer	no	100000	Stop after delivering this many URLs in total.
maxSitemaps	integer	no	500	Read at most this many sitemap files (indexes included). Big sites split their URLs over hundreds of files.
includeImages	boolean	no	true	Return each URL's image:image (loc, title, caption), video:video (title, thumbnail, content and player URLs, duration, date) and news:news (title, publication, date) entries. Same price either way.
includeAlternates	boolean	no	true	Return each URL's xhtml:link hreflang alternates (language versions). Same price either way.
dedupe	boolean	no	true	Deliver each URL once, even when it is listed in several sitemaps. Duplicates are free.
checkStatus	boolean	no	false	Also request every delivered URL (HEAD, or GET when HEAD is refused) and return its status code: 200, 301, 404… Redirects are reported, not followed. Charged separately per URL checked, and slower: about 2 requests at a time per site.
detectChanges	boolean	no	false	Save this run's URL list and, next time, mark each URL new, updated (lastmod changed) or unchanged, and list URLs that disappeared in the run summary. Runs with the same URLs and filters (or the same state key) are compared.
onlyChanges	boolean	no	false	With change detection on: deliver only new and updated URLs. Unchanged URLs are skipped and free, so a scheduled run only pays for what changed.
stateKey	string	no	—	Optional name for the saved URL list (letters, digits, - and _), e.g. "my-shop-weekly". Leave empty to derive it from the URLs and filters.

Limits & honest notes

• Reads only what the site publishes in its sitemaps: pages missing from them are not found (there is no link crawling). Sites with no sitemap at all are reported in the summary, free.
• Robots policy: sitemaps listed in the site's robots.txt Sitemap: lines are always read (the owner published them for crawlers). Any other sitemap URL, whether you give it, it is guessed at a common path or linked from an index, is skipped when robots.txt disallows it for our user agent or *. If robots.txt itself can't be fetched (HTTP 5xx or a network error, after one retry), RFC 9309 says to treat the site as off-limits; there is then no Sitemap: list to go by, so on that site only the sitemap URLs you give explicitly are read (no discovery, guessed paths or index children), and the reason is listed in the run's errors. Page URLs are never fetched unless you turn on status checks.
• A domain or page URL stands for its whole site (all of example.com's sitemaps); use 'Only URLs matching' to keep one section.
• Polite by design: at most 2 requests at a time per host, and 429/5xx answers are retried up to 3 times (honouring Retry-After). Status checks therefore run at roughly 10-20 URLs per second per site.
• Sitemap files up to 100 MB downloaded (300 MB uncompressed) are streamed; indexes are followed up to 10 levels deep; at most 'Max sitemap files' files per run (default 500). Truncated .gz files yield the URLs before the cut.
• lastmod values are normalised to ISO 8601 UTC; time-zone-less values are read as UTC, and placeholder or impossible dates (1970, 0000-00-00) become null.
• URLs on internal network addresses (localhost, private IPs, cloud metadata) are never requested and never charged.
• Respects your run budget: set a maximum total charge and the run stops cleanly at the cap, delivering exactly the URLs that were charged. A run restarted by the platform skips URLs it already delivered.

Changelog

• 2026-10-19 — Initial release: robots.txt and common-path sitemap discovery, recursive sitemap indexes with cycle guard, gzip by magic bytes, text and RSS/Atom sitemaps, streaming lenient parser, lastmod/changefreq/priority, image/video/news extensions, hreflang alternates, URL and lastmod filters, dedupe, optional HTTP status checks, change detection against the previous run, sitemap tree and lastmod histogram summary.

---

Built and maintained by ActorForge. Reliability first: every actor in the portfolio is monitored by an automated nightly self-test, and breakages are typically fixed the same day.