Sitemap URL Extractor: Sitemap.xml to URL List

Get every URL a website lists in its sitemaps. Give it domains or sitemap URLs; it finds sitemaps through robots.txt and /sitemap.xml, follows sitemap indexes and gzip files, and returns url, lastmod, changefreq and priority per row, with regex and date filters. No browser, no AI, no key.

What you get

One row per URL: {url, lastmod, changefreq, priority, sitemap}. sitemap is the sitemap file the URL came from. Fields the sitemap does not provide are null; priority is a number.

Use cases

• Build a crawl list for a site audit, SEO review or content inventory.
• Feed a list of URLs into a scraper or into the Website Change Monitor.
• Find recently updated pages with the lastmod filter.
• Compare a competitor's sitemap over time (schedule a run and diff the datasets).
• Give an AI agent a clean list of a site's pages (callable through Apify's MCP server, https://mcp.apify.com).

How it works

• Domain input (example.com): reads https://example.com/robots.txt, collects its Sitemap: lines and also tries /sitemap.xml if that is not already listed.
• Sitemap URL input (path ends in .xml or .xml.gz, or contains "sitemap"): read directly.
• Sitemap indexes are followed (up to 5 levels deep); each sitemap file is read once. Gzip files are decompressed when the data is gzip (up to 100 MB uncompressed).
• Filters: includePatterns (a URL must match at least one), excludePatterns (a URL must match none), lastmodSince (YYYY-MM-DD) and maxUrls. Duplicate URLs are returned once.
• Polite by design: robots.txt is checked for every sitemap request (a disallowed sitemap is skipped with a warning; if robots.txt cannot be fetched because of a server error or timeout, that site is not fetched), sequential requests at most 1 per second per host, an identifying User-Agent, a timeout and redirects followed with the same checks. No proxies or workarounds.

Input

Field	Meaning
domains	Domains or sitemap URLs, one per line (required, up to 50)
includePatterns	Regex list; keep URLs matching at least one
excludePatterns	Regex list; drop URLs matching any
lastmodSince	Keep URLs modified on or after this day; URLs without a readable lastmod are dropped
maxUrls	Stop after this many matching URLs (default 1,000)
maxSitemaps	Cap on sitemap files fetched (default 200)
timeoutSecs	Per-request timeout (default 30 s)

Example

Input (bundled test-input.json): example.com. The record below uses illustrative values only; our tests use synthetic sitemap files, including a sitemap index and a gzip file.

{ "url": "https://example.com/blog/hello", "lastmod": "2026-10-01", "changefreq": "weekly", "priority": 0.8, "sitemap": "https://example.com/sitemap-posts.xml" }

Limitations (read before relying on it)

• Only what the site publishes. A sitemap is the site owner's own list; it can be incomplete, stale or absent. A site with no reachable sitemap returns nothing (the run fails with the reasons if every sitemap failed).
• Discovery is limited to robots.txt and /sitemap.xml. Other locations (for example /sitemap_index.xml) are found only if robots.txt lists them or you pass them as a sitemap URL.
• Image, video and news extensions are not extracted; only the page <loc> and the standard fields. Plain-text sitemaps and RSS/Atom feeds are not supported.
• lastmod, changefreq and priority are returned as published and are often missing or unreliable. lastmodSince drops entries with no readable date.
• Regex filters are JavaScript regular expressions, searched anywhere in the URL; patterns are limited to 200 characters, and very complex patterns on very large sitemaps can be slow.
• Respects robots.txt; it does not bypass blocks, logins or CAPTCHAs, and a 403 or 429 is reported as a warning for that file. Check the site's terms before reusing its URL list.
• Large sites take time: files are fetched one at a time, at least one second apart per host.

Pricing (pay per result)

Pay-per-event with Apify's built-in dataset-item event: each URL row pushed is one result. Use maxUrls and the filters to control cost. The live price is set in Apify Console and shown on the Store page.

FAQ

Do I need an API key? No.

Can I pass several domains? Yes, up to 50 per run; each is discovered separately.

Why are some sitemaps skipped? The warnings in the run log say why: disallowed by robots.txt, HTTP error, not valid sitemap XML or limits reached.

Does it crawl the pages? No. It only reads sitemap files, never the pages themselves.