Sitemap URL Extractor: Nested, Gzip & Filters

Get the page URLs a website lists in its sitemaps. Give Sitemap URL Extractor a domain, homepage, robots.txt or sitemap URL. It finds the sitemaps, follows nested sitemap indexes, reads gzipped (.xml.gz), RSS/Atom and plain-text sitemaps, and returns a clean, de-duplicated list of URLs with lastmod, changefreq, priority, hreflang alternates and the sitemap each URL came from.

If a site has no sitemap, the run still succeeds: the report says so and (optionally) a light, robots.txt-respecting crawl lists the site's internal links instead.

What can Sitemap URL Extractor do?

• 🔎 Auto-discovers sitemaps: reads Sitemap: lines in robots.txt, then tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemaps.xml, /sitemap/sitemap.xml and /sitemap.txt. HTML "soft 404" pages returned with status 200 are detected and skipped.
• 🌳 Follows nested sitemap indexes up to 10 levels deep (default 6), fetching each sitemap once even when several indexes list it.
• 🗜️ Reads the common sitemap formats: XML urlsets and indexes, .xml.gz (detected by content, not file name), RSS 2.0, Atom and plain text. Common XML mistakes (stray &, bad entities, CDATA, odd namespace prefixes) are tolerated.
• 🎯 Filters: include/exclude regex, last-modified date range (2026-01-31 or 30 days), max URLs per run and per site, same-host only. With a date filter, whole child sitemaps older than your start date are skipped, which saves time and money.
• 🌍 Extensions: hreflang alternates (xhtml:link) and, optionally, image, video and Google News entries.
• 🕸️ No-sitemap fallback: a polite breadth-first crawl of same-host links (50 pages, 2 clicks deep by default) that respects robots.txt Disallow rules.
• 🧯 Reliable and polite: at most 2 requests per host at a time, a connect and idle timeout on every request (so a stalled server can't hang the run), retries with exponential backoff that honour Retry-After, and Apify Proxy only when a site blocks direct requests.
• 📉 Low memory: 50 MB sitemaps are parsed as they stream in; the default is 256 MB of memory.
• 🧾 Per-site report: how sitemaps were found, which failed and why (404, bot protection, DNS error, not a sitemap), and how many URLs were filtered or de-duplicated.

What can I use a sitemap URL list for?

• SEO audits: compare the URLs in the sitemap with what's indexed, find stale lastmod dates, check hreflang coverage.
• Crawl seeding: feed the URL list to a scraper (Website Content Crawler, Cheerio/Playwright scrapers) instead of crawling blindly.
• Change monitoring: schedule weekly with lastmodFrom: "7 days" to get only pages that changed.
• Content inventory and migrations: a full list of pages before a redesign or CMS move.
• AI and RAG pipelines: collect the exact page list of a docs site or blog before ingesting it.

How to get a website's URLs from its sitemap

1. Open the Actor and enter a site in Start URLs (e.g. https://www.sitemaps.org) or a bare domain in Domains.
2. Optional: add Include/Exclude patterns (e.g. exclude /tag/) or a Modified on or after date.
3. Click Start. Small sites finish in seconds.
4. Open the Output tab and download the list as JSON, CSV, Excel or HTML, or read it through the API.

Example input

{
  "startUrls": [{ "url": "https://www.nytimes.com/sitemaps/new/news.xml.gz" }],
  "domains": ["apify.com", "wordpress.org"],
  "excludePatterns": ["/tag/", "\\?page="],
  "lastmodFrom": "30 days",
  "maxUrls": 50000
}

Input

Field	What it does
Start URLs	Homepages, robots.txt files or sitemap URLs (with https://). Mix freely.
Domains	Bare domains, one per line (example.com).
Auto-discover sitemaps	On by default.
Maximum URLs / per site	0 = no limit (the run still stops at your Maximum cost per run).
Include / Exclude (regex)	e.g. include /blog/, exclude /tag/.
Modified on or after / before	ISO date or relative (7 days).
Keep URLs without a lastmod date	When a date filter is set. Default on.
Include hreflang alternates / media	Output options.
Crawl the site when no sitemap exists	Default on; max pages 50, depth 2.
Proxy configuration and mode	Default: Apify Proxy only when blocked.

Output

One dataset item per unique URL:

{
  "url": "https://apify.com/store",
  "lastmod": "2026-10-20T00:00:00.000Z",
  "changefreq": "daily",
  "priority": 0.8,
  "sourceSitemap": "https://apify.com/sitemap-0.xml",
  "alternates": [{ "hreflang": "de", "href": "https://apify.com/de/store" }],
  "domain": "apify.com",
  "discoveredBy": "sitemap"
}

Crawled URLs have discoveredBy: "crawl", sourceSitemap: null and foundOn. The OUTPUT record in the key-value store holds the per-site report:

{
  "urlsExtracted": 20000, "sites": 2, "sitesWithUrls": 1, "sitemapsRead": 19,
  "domains": [
    { "domain": "apify.com", "status": "ok", "discoveredBy": "robots.txt", "sitemapsFoundCount": 19, "urlsOutput": 20000, "sitemapsFailed": [] },
    { "domain": "example.com", "status": "no-sitemap", "notes": ["This site publishes no sitemap at robots.txt or the usual locations."] }
  ]
}

Site statuses: ok, partial, ok-all-filtered, blocked, no-sitemap, no-sitemap-crawl-empty, empty, failed, invalid-input.

How much does it cost to extract sitemap URLs?

Pay per URL extracted: you pay only for URLs that end up in your dataset (after filters and de-duplication), plus a tiny start fee per run. Prices fall with your Apify plan, from $0.20 per 1,000 URLs on the free plan to $0.08 per 1,000 on Gold and above. Set Maximum cost per run and the Actor stops cleanly at the limit. Use Maximum URLs or a date filter to preview a huge site cheaply.

Use it from Python, JavaScript or the API

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("drop_data_solutions/sitemap-url-extractor").call(
    run_input={"domains": ["sitemaps.org"], "maxUrls": 1000}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item.get("lastmod"))

Schedule it, chain it into a scraper with an integration, or call it from AI agents through the Apify MCP server.

Tips

• Only want what changed? Set Modified on or after to 7 days and schedule the Actor weekly.
• Blocked sites: some sites answer 403 to datacenter IPs, even through Apify's datacenter proxy (in our tests: zalando.de, and Wikipedia's sitemap endpoint). They are reported as blocked. A Residential proxy may get through, but it isn't guaranteed.
• Millions of URLs: set Maximum URLs or a date filter first.

FAQ

What if the website has no sitemap?

The report marks the site no-sitemap. With Crawl the site when no sitemap exists on (default), a light same-host crawl lists internal links instead.

Does it read sitemap index files and .xml.gz?

Yes. Nested indexes are followed recursively, and gzip is detected from the file's bytes, so mislabeled files work too.

Can it get past bot protection?

Not always. The Actor doesn't solve CAPTCHAs or JavaScript challenges. It retries rate limits politely and switches to Apify Proxy when a site blocks direct requests. If the site still refuses, the report marks it blocked and the run carries on with your other sites.

Is it legal to extract URLs from sitemaps?

Sitemaps are published so machines can read them. This Actor reads only public sitemap files and, in the fallback, public pages allowed by robots.txt. Make sure your use of the data complies with the site's terms and applicable law.

Support

Found a site that doesn't work? Open an issue on the Issues tab with the URL. Built and maintained by drop_data_solutions.