Sitemap URL Extractor – sitemap.xml, sitemap index & gzip via robots.txt

Extract every page URL from the sitemap.xml of any website, in bulk. Give the Actor a domain and it finds the sitemaps through robots.txt, follows sitemap indexes recursively, reads .xml.gz files, removes duplicates and returns one clean row per URL with lastmod, changefreq, priority and the sitemap it came from.

The data comes straight from the public sitemap protocol (robots.txt and sitemap files that sites publish for crawlers). No HTML scraping, no login, no captcha, no personal data.

Why use it

• ✅ Finds the sitemap for you: reads the Sitemap: lines of robots.txt, then tries /sitemap.xml, /sitemap_index.xml and /sitemap.xml.gz. You can also paste a direct sitemap URL.
• ✅ Handles the awkward cases: nested sitemap indexes, gzip files (even when served without the right headers), CDATA and XML entities, plain-text sitemaps, and a byte limit against oversized files.
• ✅ Clean output: URLs are de-duplicated across all sitemaps and all sites in the run, and every row tells you which sitemap file listed it.
• ✅ Filters built in: keep or drop URLs by regular expression, or keep only pages modified since a date, so you do not pay for URLs you would throw away.
• ✅ Polite by design: respects robots.txt and sends at most 2 requests per second to any domain.
• ✅ Pay only for results: one charge per unique URL returned, nothing for sites without a sitemap or invalid inputs.

Use cases

1. SEO audits and migrations: list every indexable URL of a site before a migration, then compare it with what you crawl or what is indexed in Google Search Console.
2. Change monitoring: schedule a weekly run with Modified since to catch new and updated pages on competitor, supplier or documentation sites.
3. Crawl seeding: feed a clean, de-duplicated URL list to your crawler (Crawlee, Scrapy, Playwright) instead of following links blindly.
4. Content and AI pipelines: build the list of documentation or blog pages to ingest into a search index or a RAG knowledge base, filtered by path.
5. Market research: count and compare the size and structure of catalogues (/product/, /category/) across many websites.

How to use it

1. Click Try for free and paste domains, site URLs or direct sitemap URLs into Websites or sitemap URLs.
2. Optionally set Max URLs per site, a regex filter or Modified since.
3. Click Start. Export the results as JSON, CSV, Excel or HTML, or fetch them through the Apify API.

Tip: create a Schedule and use Modified since to receive only the recent changes each time.

What is extracted

Field	Details
url	The page URL as declared in the sitemap
lastmod	Last modification date declared by the site, or null
changefreq, priority	Crawl hints declared by the site, or null
sitemap	The sitemap file that listed the URL
site	Hostname of the website
foundVia	robots.txt, well-known-path or direct
extractedAt	Extraction timestamp

A SUMMARY record in the run's key-value store lists, for each input, the number of sitemap files read, the number of URLs found and a note when something stopped the extraction (no sitemap, blocked by robots.txt, limit reached).

Input

{
  "items": ["www.sitemaps.org", "https://docs.apify.com", "https://example.com/sitemap_index.xml"],
  "maxUrlsPerSite": 1000,
  "excludePattern": "/blog/",
  "modifiedSince": "2026-01-01"
}

Field	Description	Default
items	Domains, site URLs or direct sitemap URLs	required
maxUrlsPerSite	Stop a site after this many unique URLs (caps the cost per site)	10000
includePattern	Regex: keep only matching URLs	none
excludePattern	Regex: drop matching URLs	none
modifiedSince	Keep URLs with lastmod on or after this date; URLs without lastmod are kept	none
maxSitemapsPerSite	Safety limit on sitemap files fetched per site	500
timeoutSecs	Network timeout per download	20

Output

One dataset item per unique URL (real example from a test run):

{
  "url": "https://www.sitemaps.org/",
  "lastmod": "2016-11-21",
  "changefreq": null,
  "priority": null,
  "sitemap": "https://www.sitemaps.org/sitemap.xml",
  "site": "www.sitemaps.org",
  "foundVia": "robots.txt",
  "extractedAt": "2026-09-18T16:34:56.345Z"
}

Pricing

Pay per event: $0.001 per URL, which is $1 per 1,000 URLs. Sites without a sitemap, invalid inputs and duplicate URLs are not charged. You can cap your spending with the maximum charge setting of each run, and with Max URLs per site.

Integrations

• API: start runs and download results from any language with the Apify API or clients.
• Schedules and webhooks: run periodically and trigger a webhook when a run finishes.
• Make, Zapier, n8n: send new URLs to Google Sheets, Slack or your ticketing tool.
• AI agents: callable as a tool through the Apify MCP server.

FAQ

Does it respect robots.txt? Yes. Sitemaps declared in robots.txt are read as the site intends. Guessed paths (such as /sitemap.xml) and sitemap URLs you provide are only fetched if robots.txt allows them, and if robots.txt cannot be fetched, guessing is skipped. Requests are limited to 2 per second per domain.

What if a site has no sitemap? The input is not charged and the SUMMARY record explains why (no sitemap found, blocked, unreachable). This Actor reads sitemaps only; it does not crawl pages to discover links.

Does it read sitemap indexes and .gz files? Yes. Indexes are followed up to 5 levels deep, and gzip is detected from the file content, not from the extension.

Can I get images, videos or news metadata? Not at the moment: the output covers the page URL and its lastmod, changefreq and priority.

How do I only get new pages? Run it on a schedule with Modified since set to the date of your last run. Only URLs declared with a more recent lastmod, or with no lastmod, are returned.

Limitations

• Only URLs declared in sitemaps are returned; pages missing from the sitemap are not discovered.
• lastmod, changefreq and priority are whatever the site publishes and are not always accurate or present.
• Sitemap files above about 60 MB uncompressed are skipped, and each site is capped by Max URLs per site and Max sitemap files per site.
• Sites that block automated requests (HTTP 403 or 429) cannot be read; the Actor does not try to bypass protections.