Sitemap URL Extractor: robots.txt, gzip, Changes

Extract every URL of a website from its XML sitemaps — with lastmod, change frequency and priority — by entering just a domain. Sitemaps are found through robots.txt, sitemap indexes and gzip files are followed, and only the sitemap files are read, never the pages. $0.30 per 1,000 URLs, no start fee.

Run it on a schedule with change tracking to get the new, changed and removed URLs since your last run.

What does this sitemap extractor do?

Give it stripe.com and it returns the URLs listed in Stripe's sitemaps. Behind that one line:

1. It reads robots.txt and takes the sitemaps declared there. Many sites keep theirs at an unusual address (stripe.com/sitemap/sitemap.xml), where tools that only try /sitemap.xml find nothing.
2. When robots.txt lists none, it tries /sitemap.xml, /sitemap_index.xml and /sitemap-index.xml.
3. It follows sitemap indexes, decompresses .xml.gz files and accepts whatever content type the server sends (binary/octet-stream included, which is how booking.com serves its sitemaps).
4. It returns one clean item per URL, deduplicated across all the sitemaps of the website.

It never requests the pages themselves, so a large website costs a handful of file downloads, not thousands of page loads. No browser, no login, no proxy needed.

Why this sitemap extractor?

• Built for large websites. It downloads the sitemap files and nothing else: no request per listed page, no browser. A site with hundreds of thousands of URLs is read in seconds to minutes, for the price of the URLs you keep.
• Finds sitemaps other tools miss. Declarations in robots.txt, nested indexes, .xml.gz files, and servers that send sitemaps as binary/octet-stream or application/gzip are all handled.
• Clear about what went wrong. A website without a sitemap, or one that blocks cloud servers, is reported with its reason in the run summary instead of failing the whole run.
• Pay only for what you keep. Filters, duplicates and, with change tracking, unchanged URLs are free.

Why extract URLs from sitemaps?

• Crawl and RAG pipelines. Get the full list of pages before crawling, and re-crawl only what changed with Modified since or Only save new, changed and removed URLs.
• SEO audits. Check what a site declares to search engines: missing lastmod, stale sections, language alternates (hreflang), image and video sitemaps.
• Competitive monitoring. Schedule a daily run on competitors and receive their new pages: products, blog posts, landing pages, job pages.
• Content inventory and migrations. Export every URL of a site to a spreadsheet in minutes.

How to extract all URLs of a website

1. Enter one or more websites in Websites or sitemaps: a domain (example.com), any page of the site, or a sitemap URL (https://example.com/sitemap.xml).
2. Set Max URLs (500 by default) to control the cost of the run.
3. Optionally add URL patterns (*/blog/*), a Modified since date, or turn on Track changes.
4. Click Start, then open the Output tab and export as JSON, CSV, Excel or XML.

Examples of input

Only the blog posts published in the last week:

{
    "startUrls": ["example.com"],
    "includeUrlPatterns": ["*/blog/*"],
    "modifiedSince": "7 days",
    "maxItems": 1000
}

A daily watch on competitors that returns only what changed:

{
    "startUrls": ["competitor-one.com", "competitor-two.com"],
    "onlyChanges": true,
    "maxItems": 20000,
    "stateStoreName": "competitor-watch"
}

Input

Field	Default	What it does
startUrls	—	Domains, pages or sitemap URLs, one per line.
maxItems	500	URLs saved in this run, across all websites. The run stops as soon as it is reached.
maxItemsPerSite	0	Limit per website, so one large site cannot use the whole budget. 0 means no limit.
includeUrlPatterns	—	Keep only matching URLs. * matches anything; a pattern without * matches URLs that contain it.
excludeUrlPatterns	—	Skip matching URLs.
modifiedSince	—	Keep URLs whose lastmod is on or after a date (2026-09-01) or a duration (7 days).
requireLastmod	false	With a date, also drop URLs that have no lastmod.
sitemapUrlPatterns	—	On large sites, follow only the child sitemaps of an index that match (*blog*).
trackChanges	false	Compare with the previous run and add changeStatus.
onlyChanges	false	Save only new, changed and removed URLs.
stateStoreName	sitemap-urls-state	Key-value store that keeps the tracked state. One name per list you track.
includeAlternates	false	Add the hreflang language alternates of each URL.
checkUrlStatus	false	Send a HEAD request to each saved URL and add its HTTP status and redirect target.
useDefaultLocations	true	Try the usual sitemap addresses when robots.txt lists none.
maxSitemapsPerSite	1000	Safety limit on sitemap files read per website.

Filters are applied before saving, so URLs you filter out are never charged.

Output

One item per URL. Download it as JSON, CSV, Excel or XML, or read it from the API.

{
    "url": "https://www.notion.com/blog/notion-ai-for-work",
    "site": "notion.com",
    "lastmod": "2026-09-14T02:36:13.300Z",
    "changefreq": "yearly",
    "priority": 0.4,
    "sitemapUrl": "https://www.notion.com/blog/sitemap.xml",
    "discoveredFrom": "robots-txt",
    "imageCount": 0,
    "videoCount": 0,
    "newsTitle": null,
    "newsPublishedAt": null,
    "alternates": null,
    "changeStatus": "changed",
    "previousLastmod": "2026-08-02T09:12:40.000Z",
    "previousScrapedAt": "2026-09-16T06:00:04.117Z",
    "scrapedAt": "2026-09-17T06:00:03.905Z"
}

Fields

Field	Description
url	The page URL, as listed in the sitemap.
site	The website it belongs to (host without www.).
lastmod, changefreq, priority	As declared in the sitemap. Dates are ISO 8601 UTC; invalid values are null.
sitemapUrl	The sitemap file the URL was read from.
discoveredFrom	How that sitemap was found: input, robots-txt or default-location.
imageCount, videoCount	Images and videos declared for the page (image and video sitemaps).
newsTitle, newsPublishedAt	Filled for Google News sitemaps.
alternates	hreflang alternates, when you ask for them.
statusCode, redirectUrl, statusError	With Check the HTTP status: the status, the redirect target, or the error.
changeStatus, previousLastmod, previousScrapedAt	Change tracking: new, changed, unchanged or removed.
scrapedAt	When the sitemap was read.

A field the sitemap does not provide is null, never missing. Field names and types do not change outside a major version.

Run summary

The SUMMARY record of the key-value store (Run summary in the Output tab) tells, for each website, what was read and why something is missing. Statuses: ok, partial (a limit or a failing sitemap left part of it unread), no_sitemap, not_reached, blocked, failed. Example with a website read completely and one that blocks cloud servers:

{
    "sitesRequested": 2,
    "sitesOk": 1,
    "urlsSaved": 3978,
    "sites": [
        {
            "input": "docs.apify.com",
            "site": "docs.apify.com",
            "status": "ok",
            "sitemapsFound": 7,
            "sitemapsRead": 7,
            "sitemapsFailed": 0,
            "urlsFound": 3986,
            "urlsSaved": 3978,
            "urlsFilteredOut": 0,
            "urlsRemoved": 0,
            "discoveredFrom": ["robots-txt"],
            "notes": []
        },
        {
            "input": "www.booking.com",
            "site": "booking.com",
            "status": "blocked",
            "sitemapsFound": 0,
            "sitemapsRead": 0,
            "sitemapsFailed": 0,
            "urlsFound": 0,
            "urlsSaved": 0,
            "urlsFilteredOut": 0,
            "urlsRemoved": 0,
            "discoveredFrom": [],
            "notes": ["robots.txt: bot challenge instead of the file (HTTP 202, text/html, 2008 bytes)."]
        }
    ],
    "invalidInputs": [],
    "duplicateInputs": 0,
    "maxItemsReached": false,
    "budgetReached": false,
    "memoryLimitReached": false,
    "changes": null
}

urlsFound counts the entries listed in the sitemaps read; urlsSaved is lower when the same URL appears in several sitemaps, since each URL is saved once.

Tracking changes

Turn on Track changes and every URL gets a changeStatus:

• new — not in the website's sitemaps last run.
• changed — its lastmod moved. Compared only when both runs have a date.
• unchanged — still listed, same date.
• removed — listed last run, gone now. Reported only when the website was read completely in both runs, so an outage or a limit can never look like a wave of removals.

Add Only save new, changed and removed URLs to skip the unchanged ones, which are then not charged. A run cut short by a limit keeps what it did not reach for next time. Changing the URL patterns starts the comparison from scratch. Use one stateStoreName per list you track.

Use it from the API, integrations or an AI agent

Every run is available through the Apify API, so the Actor works as a sitemap API for your own code:

curl -X POST "https://api.apify.com/v2/acts/succinct_glider~sitemap-urls/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["example.com"], "maxItems": 1000}'

Schedule it and send the results to Google Sheets, Slack, a webhook, Make, Zapier or n8n through the built-in integrations. AI agents connected to the Apify MCP server can call it to list the pages of a website.

How much does it cost to extract sitemap URLs?

$0.30 per 1,000 URLs saved ($0.0003 per URL). No start fee. URLs removed by your filters, duplicates and, with Only save new, changed and removed URLs, unchanged URLs are not charged. Set a maximum cost per run in the run options and the Actor stops cleanly when it is reached.

Check the HTTP status of each URL adds $0.70 per 1,000 checked URLs, so $1.00 per 1,000 URLs in total when it is on. Only URLs that answered are charged: a timeout or an unreachable address costs nothing.

Tips

• Start small. Sites like airbnb.com or booking.com list hundreds of thousands of URLs or more: set Max URLs or Max URLs per website first.
• Target a section with sitemapUrlPatterns (*blog*) on large sites: whole parts of the index are then never downloaded.
• Use modifiedSince for incremental crawls, or change tracking when the site does not set lastmod (many do not).
• Very large tracked websites need memory. With the default 1 GB, a run keeps up to about 1 million URL entries in memory (URLs met in this run, plus the previous state when tracking changes). Past that it stops cleanly and says so: give the run more memory in its options.
• A domain is enough. Enter a sitemap URL only when the sitemap is neither in robots.txt nor at a usual address.

FAQ

Which sitemaps are supported? XML sitemaps and sitemap indexes (nested up to 5 levels), gzip-compressed files, plain-text sitemaps (one URL per line), the image, video and news extensions, and RSS 2.0, RSS 1.0 and Atom feeds (which the sitemap protocol accepts too): enter a feed URL, or let robots.txt declare it.

A website came back without URLs. Why? Check its status in the run summary. no_sitemap means robots.txt declares no sitemap and none sits at the usual addresses: enter the sitemap URL yourself if you know it. blocked means the website answered with a bot challenge instead of its files, which some sites do to requests from cloud servers. failed means it did not answer. not_reached means a limit stopped the run first. None of these fail the run, except when no website at all could be reached.

Is it legal? Sitemaps are public files that websites publish so that crawlers can find their pages. This Actor reads only robots.txt and those files, at a gentle pace, and never the pages. It collects no personal data. You remain responsible for how you use the list, and for respecting each website's terms if you crawl the pages next.

What are the limits?

• Each sitemap file is read up to 100 MB (compressed size) and 200 MB once decompressed; the sitemap protocol itself caps files at 50 MB.
• Sitemap indexes are followed up to 5 levels deep, and up to Max sitemap files per website files (1,000 by default).
• Websites that answer cloud servers with a bot challenge are reported as blocked. The Actor does not try to get around it.
• Sitemaps that only exist after running JavaScript, and HTML sitemap pages, are not read.
• lastmod, changefreq and priority are what the website declares; many sites leave them out or never update them.

Does it check that each URL still works? Only if you ask. By default it lists what the website declares, which keeps runs fast and cheap. Turn on Check the HTTP status of each URL to get statusCode for every saved URL: a HEAD request each, redirects reported (301 with redirectUrl) rather than followed, and a GET only for servers that do not support HEAD. It is slower and costs $0.70 per 1,000 checked URLs on top of the URL itself — only URLs that answered are charged. One website is asked at most 4 URLs at a time, and never more than 10 a second. Use the URL status view of the Output tab to review the results.

Support

Found a bug or a sitemap that is not read correctly? Open an issue on the Issues tab with the website and the run link, and I will look at it.