# Sitemap URL Extractor: robots.txt, gzip, Changes (`succinct_glider/sitemap-urls`) Actor Extract every URL from a website's XML sitemaps: found through robots.txt, sitemap indexes and .gz files followed, lastmod and date filters, URL patterns, and new, changed or removed URLs since your last run. Reads sitemap files only, never the pages. - **URL**: https://apify.com/succinct\_glider/sitemap-urls.md - **Developed by:** [Alexandre Bobichon](https://apify.com/succinct_glider) (community) - **Categories:** SEO tools, Developer tools, AI - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.30 / 1,000 urls This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README **Extract every URL of a website from its XML sitemaps** — with `lastmod`, change frequency and priority — by entering just a **domain**. Sitemaps are found through **robots.txt**, sitemap indexes and **gzip files** are followed, and only the sitemap files are read, never the pages. **$0.30 per 1,000 URLs**, no start fee. Run it on a schedule with change tracking to get the **new, changed and removed URLs** since your last run. ### What does this sitemap extractor do? Give it `stripe.com` and it returns the URLs listed in Stripe's sitemaps. Behind that one line: 1. It reads `robots.txt` and takes the sitemaps declared there. Many sites keep theirs at an unusual address (`stripe.com/sitemap/sitemap.xml`), where tools that only try `/sitemap.xml` find nothing. 2. When `robots.txt` lists none, it tries `/sitemap.xml`, `/sitemap_index.xml` and `/sitemap-index.xml`. 3. It follows sitemap indexes, decompresses `.xml.gz` files and accepts whatever content type the server sends (`binary/octet-stream` included, which is how booking.com serves its sitemaps). 4. It returns one clean item per URL, deduplicated across all the sitemaps of the website. It never requests the pages themselves, so a large website costs a handful of file downloads, not thousands of page loads. No browser, no login, no proxy needed. ### Why this sitemap extractor? - **Built for large websites.** It downloads the sitemap files and nothing else: no request per listed page, no browser. A site with hundreds of thousands of URLs is read in seconds to minutes, for the price of the URLs you keep. - **Finds sitemaps other tools miss.** Declarations in robots.txt, nested indexes, `.xml.gz` files, and servers that send sitemaps as `binary/octet-stream` or `application/gzip` are all handled. - **Clear about what went wrong.** A website without a sitemap, or one that blocks cloud servers, is reported with its reason in the run summary instead of failing the whole run. - **Pay only for what you keep.** Filters, duplicates and, with change tracking, unchanged URLs are free. ### Why extract URLs from sitemaps? - **Crawl and RAG pipelines.** Get the full list of pages before crawling, and re-crawl only what changed with **Modified since** or **Only save new, changed and removed URLs**. - **SEO audits.** Check what a site declares to search engines: missing `lastmod`, stale sections, language alternates (`hreflang`), image and video sitemaps. - **Competitive monitoring.** Schedule a daily run on competitors and receive their new pages: products, blog posts, landing pages, job pages. - **Content inventory and migrations.** Export every URL of a site to a spreadsheet in minutes. ### How to extract all URLs of a website 1. Enter one or more websites in **Websites or sitemaps**: a domain (`example.com`), any page of the site, or a sitemap URL (`https://example.com/sitemap.xml`). 2. Set **Max URLs** (500 by default) to control the cost of the run. 3. Optionally add URL patterns (`*/blog/*`), a **Modified since** date, or turn on **Track changes**. 4. Click **Start**, then open the **Output** tab and export as JSON, CSV, Excel or XML. #### Examples of input Only the blog posts published in the last week: ```json { "startUrls": ["example.com"], "includeUrlPatterns": ["*/blog/*"], "modifiedSince": "7 days", "maxItems": 1000 } ``` A daily watch on competitors that returns only what changed: ```json { "startUrls": ["competitor-one.com", "competitor-two.com"], "onlyChanges": true, "maxItems": 20000, "stateStoreName": "competitor-watch" } ``` ### Input | Field | Default | What it does | | --------------------- | -------------------- | -------------------------------------------------------------------------------------------------- | | `startUrls` | — | Domains, pages or sitemap URLs, one per line. | | `maxItems` | 500 | URLs saved in this run, across all websites. The run stops as soon as it is reached. | | `maxItemsPerSite` | 0 | Limit per website, so one large site cannot use the whole budget. 0 means no limit. | | `includeUrlPatterns` | — | Keep only matching URLs. `*` matches anything; a pattern without `*` matches URLs that contain it. | | `excludeUrlPatterns` | — | Skip matching URLs. | | `modifiedSince` | — | Keep URLs whose `lastmod` is on or after a date (`2026-09-01`) or a duration (`7 days`). | | `requireLastmod` | false | With a date, also drop URLs that have no `lastmod`. | | `sitemapUrlPatterns` | — | On large sites, follow only the child sitemaps of an index that match (`*blog*`). | | `trackChanges` | false | Compare with the previous run and add `changeStatus`. | | `onlyChanges` | false | Save only new, changed and removed URLs. | | `stateStoreName` | `sitemap-urls-state` | Key-value store that keeps the tracked state. One name per list you track. | | `includeAlternates` | false | Add the `hreflang` language alternates of each URL. | | `checkUrlStatus` | false | Send a HEAD request to each saved URL and add its HTTP status and redirect target. | | `useDefaultLocations` | true | Try the usual sitemap addresses when robots.txt lists none. | | `maxSitemapsPerSite` | 1000 | Safety limit on sitemap files read per website. | **Filters are applied before saving**, so URLs you filter out are never charged. ### Output One item per URL. Download it as JSON, CSV, Excel or XML, or read it from the API. ```json { "url": "https://www.notion.com/blog/notion-ai-for-work", "site": "notion.com", "lastmod": "2026-09-14T02:36:13.300Z", "changefreq": "yearly", "priority": 0.4, "sitemapUrl": "https://www.notion.com/blog/sitemap.xml", "discoveredFrom": "robots-txt", "imageCount": 0, "videoCount": 0, "newsTitle": null, "newsPublishedAt": null, "alternates": null, "changeStatus": "changed", "previousLastmod": "2026-08-02T09:12:40.000Z", "previousScrapedAt": "2026-09-16T06:00:04.117Z", "scrapedAt": "2026-09-17T06:00:03.905Z" } ``` #### Fields | Field | Description | | ------------------------------------------------------ | ------------------------------------------------------------------------------ | | `url` | The page URL, as listed in the sitemap. | | `site` | The website it belongs to (host without `www.`). | | `lastmod`, `changefreq`, `priority` | As declared in the sitemap. Dates are ISO 8601 UTC; invalid values are `null`. | | `sitemapUrl` | The sitemap file the URL was read from. | | `discoveredFrom` | How that sitemap was found: `input`, `robots-txt` or `default-location`. | | `imageCount`, `videoCount` | Images and videos declared for the page (image and video sitemaps). | | `newsTitle`, `newsPublishedAt` | Filled for Google News sitemaps. | | `alternates` | `hreflang` alternates, when you ask for them. | | `statusCode`, `redirectUrl`, `statusError` | With **Check the HTTP status**: the status, the redirect target, or the error. | | `changeStatus`, `previousLastmod`, `previousScrapedAt` | Change tracking: `new`, `changed`, `unchanged` or `removed`. | | `scrapedAt` | When the sitemap was read. | A field the sitemap does not provide is `null`, never missing. Field names and types do not change outside a major version. #### Run summary The `SUMMARY` record of the key-value store (**Run summary** in the Output tab) tells, for each website, what was read and why something is missing. Statuses: `ok`, `partial` (a limit or a failing sitemap left part of it unread), `no_sitemap`, `not_reached`, `blocked`, `failed`. Example with a website read completely and one that blocks cloud servers: ```json { "sitesRequested": 2, "sitesOk": 1, "urlsSaved": 3978, "sites": [ { "input": "docs.apify.com", "site": "docs.apify.com", "status": "ok", "sitemapsFound": 7, "sitemapsRead": 7, "sitemapsFailed": 0, "urlsFound": 3986, "urlsSaved": 3978, "urlsFilteredOut": 0, "urlsRemoved": 0, "discoveredFrom": ["robots-txt"], "notes": [] }, { "input": "www.booking.com", "site": "booking.com", "status": "blocked", "sitemapsFound": 0, "sitemapsRead": 0, "sitemapsFailed": 0, "urlsFound": 0, "urlsSaved": 0, "urlsFilteredOut": 0, "urlsRemoved": 0, "discoveredFrom": [], "notes": ["robots.txt: bot challenge instead of the file (HTTP 202, text/html, 2008 bytes)."] } ], "invalidInputs": [], "duplicateInputs": 0, "maxItemsReached": false, "budgetReached": false, "memoryLimitReached": false, "changes": null } ``` `urlsFound` counts the entries listed in the sitemaps read; `urlsSaved` is lower when the same URL appears in several sitemaps, since each URL is saved once. ### Tracking changes Turn on **Track changes** and every URL gets a `changeStatus`: - `new` — not in the website's sitemaps last run. - `changed` — its `lastmod` moved. Compared only when both runs have a date. - `unchanged` — still listed, same date. - `removed` — listed last run, gone now. Reported only when the website was read **completely** in both runs, so an outage or a limit can never look like a wave of removals. Add **Only save new, changed and removed URLs** to skip the unchanged ones, which are then not charged. A run cut short by a limit keeps what it did not reach for next time. Changing the URL patterns starts the comparison from scratch. Use one `stateStoreName` per list you track. ### Use it from the API, integrations or an AI agent Every run is available through the [Apify API](https://docs.apify.com/api/v2), so the Actor works as a **sitemap API** for your own code: ```bash curl -X POST "https://api.apify.com/v2/acts/succinct_glider~sitemap-urls/run-sync-get-dataset-items?token=" \ -H "Content-Type: application/json" \ -d '{"startUrls": ["example.com"], "maxItems": 1000}' ``` Schedule it and send the results to Google Sheets, Slack, a webhook, Make, Zapier or n8n through the built-in integrations. AI agents connected to the [Apify MCP server](https://mcp.apify.com) can call it to list the pages of a website. ### How much does it cost to extract sitemap URLs? **$0.30 per 1,000 URLs saved ($0.0003 per URL). No start fee.** URLs removed by your filters, duplicates and, with **Only save new, changed and removed URLs**, unchanged URLs are not charged. Set a **maximum cost per run** in the run options and the Actor stops cleanly when it is reached. **Check the HTTP status of each URL** adds $0.70 per 1,000 checked URLs, so $1.00 per 1,000 URLs in total when it is on. Only URLs that answered are charged: a timeout or an unreachable address costs nothing. ### Tips - **Start small.** Sites like airbnb.com or booking.com list hundreds of thousands of URLs or more: set **Max URLs** or **Max URLs per website** first. - **Target a section** with `sitemapUrlPatterns` (`*blog*`) on large sites: whole parts of the index are then never downloaded. - **Use `modifiedSince` for incremental crawls**, or change tracking when the site does not set `lastmod` (many do not). - **Very large tracked websites need memory.** With the default 1 GB, a run keeps up to about 1 million URL entries in memory (URLs met in this run, plus the previous state when tracking changes). Past that it stops cleanly and says so: give the run more memory in its options. - **A domain is enough.** Enter a sitemap URL only when the sitemap is neither in robots.txt nor at a usual address. ### FAQ **Which sitemaps are supported?** XML sitemaps and sitemap indexes (nested up to 5 levels), gzip-compressed files, plain-text sitemaps (one URL per line), the image, video and news extensions, and RSS 2.0, RSS 1.0 and Atom feeds (which the sitemap protocol accepts too): enter a feed URL, or let robots.txt declare it. **A website came back without URLs. Why?** Check its status in the run summary. `no_sitemap` means robots.txt declares no sitemap and none sits at the usual addresses: enter the sitemap URL yourself if you know it. `blocked` means the website answered with a bot challenge instead of its files, which some sites do to requests from cloud servers. `failed` means it did not answer. `not_reached` means a limit stopped the run first. None of these fail the run, except when no website at all could be reached. **Is it legal?** Sitemaps are public files that websites publish so that crawlers can find their pages. This Actor reads only `robots.txt` and those files, at a gentle pace, and never the pages. It collects no personal data. You remain responsible for how you use the list, and for respecting each website's terms if you crawl the pages next. **What are the limits?** - Each sitemap file is read up to 100 MB (compressed size) and 200 MB once decompressed; the sitemap protocol itself caps files at 50 MB. - Sitemap indexes are followed up to 5 levels deep, and up to **Max sitemap files per website** files (1,000 by default). - Websites that answer cloud servers with a bot challenge are reported as `blocked`. The Actor does not try to get around it. - Sitemaps that only exist after running JavaScript, and HTML sitemap pages, are not read. - `lastmod`, `changefreq` and `priority` are what the website declares; many sites leave them out or never update them. **Does it check that each URL still works?** Only if you ask. By default it lists what the website declares, which keeps runs fast and cheap. Turn on **Check the HTTP status of each URL** to get `statusCode` for every saved URL: a HEAD request each, redirects reported (`301` with `redirectUrl`) rather than followed, and a GET only for servers that do not support HEAD. It is slower and costs $0.70 per 1,000 checked URLs on top of the URL itself — only URLs that answered are charged. One website is asked at most 4 URLs at a time, and never more than 10 a second. Use the **URL status** view of the Output tab to review the results. ### Support Found a bug or a sitemap that is not read correctly? Open an issue on the **Issues** tab with the website and the run link, and I will look at it. # Changelog This Actor's version history is a separate document: https://apify.com/succinct\_glider/sitemap-urls/changelog.md # Actor input Schema ## `startUrls` (type: `array`): One entry per line: a domain (example.com), a page of the site (https://example.com/blog), or the URL of a sitemap or sitemap index (https://example.com/sitemap.xml, .xml.gz). For a domain or page, the sitemaps listed in robots.txt are used, then the usual locations such as /sitemap.xml. ## `maxItems` (type: `integer`): Maximum number of URLs saved in this run, across all websites. The run stops reading sitemaps as soon as it is reached. ## `maxItemsPerSite` (type: `integer`): Limit per website, so one large site cannot use the whole budget. 0 means no limit. ## `includeUrlPatterns` (type: `array`): Keep only URLs that match one of these patterns. `*` matches any characters, e.g. `*/blog/*` or `https://example.com/products/*`; a pattern without `*` matches URLs that contain it. Matching is case-insensitive. Filtering happens before saving, so filtered-out URLs are never charged. ## `excludeUrlPatterns` (type: `array`): Skip URLs that match one of these patterns, e.g. `*/tag/*` or `*?page=*`. Applied after the include patterns. ## `modifiedSince` (type: `string`): Keep only URLs whose sitemap `lastmod` is on or after this date. Absolute (2026-09-01) or relative (7 days). URLs without a `lastmod` are kept unless you enable the option below. ## `requireLastmod` (type: `boolean`): With a date filter, drop URLs whose sitemap entry has no `lastmod`. Many sites never set it. ## `sitemapUrlPatterns` (type: `array`): On large sites, follow only the child sitemaps of an index whose URL matches one of these patterns, e.g. `*blog*` or `*products*`. The sitemaps you list yourself are always read. ## `trackChanges` (type: `boolean`): Compare each website with its previous tracked run and add `changeStatus` (new, changed, unchanged, removed). Removed URLs are reported only when the website was read completely in both runs. ## `onlyChanges` (type: `boolean`): Skip unchanged URLs: they are not saved, so you are not charged for them. Enables change tracking automatically. The first run saves everything. ## `stateStoreName` (type: `string`): Named key-value store of your account that keeps the previous state of each website. Use a different name for each independent list you track. Letters, digits and hyphens only. ## `checkUrlStatus` (type: `boolean`): Send a HEAD request to every saved URL and add its HTTP status (`statusCode`) and redirect target (`redirectUrl`), without following redirects. Slower, and charged as an extra event per checked URL (see the Pricing tab). Off by default: most uses only need the list. ## `includeAlternates` (type: `boolean`): Add the `hreflang` alternate URLs declared for each entry (`xhtml:link`). They can make items much larger on multilingual sites. ## `useDefaultLocations` (type: `boolean`): When robots.txt lists no sitemap, also try /sitemap.xml, /sitemap\_index.xml and /sitemap-index.xml. ## `maxSitemapsPerSite` (type: `integer`): Safety limit on the number of sitemap files read for one website, including nested indexes. ## `proxyConfiguration` (type: `object`): Optional. Sitemaps are public files meant for crawlers and usually need no proxy. ## Actor input object example ```json { "startUrls": [ "docs.apify.com" ], "maxItems": 500, "maxItemsPerSite": 0, "requireLastmod": false, "trackChanges": false, "onlyChanges": false, "stateStoreName": "sitemap-urls-state", "checkUrlStatus": false, "includeAlternates": false, "useDefaultLocations": true, "maxSitemapsPerSite": 1000, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `results` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "docs.apify.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("succinct_glider/sitemap-urls").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["docs.apify.com"] } # Run the Actor and wait for it to finish run = client.actor("succinct_glider/sitemap-urls").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "docs.apify.com" ] }' | apify call succinct_glider/sitemap-urls --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,succinct_glider/sitemap-urls" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/sgOi2QDZ2WXxQdLci/builds/zHYnzbdSl9Cgo82aJ/openapi.json