# Sitemap URL Extractor: All URLs from XML Sitemaps & Changes (`humble-echidna/sitemap-urls`) Actor Extract all URLs from a website's XML sitemaps: finds them via robots.txt, follows sitemap indexes, reads .xml.gz. Returns lastmod, changefreq, priority, image, news, video and hreflang data. Filter by URL pattern and date, or monitor sitemap changes: only new and removed URLs since the last run. - **URL**: https://apify.com/humble-echidna/sitemap-urls.md - **Developed by:** [Michael Costa](https://apify.com/humble-echidna) (community) - **Categories:** SEO tools, Developer tools, Automation - **Stats:** 1 total users, 0 monthly users, 90.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.10 / 1,000 urls This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ### What does Sitemap URL Extractor do? **Sitemap URL Extractor** gets **all the URLs of a website from its XML sitemaps**. Give it a domain or a sitemap address; you get one row per URL with its **last-modified date**, change frequency, priority, and the image, news, video and hreflang entries the site publishes. It finds the sitemaps the way search engines do (robots.txt `Sitemap:` lines, then `/sitemap.xml` and `/sitemap_index.xml`), follows sitemap indexes, reads gzipped and plain-text sitemaps, and filters by URL pattern and last-modified date. On a schedule, **Only new and removed URLs** turns it into a sitemap change monitor: each run returns just the URLs a site added to or dropped from its sitemaps since the last run. It reads sitemaps only; it doesn't crawl pages. [Input fields](https://apify.com/humble-echidna/sitemap-urls/input-schema) · [API](https://apify.com/humble-echidna/sitemap-urls/api) · [Use it from Claude, ChatGPT or Cursor (MCP)](#can-i-use-sitemap-url-extractor-from-an-ai-agent-mcp) **Jump to:** [Monitor a sitemap](#monitor-a-sites-sitemap-new-and-removed-urls-since-the-last-run) · [Fields](#what-data-does-sitemap-url-extractor-return) · [Price](#how-much-does-it-cost-to-extract-sitemap-urls) · [How to use](#how-to-extract-all-urls-from-a-sitemap) · [Input](#input) · [Switching from another actor](#switching-from-another-sitemap-actor) · [Output](#output) · [Chain after another actor](#chain-it-after-another-actor) · [AI agents (MCP)](#can-i-use-sitemap-url-extractor-from-an-ai-agent-mcp) · [Limits](#limits) · [FAQ](#faq) **Try it in one click:** the input comes pre-filled with `sitemaps.org`, the home of the sitemap standard. That's 84 URLs, about $0.025 (84 × $0.0003, plus $0.00005 for the run start). **Then replace it with the sites you want.** ### Monitor a site's sitemap: new and removed URLs since the last run Turn on **Only new and removed URLs since the last run** (`"onlyNewAndRemovedUrls": true`) and schedule it: each run compares the sitemaps with the previous run of the same input and returns only the URLs that are **new** (not listed last time) or **removed** (listed last time, gone now), marked in `changeType`. It's a true comparison of the lists, so it works on sites whose `lastmod` dates are missing or wrong. URLs listed both times are counted, not returned and not charged, so a run where nothing changed costs only the $0.00005 start fee. 1. Put the sites in **Websites or sitemap URLs**, optionally narrow them with **Only URLs matching** (`*/blog/*` to watch a competitor's blog), turn on **Only new and removed URLs since the last run**, and click **Start**. The first run returns every URL (`changeType` `null`): that's the baseline the next runs compare with. 2. Click **Save as a new task** (top right of the actor page), then in **Schedules** click **Create new**, set how often (for example daily at 07:00) and **Add** the task. 3. On the task, open the **Integrations** tab to send each run's results to **Slack**, **Gmail** or a **webhook** (event `ACTOR.RUN.SUCCEEDED`; `resource.defaultDatasetId` is the run's new and removed URLs). A quiet run's dataset is empty and its status message says how many URLs were unchanged. What it compares, and when it holds back: - **Per site and settings.** Each line of input has its own list (in your account's key-value store `sitemap-urls-memory`), tied to its URL patterns and the sitemap depth and file limits: adding a site to the task doesn't reset the others, and changing a line's patterns starts a new baseline for that line. - **Removed only after a complete read.** A URL is reported removed only when every sitemap found for that line was read to the end. If a sitemap failed to download, robots.txt blocked one, the file or depth limit skipped some, a file had more than 50,000 entries, or the run stopped at its max results or max cost, no URL is reported removed and `removedNotReported` in `RUN_STATS` says why. New URLs are still reported. - **A sudden drop waits one run.** If a clean read lists nothing, or more than half of the tracked URLs (at least 20\) disappear at once, it looks like a broken or replaced sitemap: the removals are held and reported only if the next run sees the same (`removedHeld` in `RUN_STATS`). - **Nothing lost to a limit.** A new or removed URL is remembered only once its row is in your dataset, so one cut by the max results or max cost per run is reported again next run. - It compares URLs only: a changed `lastmod`, priority or image on a URL that's still listed isn't reported. The pre-filled example, run twice with the option on (local runs, 2026-09-26): the first run returned all 84 URLs of sitemaps.org as the baseline; the second returned none and said "0 URLs returned: 0 new and 0 removed since the last run, 84 unchanged (not returned), from 1 site". ### What data does Sitemap URL Extractor return? | Field | Example | Notes | |---|---|---| | `url` | `https://www.sitemaps.org/` | Each URL once per site, even if several sitemaps list it. | | `lastmod` | `2016-11-21` | Exactly what the sitemap says (a date or a timestamp); `null` if none. | | `changefreq` | `weekly` | `null` if the sitemap doesn't say. | | `priority` | `0.8` | `null` if the sitemap doesn't say. | | `images` | `[{"url": "https://www.example.com/img/launch.jpg", "title": null, ...}]` | Image sitemap entries. | | `news` | `null` | News sitemap entry: publication, date, title, keywords. | | `videos` | `[]` | Video sitemap entries: title, thumbnail, duration, dates. | | `alternates` | `[{"hreflang": "de", "url": "https://www.example.com/de/blog/launch"}]` | hreflang language alternates. | | `sitemapUrl` | `https://www.sitemaps.org/sitemap.xml` | The sitemap file that listed it. | | `site` | `sitemaps.org` | The input line it came from. | | `changeType` | `"new"` | With **Only new and removed URLs** on: `new` or `removed`; `null` on the first run and with it off. | | `sourceTitle`, `sourcePlaceId`, `sourceIndex` | `"Rosie's Bakery"`, `"ChIJ..."`, `12` | From the dataset item the site came from (see [Chain it after another actor](#chain-it-after-another-actor)); `null` for sites you type in, and `sourcePlaceId` when the item has no `placeId`. | The full list is under [Output](#output). ### How much does it cost to extract sitemap URLs? You pay per URL returned: **$0.30 per 1,000 URLs**, plus $0.00005 each time a run starts. URLs your filters drop are never charged, and neither is a URL listed twice. With **Only new and removed URLs** on, you pay for the new and removed URLs returned, at the same price; URLs listed both times are free. **It's cheaper on paid Apify plans:** $0.25 per 1,000 URLs on Starter, $0.18 on Scale and $0.10 on Business. The prices on this page are the Free-plan price, so on a paid plan you pay less than the examples show. - **The example below:** 84 URLs × $0.0003 = $0.0252, plus the $0.00005 start fee: about $0.025. - **A month, for example:** a weekly full export of a 10,000-URL site is 4 × 10,000 × $0.0003 = **$12.00**; a daily run with "Last modified on or after" set to `1 day`, returning about 50 changed URLs a day, is 1,500 × $0.0003 = **$0.45** plus 30 starts ($0.0015). - **Monitoring, for example:** a daily watch on a 10,000-URL site with **Only new and removed URLs**: the first run returns all 10,000 URLs ($3.00); then about 20 new or removed URLs a day is 30 × 20 × $0.0003 = **$0.18 a month** plus 30 starts ($0.0015). A day with no change costs the $0.00005 start fee. - **Caps:** **Max results per run** in the input, and **Maximum cost per run** in the run options. The run stops cleanly at whichever comes first. It checks the limit before fetching each sitemap, so a capped run stops downloading as soon as the limit is covered. ### How to extract all URLs from a sitemap 1. Open Sitemap URL Extractor and click **Try for free** (or **Start** if you're signed in). 2. Put your sites in **Websites or sitemap URLs**, one per line: a domain (`example.com`) or a sitemap's address. 3. Optional: keep only some URLs with **Only URLs matching** (`*/blog/*`) or **Skip URLs matching**, or set a date window with **Last modified on or after** (`7 days`, `2026-01-01`). 4. Click **Start**, then open the **Output** tab and export as JSON, CSV or Excel. ### Example: sitemaps.org The pre-filled input: ```json {"sites": ["sitemaps.org"], "maxSitemapsPerSite": 200, "maxDepth": 3} ``` The first of the 84 URLs it returned (real output from a local run on 2026-09-25): ```json { "url": "https://www.sitemaps.org/", "lastmod": "2016-11-21", "changefreq": null, "priority": null, "images": [], "news": null, "videos": [], "alternates": [], "sitemapUrl": "https://www.sitemaps.org/sitemap.xml", "site": "sitemaps.org", "scrapedAt": "2026-09-25T17:45:17.769649Z" } ``` ### Ready-to-run examples Each example opens with the input already filled in. Run it as it is, or change the input first. - **[Extract all URLs from a website's sitemap](https://apify.com/humble-echidna/sitemap-urls/examples/extract-all-sitemap-urls)**: Give it a domain and get every URL its sitemaps list, with last-modified dates, found through robots.txt and nested sitemap indexes. - **[Monitor a sitemap for new and removed pages](https://apify.com/humble-echidna/sitemap-urls/examples/monitor-sitemap-new-removed)**: Run on a schedule and get only the URLs added to or removed from a site's sitemap since the last run. ### Input | Field | What it does | |---|---| | **Websites or sitemap URLs** | One per line: a domain or site address (the actor finds its sitemaps), or a sitemap's address (`.xml`, `.xml.gz`, `.txt`, or a sitemap index), which is read directly. Ignored with a dataset. | | Or: websites from a dataset (`datasetId`) | One of your Apify datasets, e.g. a Google Maps scraper's results or another actor's run in an integration (`{{resource.defaultDatasetId}}`): each item's website is read as a site. See [Chain it after another actor](#chain-it-after-another-actor). | | Field with the website (`datasetUrlField`) | Only with a dataset. Empty (default): found automatically among `website`, `url`, `domain`, `site` and `homepage`, then one level down (`contact.website`). | | Only URLs matching | Keep URLs matching any of these patterns. `*` matches anything; a pattern without `*` matches anywhere in the URL. Case-insensitive. | | Skip URLs matching | Drop URLs matching any of these patterns. | | Last modified on or after / before | A date window on each URL's ``: `2026-01-31`, a full timestamp, or relative (`7 days`, `2 weeks`, `3 months`). "Before" is exclusive. | | Keep URLs without a lastmod date | With a date filter set, URLs whose sitemap gives no date are dropped unless this is on. | | Only new and removed URLs since the last run | Off by default. On: return only URLs not listed on the previous run of the same input (`new`) and URLs no longer listed (`removed`). The first run returns every URL. Can't be combined with the lastmod date filters. See [Monitor a site's sitemap](#monitor-a-sites-sitemap-new-and-removed-urls-since-the-last-run). | | Max results per run | Cap the total number of URLs across all sites. | | Max sitemap files per site | The most sitemap files (indexes included) read per input line. Default 200. | | Max sitemap index depth | How many levels of nested indexes to follow. Default 3. | ```json { "sites": ["example.com", "https://www.example.org/sitemap_index.xml"], "includeUrlPatterns": ["*/blog/*"], "lastmodAfter": "30 days", "maxResults": 10000 } ``` ### Switching from another sitemap actor? The inputs other sitemap actors use map onto Sitemap URL Extractor like this. An API call or integration that sends `startUrls` and `maxUrls` works as it is: change only the actor id. | If you used | In Sitemap URL Extractor | Notes | |---|---|---| | `startUrls` (strings or `{"url": ...}` objects) | `sites` | `startUrls` is read too. A domain, a site address or a sitemap URL (`.xml`, `.xml.gz`, `.txt`, an index). | | `maxUrls` (0 = no limit) | `maxResults` (empty = no limit) | `maxUrls` is read too. | | Include / exclude regular expressions | `includeUrlPatterns` / `excludeUrlPatterns` | Wildcards, one per line: `.*` becomes `*` (`https://example.com/blog/.*` is `https://example.com/blog/*`), and a pattern without `*` matches anywhere in the URL, so `/blog/` alone works. A regex field from another actor (`urlFilterInclude`, `urlFilterExclude`) stops the run with this hint before anything is fetched or charged. | | Follow sitemap indexes | Always on | **Max sitemap index depth** (`maxDepth`) 0 reads only the sitemaps you gave or robots.txt lists. | | Max crawling depth, 0 = unlimited | `maxDepth`, 0 to 10, default 3 | Here 0 means "don't follow indexes"; 10 covers any real site. | | Check each page's status | [Bulk URL & Broken Link Checker](https://apify.com/humble-echidna/url-checker) | Its `sitemap` mode reads the sitemap and returns each URL's status code, redirects and final URL in one run (see the FAQ below). | | Proxy configuration | Not needed | Sitemaps are read directly, the way search engines read them. A site that refuses unknown crawlers is reported, not worked around. | The output keeps the usual names: `url`, `lastmod`, `changefreq` and `priority`, plus `images`, `videos`, `news` and `alternates` (hreflang) as lists, and `sitemapUrl` (the file the URL came from) and `site` (your input line). ### Output One row per URL. Fields the sitemap doesn't publish are `null` (or an empty list). ```json { "url": "https://www.example.com/blog/launch", "lastmod": "2026-09-01T10:00:00+00:00", "changefreq": "weekly", "priority": 0.8, "sitemapUrl": "https://www.example.com/post-sitemap.xml.gz", "site": "example.com", "images": [{"url": "https://www.example.com/img/launch.jpg", "title": null, "caption": null, "geoLocation": null, "license": null}], "news": null, "videos": [], "alternates": [{"hreflang": "de", "url": "https://www.example.com/de/blog/launch"}], "changeType": null, "scrapedAt": "2026-09-24T12:00:00Z", "sourceTitle": null, "sourcePlaceId": null, "sourceIndex": null } ``` A removed URL (with **Only new and removed URLs** on) has only its address, the sitemap that last listed it and the input line; the rest is empty, because its sitemap entry is gone: ```json {"url": "https://www.example.com/blog/old-post", "lastmod": null, "changefreq": null, "priority": null, "sitemapUrl": "https://www.example.com/post-sitemap.xml.gz", "site": "example.com", "images": [], "news": null, "videos": [], "alternates": [], "changeType": "removed", "scrapedAt": "2026-09-25T07:00:00Z", "sourceTitle": null, "sourcePlaceId": null, "sourceIndex": null} ``` - `lastmod` is exactly what the sitemap says (a date or a timestamp). - `news`: `publicationName`, `publicationLanguage`, `publicationDate`, `title`, `keywords`, `genres`. - `videos`: `thumbnailUrl`, `title`, `description`, `contentUrl`, `playerUrl`, `durationSeconds`, `publicationDate`, `expirationDate`, `familyFriendly`, `tags`. - `site` is the input line the URL came from, so results from several sites are easy to split (with a dataset, the website's home page, e.g. `https://www.example.com`). - `sourceTitle`, `sourcePlaceId`, `sourceIndex`: the dataset item the site came from; `null` for sites typed in. - `changeType`: `new` or `removed` with **Only new and removed URLs** on; `null` otherwise, and on the first (baseline) run. The `RUN_STATS` record has per-site numbers: sitemaps read (`sitemapsRead`), failed (`sitemapsFailed`) and skipped (`sitemapsSkipped`, and why: robots.txt, depth or file limit), URLs listed, filtered out, duplicated and returned (`emitted`), and the first errors. It has the same keys on every run, including failed ones. With **Only new and removed URLs** on, each site also has `monitor`: whether this was its baseline, how many URLs were tracked, `new`, `removed` and `unchanged`, and `removedNotReported` / `removedHeld` when removals were held back (and why); `changes` totals new and removed. With a dataset, `dataset` says which field was read and how many items were skipped, and why. ### Chain it after another actor To list the sitemap URLs of the websites another actor found as soon as it finishes (a Google Maps scraper's businesses, a list of companies, our [Website Technology Detector](https://apify.com/humble-echidna/tech-stack-detector); any actor, ours or not), chain it with an integration: 1. Open the other actor or its saved task, go to the **Integrations** tab and add **Apify actor** → **Sitemap URL Extractor**, triggered when a run **succeeds**. 2. Set `"datasetId": "{{resource.defaultDatasetId}}"` in the input (Apify fills in the finished run's dataset) and the rest as usual, e.g. `"includeUrlPatterns": ["*/blog/*"]`. **Websites or sitemap URLs** is ignored then. From the API it's the same input: pass a run's `defaultDatasetId` as `datasetId`. In the form, pick the dataset in **Or: websites from a dataset**. - **Which field:** the website is found automatically in `website`, `url`, `domain`, `site` or `homepage` (the first that holds one in the first 100 items), or one level down (`contact.website`). Otherwise set **Field with the website** (`datasetUrlField`). A Google Maps scraper's own `url` (the Maps link) is never used; its `website` is. - **One line per site.** A sitemap belongs to a whole site, so each website is read from its home page, whatever page the item links to (`https://bakery.com/locations/denver` is read as `https://bakery.com`, and that's its `site`). Several items on one website (branches of a chain, `www.` or not) read it once, with the first item's source fields. An item that gives a sitemap's own address (`.../sitemap.xml`) has it read directly. - **Options apply as usual:** URL patterns, date filters, depth and file limits, **Max results per run** and your maximum cost per run. - **Only new and removed URLs** remembers each site, not the dataset: a chained run reads a new dataset every time and still returns only what changed since the last run, and a site from a dataset shares its list with the same site typed in (`https://www.example.com` with `www.example.com`). Inputs without a dataset keep their lists from earlier versions. - **Joining back:** every URL carries `sourceTitle`, `sourcePlaceId` and `sourceIndex` (0 = the dataset's first item) from the item its site came from. - **Skipped, never fetched or charged:** items without a website, values that aren't a web address (a phone number, `N/A`) and Google Maps links; they're counted in `RUN_STATS.dataset`. A dataset with no website at all, or one the run can't read, fails the run with the reason. - **Limits:** the first 20,000 items and 10,000 distinct websites of a dataset per run, read with your own account's access, read-only. A URL costs the same whichever way its site came in; the other actor's run is billed by it. ### Run it on a schedule, or from your own code 1. Save your input as a [task](https://docs.apify.com/platform/actors/running/tasks) (**Save as a new task**, top right of the actor page) and add it to a [schedule](https://docs.apify.com/platform/schedules) (Console → Schedules → **Create new**): for example daily, with **Only new and removed URLs since the last run** on, for the pages the site added or dropped since yesterday (see [Monitor a site's sitemap](#monitor-a-sites-sitemap-new-and-removed-urls-since-the-last-run)). Or, with the option off, set **Last modified on or after** to `1 day` for the pages whose sitemap `lastmod` says they changed (this relies on the site keeping its `lastmod` dates accurate). 2. Collect results: download the dataset as JSON, CSV or Excel; fetch the latest run's results from the [API](https://apify.com/humble-echidna/sitemap-urls/api) (`GET https://api.apify.com/v2/actor-tasks//runs/last/dataset/items?status=SUCCEEDED&format=csv`, with your API token); let a [webhook](https://docs.apify.com/platform/integrations/webhooks) tell your system when a run succeeds; or connect it to Make, Zapier or n8n through [Apify's integrations](https://docs.apify.com/platform/integrations). The task's **Integrations** tab can also post to Slack or email you (Gmail) after each run. #### Can I use Sitemap URL Extractor from an AI agent (MCP)? Yes, through [Apify's MCP server](https://docs.apify.com/platform/integrations/mcp), from Claude, ChatGPT, Cursor or any other MCP client. Add this to your client's MCP configuration (or let the agent find it with the server's actor search); your client signs you in to Apify: ```json { "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=humble-echidna/sitemap-urls" } } } ``` To use an [Apify API token](https://console.apify.com/settings/integrations) instead of signing in, add `"headers": {"Authorization": "Bearer "}` next to `url`. The agent passes the site and a cap, e.g. `{"sites": ["example.com"], "includeUrlPatterns": ["*/docs/*"], "maxResults": 200}`. ### Who it's for SEO and content teams who need a site's complete, current URL list: to plan a crawl or an audit, to find pages missing from the sitemap, to spot recently changed pages by `lastmod`, or to feed URLs into another tool. And anyone watching a site over time (a competitor's new pages, a client's pages that disappeared from its sitemap). ### Why this one? - **One flat price, no usage bill.** You pay per result at the price above and nothing else: no compute, proxy or storage charges on top, apart from the $0.00005 run-start fee, so you know what a run costs before it starts. Results you don't get aren't charged. - **Complete discovery.** robots.txt `Sitemap:` lines first (every one of them), then the usual locations. Sitemap indexes are followed to any depth you allow, gzipped and plain-text sitemaps are read, and HTML "soft 404" pages served at `/sitemap.xml` are recognised and skipped instead of failing the run. - **All the sitemap data, not just the links.** `lastmod`, `changefreq`, `priority`, plus image, news, video and hreflang alternate entries as structured fields. - **Filters before you pay.** Keep only URLs matching your patterns (`*/blog/*`) or changed in a date window (`7 days`, `2026-01-01`). Filtered-out URLs are never charged. - **New and removed URLs, not a date guess.** The monitoring mode compares the actual lists from one run to the next, so it catches new pages on sites with missing or wrong `lastmod` dates, and removed pages, which a date filter can't see. It never reports URLs removed from a list it couldn't read completely, and a quiet run costs only the start fee. - **Each URL once.** A URL listed in several of a site's sitemaps is returned once; a sitemap linked from several indexes is read once. - **Polite and honest.** It honours robots.txt (Disallow and Crawl-delay, retries included; the file is read once per site per run) and `Retry-After`, identifies itself with its own User-Agent (`HumbleEchidnaApify/1.0 (+https://apify.com/humble-echidna)`; robots.txt rules for `HumbleEchidnaApify` apply to it), and never tries to get around a site that refuses it. - **Safe on hostile input.** Size caps (50 MB per sitemap, the standard's own limit, before and after decompression, counted while downloading), a time limit per download, a hardened XML parser (no DTDs or entity expansion) that reads each file as a stream so memory stays flat however big it is, and no requests to private or internal network addresses or non-standard ports, on every redirect. - **Reliable.** One failing site never affects the others in your run. The run log and the `RUN_STATS` record say exactly which site or sitemap had a problem and why. ### Limits - It reads sitemaps only; it doesn't crawl pages. A site without a sitemap returns an error saying so; if you know its sitemap's address, enter that directly. - Sites that block unknown crawlers (HTTP 401/403) or whose robots.txt disallows the sitemap are reported, not worked around. A robots.txt that answers with a server error counts as "disallow everything", as the robots.txt standard (RFC 9309) requires. - If a site asks for more than 30 seconds between requests (Crawl-delay), or more than 90 seconds before a retry (Retry-After; one such long wait per site, at most 3 minutes of them per run), it's reported with that reason instead of being read slowly. - A sitemap over 50 MB (before or after decompression) is skipped and reported. A sitemap with more than 50,000 entries (the standard's per-file limit) is read up to there, and that's reported (`sitemapsOverEntryLimit`). - Only public web addresses on the standard ports (80 for `http://`, 443 for `https://`) are read: a URL with another port such as `:8080`, or a name that resolves to a private network, is refused with the reason. A domain that doesn't exist is reported as such. - `example.com`, `https://example.com/` and `http://example.com` in the same input are one site, read and charged once. - RSS and Atom feeds are not read as sitemaps. - Monitoring (**Only new and removed URLs**) remembers up to 200,000 URLs per line of input; past that, further new URLs aren't reported and nothing is reported removed (split a bigger site with URL patterns, one line each). It can't be combined with the lastmod date filters: a URL leaving the date window would look removed. Two tasks with the same site and settings share one list, so run one of them, or give them different URL patterns. ### FAQ #### Why did a site return fewer URLs than I expected? Check `RUN_STATS`: large sites often split their sitemap over more than 200 files (raise "Max sitemap files per site"), or list some sitemaps under paths their own robots.txt disallows. #### Can I get only the URLs that are new since my last run? Yes: turn on **Only new and removed URLs since the last run** and schedule the task. Each run returns the URLs that weren't in the sitemaps last time (`changeType` `new`) and the ones that are gone (`removed`); the first run returns every URL as the baseline. Unchanged URLs aren't returned or charged. If you want URLs whose `lastmod` date is recent instead (including changed pages that were already listed), leave it off and set **Last modified on or after** to `1 day`. #### Why weren't removed URLs reported? Removed URLs are only reported when every sitemap for that site was read completely. Check `RUN_STATS`: the site's `monitor.removedNotReported` names what stopped it (a sitemap that failed or robots.txt blocked, the file or depth limit, the run's max results or max cost), and `monitor.removedHeld` says when a sudden big drop is waiting for the next run to confirm it. The URLs stay remembered, so they're reported on the first run that reads the site completely. #### How much does monitoring cost when nothing changes? The $0.00005 start fee: URLs that are listed both times aren't returned or charged. You pay $0.30 per 1,000 only for new and removed URLs, and for every URL on the first run. #### Something that used to work now fails or returns fewer results. Why? The site may have moved or changed its sitemaps. The run log names the site and sitemap and what went wrong, and every other site in the run is unaffected. Please open an issue with the input you used. #### Can it check each sitemap URL's HTTP status code (find 404s)? Not this actor: it reads sitemaps only, which is why it's fast and charges only for URLs. For status codes, run [Bulk URL & Broken Link Checker](https://apify.com/humble-echidna/url-checker) with **What to check** set to `sitemap` and your site: it reads the sitemap itself and checks every URL it lists (status code, every redirect hop, final and canonical URL, response time) in one run, at its own price per URL. Or chain the two: extract and filter the URLs here, then pass this run's dataset to the checker's `datasetId` with **What to check** set to `urls`. #### Is it legal to extract URLs from a sitemap? Sitemaps are published by site owners for crawlers to read. The actor reads only the sites you enter, obeys their robots.txt, uses no login, and collects no personal data. You are responsible for how you use the URLs. ### Related actors Sitemap URLs are often the first step of a bigger job: | Actor | Use it when | |---|---| | [Bulk URL & Broken Link Checker](https://apify.com/humble-echidna/url-checker) | You want every sitemap URL's status, redirects and canonical (it can read the sitemap itself in its sitemap mode). | | [SEO Audit Crawler](https://apify.com/humble-echidna/seo-audit) | You want the pages' on-page SEO audited; its sitemap mode reads the sitemap itself. | | [Website & Page to Markdown for RAG and LLMs](https://apify.com/humble-echidna/page-to-markdown) | You want the pages as Markdown; its whole-site mode reads the sitemap first. | ### Feedback and support Found a bug, or need a sitemap field that isn't here? Open an issue on the **Issues** tab with the input you used. ### Versions Current version: **1.3**. See the Changelog tab for what changed in each version. # Changelog This Actor's version history is a separate document: https://apify.com/humble-echidna/sitemap-urls/changelog.md # Actor input Schema ## `sites` (type: `array`): One per line. A domain or site address (example.com, https://www.example.com/) finds the site's sitemaps through its robots.txt, then /sitemap.xml and /sitemap_index.xml. A sitemap address (https://example.com/sitemap.xml, .xml.gz or a plain-text .txt sitemap) is read directly. Sitemap indexes are followed automatically. API callers may also send the list as `startUrls` (strings or {"url": ...} objects). Ignored when datasetId is set. ## `datasetId` (type: `string`): Default empty. One of your Apify datasets, picked here or given by id, e.g. a Google Maps scraper's results; to chain this actor after another in an integration, `{{resource.defaultDatasetId}}`. Each item's website becomes a line of sites (sites is ignored): its home page, or a sitemap URL as given. A site shared by several items is read once. Each URL carries sourceTitle, sourcePlaceId and sourceIndex from its item. Reads at most 20,000 items and 10,000 websites, read-only. ## `datasetUrlField` (type: `string`): Only with datasetId: the item field that holds the website, e.g. `website`, or a dotted path such as `contact.website`. Leave empty (the default) to find it automatically: the first of website, url, domain, site and homepage that has a website in the first 100 items, then the same names one level down (contact.website). Google Maps links are never used. ## `includeUrlPatterns` (type: `array`): Keep only URLs that match at least one of these patterns. \* matches anything (https://example.com/blog/\*); a pattern without \* matches anywhere in the URL (/blog/). Case-insensitive. Leave empty to keep every URL. ## `excludeUrlPatterns` (type: `array`): Drop URLs that match any of these patterns (same syntax as above), e.g. /tag/ or \*.pdf. Skipped URLs are not charged. ## `lastmodAfter` (type: `string`): Keep only URLs whose sitemap is on or after this date: 2026-01-31, a full timestamp (2026-01-31T12:00:00Z), or a relative time like 7 days, 2 weeks, 3 months. Dates without a time zone count as UTC. ## `lastmodBefore` (type: `string`): Keep only URLs whose sitemap is before this date (same formats as above). ## `keepUrlsWithoutLastmod` (type: `boolean`): When a date filter is set, URLs whose sitemap gives no (or an unreadable) are dropped. Turn this on to keep them. Has no effect without a date filter. ## `onlyNewAndRemovedUrls` (type: `boolean`): Monitor the sitemaps: return only URLs that weren't listed on the previous run of this same input (changeType new) and URLs that are no longer listed (changeType removed); URLs listed both times are counted, not returned or charged. The list is kept per site and settings in your own account's key-value store "sitemap-urls-memory". The first run returns every URL (the baseline). Removed URLs are only reported when every sitemap was read completely. Can't be combined with the lastmod date filters. ## `maxResults` (type: `integer`): Stop after this many URLs in total. Leave empty for no limit. The run also stops cleanly at the maximum cost per run you set in the run options, whichever comes first. Either way it stops fetching sitemaps, not just writing results. ## `maxSitemapsPerSite` (type: `integer`): The most sitemap files (indexes included) read for each line of input, 1 to 10,000 (default 200). Very large sites split their URLs over hundreds of files; raise this to get all of them. ## `maxDepth` (type: `integer`): How many levels of nested sitemap indexes to follow, 0 to 10 (default 3). 0 reads only the sitemaps you gave or that robots.txt lists. Most sites need 1; 3 covers indexes of indexes. ## Actor input object example ```json { "sites": [ "sitemaps.org" ], "keepUrlsWithoutLastmod": false, "onlyNewAndRemovedUrls": false, "maxSitemapsPerSite": 200, "maxDepth": 3 } ``` # Actor output Schema ## `results` (type: `string`): No description ## `runStats` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "sites": [ "sitemaps.org" ] }; // Run the Actor and wait for it to finish const run = await client.actor("humble-echidna/sitemap-urls").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "sites": ["sitemaps.org"] } # Run the Actor and wait for it to finish run = client.actor("humble-echidna/sitemap-urls").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "sites": [ "sitemaps.org" ] }' | apify call humble-echidna/sitemap-urls --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,humble-echidna/sitemap-urls" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/JKEPZIqHoTnv3YaJ8/builds/PCjRrANFXaeWVjRGx/openapi.json