Sitemap URL Extractor: All URLs from XML Sitemaps & Changes

What does Sitemap URL Extractor do?

Sitemap URL Extractor gets all the URLs of a website from its XML sitemaps. Give it a domain or a sitemap address; you get one row per URL with its last-modified date, change frequency, priority, and the image, news, video and hreflang entries the site publishes.

It finds the sitemaps the way search engines do (robots.txt Sitemap: lines, then /sitemap.xml and /sitemap_index.xml), follows sitemap indexes, reads gzipped and plain-text sitemaps, and filters by URL pattern and last-modified date. On a schedule, Only new and removed URLs turns it into a sitemap change monitor: each run returns just the URLs a site added to or dropped from its sitemaps since the last run. It reads sitemaps only; it doesn't crawl pages.

Input fields · API · Use it from Claude, ChatGPT or Cursor (MCP)

Jump to: Monitor a sitemap · Fields · Price · How to use · Input · Switching from another actor · Output · Chain after another actor · AI agents (MCP) · Limits · FAQ

Try it in one click: the input comes pre-filled with sitemaps.org, the home of the sitemap standard. That's 84 URLs, about $0.025 (84 × $0.0003, plus $0.00005 for the run start). Then replace it with the sites you want.

Monitor a site's sitemap: new and removed URLs since the last run

Turn on Only new and removed URLs since the last run ("onlyNewAndRemovedUrls": true) and schedule it: each run compares the sitemaps with the previous run of the same input and returns only the URLs that are new (not listed last time) or removed (listed last time, gone now), marked in changeType. It's a true comparison of the lists, so it works on sites whose lastmod dates are missing or wrong. URLs listed both times are counted, not returned and not charged, so a run where nothing changed costs only the $0.00005 start fee.

1. Put the sites in Websites or sitemap URLs, optionally narrow them with Only URLs matching (*/blog/* to watch a competitor's blog), turn on Only new and removed URLs since the last run, and click Start. The first run returns every URL (changeType null): that's the baseline the next runs compare with.
2. Click Save as a new task (top right of the actor page), then in Schedules click Create new, set how often (for example daily at 07:00) and Add the task.
3. On the task, open the Integrations tab to send each run's results to Slack, Gmail or a webhook (event ACTOR.RUN.SUCCEEDED; resource.defaultDatasetId is the run's new and removed URLs). A quiet run's dataset is empty and its status message says how many URLs were unchanged.

What it compares, and when it holds back:

• Per site and settings. Each line of input has its own list (in your account's key-value store sitemap-urls-memory), tied to its URL patterns and the sitemap depth and file limits: adding a site to the task doesn't reset the others, and changing a line's patterns starts a new baseline for that line.
• Removed only after a complete read. A URL is reported removed only when every sitemap found for that line was read to the end. If a sitemap failed to download, robots.txt blocked one, the file or depth limit skipped some, a file had more than 50,000 entries, or the run stopped at its max results or max cost, no URL is reported removed and removedNotReported in RUN_STATS says why. New URLs are still reported.
• A sudden drop waits one run. If a clean read lists nothing, or more than half of the tracked URLs (at least 20) disappear at once, it looks like a broken or replaced sitemap: the removals are held and reported only if the next run sees the same (removedHeld in RUN_STATS).
• Nothing lost to a limit. A new or removed URL is remembered only once its row is in your dataset, so one cut by the max results or max cost per run is reported again next run.
• It compares URLs only: a changed lastmod, priority or image on a URL that's still listed isn't reported.

The pre-filled example, run twice with the option on (local runs, 2026-09-26): the first run returned all 84 URLs of sitemaps.org as the baseline; the second returned none and said "0 URLs returned: 0 new and 0 removed since the last run, 84 unchanged (not returned), from 1 site".

What data does Sitemap URL Extractor return?

Field	Example	Notes
url	https://www.sitemaps.org/	Each URL once per site, even if several sitemaps list it.
lastmod	2016-11-21	Exactly what the sitemap says (a date or a timestamp); null if none.
changefreq	weekly	null if the sitemap doesn't say.
priority	0.8	null if the sitemap doesn't say.
images	[{"url": "https://www.example.com/img/launch.jpg", "title": null, ...}]	Image sitemap entries.
news	null	News sitemap entry: publication, date, title, keywords.
videos	[]	Video sitemap entries: title, thumbnail, duration, dates.
alternates	[{"hreflang": "de", "url": "https://www.example.com/de/blog/launch"}]	hreflang language alternates.
sitemapUrl	https://www.sitemaps.org/sitemap.xml	The sitemap file that listed it.
site	sitemaps.org	The input line it came from.
changeType	"new"	With Only new and removed URLs on: new or removed; null on the first run and with it off.
sourceTitle, sourcePlaceId, sourceIndex	"Rosie's Bakery", "ChIJ...", 12	From the dataset item the site came from (see Chain it after another actor); null for sites you type in, and sourcePlaceId when the item has no placeId.

The full list is under Output.

How much does it cost to extract sitemap URLs?

You pay per URL returned: $0.30 per 1,000 URLs, plus $0.00005 each time a run starts. URLs your filters drop are never charged, and neither is a URL listed twice. With Only new and removed URLs on, you pay for the new and removed URLs returned, at the same price; URLs listed both times are free.

It's cheaper on paid Apify plans: $0.25 per 1,000 URLs on Starter, $0.18 on Scale and $0.10 on Business. The prices on this page are the Free-plan price, so on a paid plan you pay less than the examples show.

• The example below: 84 URLs × $0.0003 = $0.0252, plus the $0.00005 start fee: about $0.025.
• A month, for example: a weekly full export of a 10,000-URL site is 4 × 10,000 × $0.0003 = $12.00; a daily run with "Last modified on or after" set to 1 day, returning about 50 changed URLs a day, is 1,500 × $0.0003 = $0.45 plus 30 starts ($0.0015).
• Monitoring, for example: a daily watch on a 10,000-URL site with Only new and removed URLs: the first run returns all 10,000 URLs ($3.00); then about 20 new or removed URLs a day is 30 × 20 × $0.0003 = $0.18 a month plus 30 starts ($0.0015). A day with no change costs the $0.00005 start fee.
• Caps: Max results per run in the input, and Maximum cost per run in the run options. The run stops cleanly at whichever comes first. It checks the limit before fetching each sitemap, so a capped run stops downloading as soon as the limit is covered.

How to extract all URLs from a sitemap

1. Open Sitemap URL Extractor and click Try for free (or Start if you're signed in).
2. Put your sites in Websites or sitemap URLs, one per line: a domain (example.com) or a sitemap's address.
3. Optional: keep only some URLs with Only URLs matching (*/blog/*) or Skip URLs matching, or set a date window with Last modified on or after (7 days, 2026-01-01).
4. Click Start, then open the Output tab and export as JSON, CSV or Excel.

Example: sitemaps.org

The pre-filled input:

{"sites": ["sitemaps.org"], "maxSitemapsPerSite": 200, "maxDepth": 3}

The first of the 84 URLs it returned (real output from a local run on 2026-09-25):

{
  "url": "https://www.sitemaps.org/",
  "lastmod": "2016-11-21",
  "changefreq": null,
  "priority": null,
  "images": [],
  "news": null,
  "videos": [],
  "alternates": [],
  "sitemapUrl": "https://www.sitemaps.org/sitemap.xml",
  "site": "sitemaps.org",
  "scrapedAt": "2026-09-25T17:45:17.769649Z"
}

Ready-to-run examples

Each example opens with the input already filled in. Run it as it is, or change the input first.

• Extract all URLs from a website's sitemap: Give it a domain and get every URL its sitemaps list, with last-modified dates, found through robots.txt and nested sitemap indexes.
• Monitor a sitemap for new and removed pages: Run on a schedule and get only the URLs added to or removed from a site's sitemap since the last run.

Input

Field	What it does
Websites or sitemap URLs	One per line: a domain or site address (the actor finds its sitemaps), or a sitemap's address (.xml, .xml.gz, .txt, or a sitemap index), which is read directly. Ignored with a dataset.
Or: websites from a dataset (datasetId)	One of your Apify datasets, e.g. a Google Maps scraper's results or another actor's run in an integration ({{resource.defaultDatasetId}}): each item's website is read as a site. See Chain it after another actor.
Field with the website (datasetUrlField)	Only with a dataset. Empty (default): found automatically among website, url, domain, site and homepage, then one level down (contact.website).
Only URLs matching	Keep URLs matching any of these patterns. * matches anything; a pattern without * matches anywhere in the URL. Case-insensitive.
Skip URLs matching	Drop URLs matching any of these patterns.
Last modified on or after / before	A date window on each URL's <lastmod>: 2026-01-31, a full timestamp, or relative (7 days, 2 weeks, 3 months). "Before" is exclusive.
Keep URLs without a lastmod date	With a date filter set, URLs whose sitemap gives no date are dropped unless this is on.
Only new and removed URLs since the last run	Off by default. On: return only URLs not listed on the previous run of the same input (new) and URLs no longer listed (removed). The first run returns every URL. Can't be combined with the lastmod date filters. See Monitor a site's sitemap.
Max results per run	Cap the total number of URLs across all sites.
Max sitemap files per site	The most sitemap files (indexes included) read per input line. Default 200.
Max sitemap index depth	How many levels of nested indexes to follow. Default 3.

{
  "sites": ["example.com", "https://www.example.org/sitemap_index.xml"],
  "includeUrlPatterns": ["*/blog/*"],
  "lastmodAfter": "30 days",
  "maxResults": 10000
}

Switching from another sitemap actor?

The inputs other sitemap actors use map onto Sitemap URL Extractor like this. An API call or integration that sends startUrls and maxUrls works as it is: change only the actor id.

If you used	In Sitemap URL Extractor	Notes
startUrls (strings or {"url": ...} objects)	sites	startUrls is read too. A domain, a site address or a sitemap URL (.xml, .xml.gz, .txt, an index).
maxUrls (0 = no limit)	maxResults (empty = no limit)	maxUrls is read too.
Include / exclude regular expressions	includeUrlPatterns / excludeUrlPatterns	Wildcards, one per line: .* becomes * (https://example.com/blog/.* is https://example.com/blog/*), and a pattern without * matches anywhere in the URL, so /blog/ alone works. A regex field from another actor (urlFilterInclude, urlFilterExclude) stops the run with this hint before anything is fetched or charged.
Follow sitemap indexes	Always on	Max sitemap index depth (maxDepth) 0 reads only the sitemaps you gave or robots.txt lists.
Max crawling depth, 0 = unlimited	maxDepth, 0 to 10, default 3	Here 0 means "don't follow indexes"; 10 covers any real site.
Check each page's status	Bulk URL & Broken Link Checker	Its sitemap mode reads the sitemap and returns each URL's status code, redirects and final URL in one run (see the FAQ below).
Proxy configuration	Not needed	Sitemaps are read directly, the way search engines read them. A site that refuses unknown crawlers is reported, not worked around.

The output keeps the usual names: url, lastmod, changefreq and priority, plus images, videos, news and alternates (hreflang) as lists, and sitemapUrl (the file the URL came from) and site (your input line).

Output

One row per URL. Fields the sitemap doesn't publish are null (or an empty list).

{
  "url": "https://www.example.com/blog/launch",
  "lastmod": "2026-09-01T10:00:00+00:00",
  "changefreq": "weekly",
  "priority": 0.8,
  "sitemapUrl": "https://www.example.com/post-sitemap.xml.gz",
  "site": "example.com",
  "images": [{"url": "https://www.example.com/img/launch.jpg", "title": null, "caption": null,
              "geoLocation": null, "license": null}],
  "news": null,
  "videos": [],
  "alternates": [{"hreflang": "de", "url": "https://www.example.com/de/blog/launch"}],
  "changeType": null,
  "scrapedAt": "2026-09-24T12:00:00Z",
  "sourceTitle": null,
  "sourcePlaceId": null,
  "sourceIndex": null
}

A removed URL (with Only new and removed URLs on) has only its address, the sitemap that last listed it and the input line; the rest is empty, because its sitemap entry is gone:

{"url": "https://www.example.com/blog/old-post", "lastmod": null, "changefreq": null, "priority": null,
 "sitemapUrl": "https://www.example.com/post-sitemap.xml.gz", "site": "example.com", "images": [], "news": null,
 "videos": [], "alternates": [], "changeType": "removed", "scrapedAt": "2026-09-25T07:00:00Z",
 "sourceTitle": null, "sourcePlaceId": null, "sourceIndex": null}

• lastmod is exactly what the sitemap says (a date or a timestamp).
• news: publicationName, publicationLanguage, publicationDate, title, keywords, genres.
• videos: thumbnailUrl, title, description, contentUrl, playerUrl, durationSeconds, publicationDate, expirationDate, familyFriendly, tags.
• site is the input line the URL came from, so results from several sites are easy to split (with a dataset, the website's home page, e.g. https://www.example.com).
• sourceTitle, sourcePlaceId, sourceIndex: the dataset item the site came from; null for sites typed in.
• changeType: new or removed with Only new and removed URLs on; null otherwise, and on the first (baseline) run.

The RUN_STATS record has per-site numbers: sitemaps read (sitemapsRead), failed (sitemapsFailed) and skipped (sitemapsSkipped, and why: robots.txt, depth or file limit), URLs listed, filtered out, duplicated and returned (emitted), and the first errors. It has the same keys on every run, including failed ones. With Only new and removed URLs on, each site also has monitor: whether this was its baseline, how many URLs were tracked, new, removed and unchanged, and removedNotReported / removedHeld when removals were held back (and why); changes totals new and removed. With a dataset, dataset says which field was read and how many items were skipped, and why.

Chain it after another actor

To list the sitemap URLs of the websites another actor found as soon as it finishes (a Google Maps scraper's businesses, a list of companies, our Website Technology Detector; any actor, ours or not), chain it with an integration:

1. Open the other actor or its saved task, go to the Integrations tab and add Apify actor → Sitemap URL Extractor, triggered when a run succeeds.
2. Set "datasetId": "{{resource.defaultDatasetId}}" in the input (Apify fills in the finished run's dataset) and the rest as usual, e.g. "includeUrlPatterns": ["*/blog/*"]. Websites or sitemap URLs is ignored then.

From the API it's the same input: pass a run's defaultDatasetId as datasetId. In the form, pick the dataset in Or: websites from a dataset.

• Which field: the website is found automatically in website, url, domain, site or homepage (the first that holds one in the first 100 items), or one level down (contact.website). Otherwise set Field with the website (datasetUrlField). A Google Maps scraper's own url (the Maps link) is never used; its website is.
• One line per site. A sitemap belongs to a whole site, so each website is read from its home page, whatever page the item links to (https://bakery.com/locations/denver is read as https://bakery.com, and that's its site). Several items on one website (branches of a chain, www. or not) read it once, with the first item's source fields. An item that gives a sitemap's own address (.../sitemap.xml) has it read directly.
• Options apply as usual: URL patterns, date filters, depth and file limits, Max results per run and your maximum cost per run.
• Only new and removed URLs remembers each site, not the dataset: a chained run reads a new dataset every time and still returns only what changed since the last run, and a site from a dataset shares its list with the same site typed in (https://www.example.com with www.example.com). Inputs without a dataset keep their lists from earlier versions.
• Joining back: every URL carries sourceTitle, sourcePlaceId and sourceIndex (0 = the dataset's first item) from the item its site came from.
• Skipped, never fetched or charged: items without a website, values that aren't a web address (a phone number, N/A) and Google Maps links; they're counted in RUN_STATS.dataset. A dataset with no website at all, or one the run can't read, fails the run with the reason.
• Limits: the first 20,000 items and 10,000 distinct websites of a dataset per run, read with your own account's access, read-only. A URL costs the same whichever way its site came in; the other actor's run is billed by it.

Run it on a schedule, or from your own code

1. Save your input as a task (Save as a new task, top right of the actor page) and add it to a schedule (Console → Schedules → Create new): for example daily, with Only new and removed URLs since the last run on, for the pages the site added or dropped since yesterday (see Monitor a site's sitemap). Or, with the option off, set Last modified on or after to 1 day for the pages whose sitemap lastmod says they changed (this relies on the site keeping its lastmod dates accurate).
2. Collect results: download the dataset as JSON, CSV or Excel; fetch the latest run's results from the API (GET https://api.apify.com/v2/actor-tasks/<task id>/runs/last/dataset/items?status=SUCCEEDED&format=csv, with your API token); let a webhook tell your system when a run succeeds; or connect it to Make, Zapier or n8n through Apify's integrations. The task's Integrations tab can also post to Slack or email you (Gmail) after each run.

Can I use Sitemap URL Extractor from an AI agent (MCP)?

Yes, through Apify's MCP server, from Claude, ChatGPT, Cursor or any other MCP client. Add this to your client's MCP configuration (or let the agent find it with the server's actor search); your client signs you in to Apify:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=humble-echidna/sitemap-urls"
    }
  }
}

To use an Apify API token instead of signing in, add "headers": {"Authorization": "Bearer <APIFY_TOKEN>"} next to url.

The agent passes the site and a cap, e.g. {"sites": ["example.com"], "includeUrlPatterns": ["*/docs/*"], "maxResults": 200}.

Who it's for

SEO and content teams who need a site's complete, current URL list: to plan a crawl or an audit, to find pages missing from the sitemap, to spot recently changed pages by lastmod, or to feed URLs into another tool. And anyone watching a site over time (a competitor's new pages, a client's pages that disappeared from its sitemap).

Why this one?

• One flat price, no usage bill. You pay per result at the price above and nothing else: no compute, proxy or storage charges on top, apart from the $0.00005 run-start fee, so you know what a run costs before it starts. Results you don't get aren't charged.
• Complete discovery. robots.txt Sitemap: lines first (every one of them), then the usual locations. Sitemap indexes are followed to any depth you allow, gzipped and plain-text sitemaps are read, and HTML "soft 404" pages served at /sitemap.xml are recognised and skipped instead of failing the run.
• All the sitemap data, not just the links. lastmod, changefreq, priority, plus image, news, video and hreflang alternate entries as structured fields.
• Filters before you pay. Keep only URLs matching your patterns (*/blog/*) or changed in a date window (7 days, 2026-01-01). Filtered-out URLs are never charged.
• New and removed URLs, not a date guess. The monitoring mode compares the actual lists from one run to the next, so it catches new pages on sites with missing or wrong lastmod dates, and removed pages, which a date filter can't see. It never reports URLs removed from a list it couldn't read completely, and a quiet run costs only the start fee.
• Each URL once. A URL listed in several of a site's sitemaps is returned once; a sitemap linked from several indexes is read once.
• Polite and honest. It honours robots.txt (Disallow and Crawl-delay, retries included; the file is read once per site per run) and Retry-After, identifies itself with its own User-Agent (HumbleEchidnaApify/1.0 (+https://apify.com/humble-echidna); robots.txt rules for HumbleEchidnaApify apply to it), and never tries to get around a site that refuses it.
• Safe on hostile input. Size caps (50 MB per sitemap, the standard's own limit, before and after decompression, counted while downloading), a time limit per download, a hardened XML parser (no DTDs or entity expansion) that reads each file as a stream so memory stays flat however big it is, and no requests to private or internal network addresses or non-standard ports, on every redirect.
• Reliable. One failing site never affects the others in your run. The run log and the RUN_STATS record say exactly which site or sitemap had a problem and why.

Limits

• It reads sitemaps only; it doesn't crawl pages. A site without a sitemap returns an error saying so; if you know its sitemap's address, enter that directly.
• Sites that block unknown crawlers (HTTP 401/403) or whose robots.txt disallows the sitemap are reported, not worked around. A robots.txt that answers with a server error counts as "disallow everything", as the robots.txt standard (RFC 9309) requires.
• If a site asks for more than 30 seconds between requests (Crawl-delay), or more than 90 seconds before a retry (Retry-After; one such long wait per site, at most 3 minutes of them per run), it's reported with that reason instead of being read slowly.
• A sitemap over 50 MB (before or after decompression) is skipped and reported. A sitemap with more than 50,000 entries (the standard's per-file limit) is read up to there, and that's reported (sitemapsOverEntryLimit).
• Only public web addresses on the standard ports (80 for http://, 443 for https://) are read: a URL with another port such as :8080, or a name that resolves to a private network, is refused with the reason. A domain that doesn't exist is reported as such.
• example.com, https://example.com/ and http://example.com in the same input are one site, read and charged once.
• RSS and Atom feeds are not read as sitemaps.
• Monitoring (Only new and removed URLs) remembers up to 200,000 URLs per line of input; past that, further new URLs aren't reported and nothing is reported removed (split a bigger site with URL patterns, one line each). It can't be combined with the lastmod date filters: a URL leaving the date window would look removed. Two tasks with the same site and settings share one list, so run one of them, or give them different URL patterns.

FAQ

Why did a site return fewer URLs than I expected?

Check RUN_STATS: large sites often split their sitemap over more than 200 files (raise "Max sitemap files per site"), or list some sitemaps under paths their own robots.txt disallows.

Can I get only the URLs that are new since my last run?

Yes: turn on Only new and removed URLs since the last run and schedule the task. Each run returns the URLs that weren't in the sitemaps last time (changeType new) and the ones that are gone (removed); the first run returns every URL as the baseline. Unchanged URLs aren't returned or charged. If you want URLs whose lastmod date is recent instead (including changed pages that were already listed), leave it off and set Last modified on or after to 1 day.

Why weren't removed URLs reported?

Removed URLs are only reported when every sitemap for that site was read completely. Check RUN_STATS: the site's monitor.removedNotReported names what stopped it (a sitemap that failed or robots.txt blocked, the file or depth limit, the run's max results or max cost), and monitor.removedHeld says when a sudden big drop is waiting for the next run to confirm it. The URLs stay remembered, so they're reported on the first run that reads the site completely.

How much does monitoring cost when nothing changes?

The $0.00005 start fee: URLs that are listed both times aren't returned or charged. You pay $0.30 per 1,000 only for new and removed URLs, and for every URL on the first run.

Something that used to work now fails or returns fewer results. Why?

The site may have moved or changed its sitemaps. The run log names the site and sitemap and what went wrong, and every other site in the run is unaffected. Please open an issue with the input you used.

Can it check each sitemap URL's HTTP status code (find 404s)?

Not this actor: it reads sitemaps only, which is why it's fast and charges only for URLs. For status codes, run Bulk URL & Broken Link Checker with What to check set to sitemap and your site: it reads the sitemap itself and checks every URL it lists (status code, every redirect hop, final and canonical URL, response time) in one run, at its own price per URL. Or chain the two: extract and filter the URLs here, then pass this run's dataset to the checker's datasetId with What to check set to urls.

Is it legal to extract URLs from a sitemap?

Sitemaps are published by site owners for crawlers to read. The actor reads only the sites you enter, obeys their robots.txt, uses no login, and collects no personal data. You are responsible for how you use the URLs.

Related actors

Sitemap URLs are often the first step of a bigger job:

Actor	Use it when
Bulk URL & Broken Link Checker	You want every sitemap URL's status, redirects and canonical (it can read the sitemap itself in its sitemap mode).
SEO Audit Crawler	You want the pages' on-page SEO audited; its sitemap mode reads the sitemap itself.
Website & Page to Markdown for RAG and LLMs	You want the pages as Markdown; its whole-site mode reads the sitemap first.

Feedback and support

Found a bug, or need a sitemap field that isn't here? Open an issue on the Issues tab with the input you used.

Versions

Current version: 1.3. See the Changelog tab for what changed in each version.