Sitemap Extractor & Change Tracker

Get every URL from a website's sitemaps, and see which pages are new, updated or removed since your last run. Enter a domain and the Actor finds the sitemaps itself, reads nested and compressed ones, and tells you exactly what it could and couldn't read.

• Finds sitemaps other tools miss. It checks robots.txt first, then the usual locations (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt and more), then the home page's <link rel="sitemap">. If robots.txt lists sitemaps that no longer exist, it falls back to the usual locations.
• Reads what's actually out there: sitemap indexes nested several levels deep, .gz files (even when the server sends the wrong content type or compresses them twice), plain-text sitemaps, RSS and Atom feeds, namespaced and slightly broken XML.
• Tracks changes between runs. Each URL is marked new, updated (its last-modified date changed), unchanged or removed. Schedule it weekly and save only the changes: new product pages, new blog posts, deleted pages.
• No false alarms. If a sitemap fails to load this time, its pages are not reported as removed. They're remembered and compared again next run.
• Honest reports, not silent failures. A web page served instead of a sitemap, a corrupt .gz file, a timeout or bot protection: each problem is named per sitemap file in the report. One broken sitemap doesn't fail the run; the URLs from the others are kept.
• Polite by design. One request at a time per website, at least half a second apart, slower if robots.txt sets a Crawl-delay. Obeys robots.txt and identifies itself honestly. Never bypasses bot protection: if a site blocks automated access, you're told so.
• You don't pay for nothing. Sites where no sitemap could be read cost nothing beyond the tiny start fee.

Quick start

1. Enter one or more websites (example.com) or sitemap addresses under Websites or sitemap URLs.
2. Run it. The first run lists every URL and remembers them.
3. To monitor a site, set What to save to Only changes and add a schedule (for example every Monday). Each run then saves only the pages that are new, updated or removed since the previous run.
4. Optional: connect an integration (Slack, email, Google Sheets, Zapier, Make, a webhook) to get the changes delivered.

Tip: enter a section such as example.com/blog/ to get only that section's URLs. The Actor checks for the section's own sitemap first, which is much faster on big sites. Enter the bare domain to get the whole site.

Output

One row per URL:

{
  "url": "https://crawlee.dev/blog/scrapy-vs-crawlee",
  "changeType": "updated",
  "lastmod": "2026-09-30",
  "previousLastmod": "2026-06-12",
  "firstSeenAt": "2026-08-04",
  "changefreq": "weekly",
  "priority": 0.5,
  "site": "https://crawlee.dev/",
  "sitemapUrl": "https://crawlee.dev/sitemap.xml"
}

Field	Meaning
changeType	new, updated, unchanged or removed compared with the previous run. Empty when change tracking is off.
lastmod / previousLastmod	The page's last-modified date from the sitemap, now and last time.
firstSeenAt	The date this tracker first saw the URL.
changefreq, priority	As given in the sitemap, if present.
sitemapUrl	The sitemap file the URL came from.

Rows are ordered new, updated, removed, then unchanged. Download them as JSON, CSV, Excel or HTML from the run's Storage tab, or via the API.

The OUTPUT record in the key-value store has a report per site: how its sitemaps were found, how many were read, which ones failed and why, and the counts of new, updated and removed pages.

How change tracking works

• After each run the Actor stores each site's URLs (with their last-modified dates) in a storage on your own Apify account, named sitemap-change-tracker by default. Nothing is shared with anyone else.
• The next run compares against it. A URL is updated only when its sitemap date changes. Pages whose sitemap has no dates can show up as new or removed, but not as updated.
• Only changes on a first run saves no rows; it just records the starting point. Use All URLs if you want the full list on the first run.
• Histories are kept per site and per filter setting, so changing Only URLs containing or Skip URLs containing starts a fresh history for that combination. Use different History names to keep separate histories for separate projects, or Start history fresh to reset.
• If any of a site's sitemaps couldn't be read, or a URL, memory or cost limit cut the scan short, missing pages are not reported as removed in that run.

Filters

• Only URLs containing / Skip URLs containing: text that must, or must not, appear in the URL. * is a wildcard (/products/*/reviews). Start with re: for a regular expression (re:/20(25|26)/).
• Modified since: a date (2026-09-01) or a period (7 days, 2 weeks, 3 months). Keeps URLs whose sitemap date is on or after it; URLs without a date are left out. With change tracking on, all URLs are still remembered so the comparison stays correct. Removed pages are always reported.
• Max URLs per site: stop reading a site after this many URLs.

Pricing

You pay only for what the Actor does, with no monthly fee:

Event	Price	When
Actor start	$0.00005	Each run (Apify's standard start event)
Site scanned	$0.002	Per website where at least one sitemap was read
URL found	$0.05 per 1,000	Each unique URL read from the sitemaps (after your filters)
URL saved	$0.25 per 1,000	Each row saved to the dataset

Examples:

• Full list of a 5,000-page site: 0.002 + 5,000 × (0.00005 + 0.00025) ≈ $1.50.
• Weekly monitoring of that site with Only changes, 40 changes a week: 0.002 + 5,000 × 0.00005 + 40 × 0.00025 ≈ $0.26 per week.
• A site with no readable sitemap: $0.00005.

Set a maximum cost per run in the run options. The Actor stops cleanly at that limit. Changes it couldn't save are not lost: they're reported again on the next run.

Good to know

• Big sites: the default 1 GB of memory holds about 400,000 URLs for one site. Up to three sites are read at the same time and share that memory equally: about 200,000 URLs each for two sites, 135,000 each for three or more. If a site is bigger, the Actor stops cleanly at that point and the report says so. Raise the memory in the run options to read it all (each extra GB adds about 400,000 URLs). The cut-off is the same every run, so it never shows up as false "new" or "removed" pages.
• Speed: a site with one sitemap takes a few seconds. Sites with hundreds of sitemap files take longer, because the Actor waits between requests to the same website (and longer if robots.txt asks for it).
• Not found? If no sitemap is found, enter the sitemap's address directly. Some sites only mention it in Google Search Console.
• Blocked? Some sites block all automated requests (for example, with a Cloudflare challenge). The report says so. The Actor does not try to get around it.
• This Actor reads sitemaps, files that websites publish specifically for automated crawlers. It doesn't visit or download the pages themselves, and it never collects personal data.

FAQ

Why are some sites slow? Each website gets one request at a time, half a second apart or more. That keeps the Actor welcome on the sites you monitor. Different websites are read in parallel.

Why was a URL I can see on the site not found? It isn't in the site's sitemaps. Sitemaps list what the site owner chose to list. This Actor reads sitemaps rather than crawling links.

Can I use it from an AI agent or the API? Yes. Call it with a list of URLs and read the dataset. The OUTPUT record gives a compact summary per site.

Something wrong? Open an issue on the Issues tab with the site and the run link. Fixes for real sitemaps that fail are a priority.