Sitemap Change Monitor: detect new, removed and modified pages on any website

Monitor a website's sitemap and get only the new, removed and modified pages since your last run, or use it as a sitemap URL extractor that lists every page URL a site publishes. It reads the site's robots.txt and sitemaps, requests only robots.txt and sitemap URLs, abandons any HTML response unread, and no page content is stored or output. Pay per result, with no charge for starting a run: $0.30 per 1,000 url records and $2.00 per 1,000 change records.

When to use it, and when not to

Use it when you want to:

• watch a competitor's site, docs site, shop or news site for new pages on a daily schedule;
• extract every URL from a sitemap (with lastmod, changefreq and priority) for SEO audits or crawl lists;
• track pages removed from a site without false alarms from failed, truncated or rolling sitemaps.

Pick something else when you need:

• the text, prices or other content of pages: this Actor never reads page content, and lastmod is its only signal for modified pages;
• sites with no XML, gzip or plain-text sitemap (RSS/Atom and HTML sitemaps are not read, and a site with no sitemap is reported failed and billed nothing);
• anything behind a login, or built-in alerts: schedule it and connect Apify's own notifications instead.

What it does

• Extract mode: every page URL listed, with lastmod, changefreq, priority and the sitemap it came from.
• Monitor mode: remembers each sitemap between runs and outputs new, removed and modified pages.
• Built for messy real sitemaps: an unchanged site gives zero changes on its second run; a failed, truncated, dead or rolling sitemap never produces false removed records; deploy stamps and host migrations are not billed as whole-site changes.
• Only robots.txt and sitemap URLs are requested. If one of them answers with an HTML page, it is abandoned unread (the sitemap is reported as unsupported format), and no page content is stored or output.

Use cases

• Competitor new-page alerts: { "sites": ["competitor.example"], "mode": "monitor", "maxChanges": 500 } — schedule daily and connect Apify's own notifications (email, Slack, webhook). The Actor has no alerts of its own.
• Docs change feed: { "sites": ["https://docs.example.com/sitemap.xml"], "mode": "monitor" } — a page counts as modified only when its lastmod changes; if a site lists no lastmod, its pages can be new or removed but never modified.
• E-commerce new-product watch: { "sites": ["https://shop.example.com/sitemap_products.xml"], "mode": "monitor", "stateKey": "shop-products" } — product URLs only; no page content, prices or stock are read.
• News sitemap monitor: { "sites": ["https://news.example.com/news-sitemap.xml"], "mode": "monitor", "rollingSitemaps": ["news-sitemap"] } — news sitemaps are rolling, so articles that age out are not billed as removed.
• Plain sitemap URL extractor: { "sites": ["example.com"], "mode": "extract" } — extract mode keeps no state and bills each URL as a url event on every run.

Use it from Claude, Cursor or any AI agent

Add Sitemap Change Monitor as a tool through Apify's MCP server: connect your MCP client to https://mcp.apify.com?tools=frostpine456/sitemap-monitor with your Apify API token. Then ask things like "List every URL in this site's sitemap" (extract mode) or "What pages did this site add since last week?" (monitor mode with the same stateKey). For quick answers keep maxUrlsPerSite small; the same prices apply.

Input

Field	Type	Default	Meaning
sites	string[], 1–50	required	Each item is a domain (example.com), a site URL reduced to its host (https://example.com/any/path), or a direct sitemap URL whose path ends in .xml, .xml.gz, .gz or .txt
mode	"monitor" | "extract"	"monitor"	See above
maxUrlsPerSite	integer 1–50,000	10,000	URLs read per site per run (de-duplicated canonical URLs); a larger site is truncated
maxChanges	integer 1–20,000	2,000	change-billed records per run across all sites; the rest are deferred to the next run, and sites after the cap is reached still get their baseline (and existing) records
includeExisting	boolean	false	Monitor mode: also output unchanged URLs, billed as url on every run
includeProfileUrls	boolean	false	Include sitemaps and URLs that look like people's profile pages
rollingSitemaps	string[]	[]	Extra substrings that mark a sitemap as rolling when found in its URL's path or query
stateKey	string [a-z0-9-]{1,40}	"default"	Names the saved state; changing it starts a new paid baseline
shrinkGuardPercent	integer 10–100	50	Per-sitemap drop allowed before removals are held

The Console example input runs extract mode on https://docs.apify.com with maxUrlsPerSite 100, so every try returns URLs (about $0.03 per run). The default mode is still monitor: pick it, or leave mode unset in the API, to track changes between runs.

Output

One dataset record per URL: site, url, lastmod, changefreq, priority, sitemapUrl, fetchedAt, and in monitor mode changeType (baseline, new, removed, modified, existing), previousLastmod (for modified) and firstSeenAt. No person data and no page content.

The free SUMMARY key-value record reports each site's status (ok, partial, truncated, failed, skipped), each sitemap's status and reason, per-site counts, flags (for example shrinkSuspected, stateWasInvalid) and per-section change counts (first path segment, top 50).

Pricing

Pay per event, no start event:

Event	Price	Records
url	$0.30 / 1,000	extract-mode records; baseline records; existing records
change	$2.00 / 1,000	new, removed, modified

Items that fail before any record is output are free, and so are failed sitemaps. An item that fails while writing records or state may have billed records, which the next run can repeat (at-least-once). SUMMARY is free.

The first monitor run is a paid baseline, later runs bill only changes. Every URL of a sitemap is output as baseline and billed as url the first time it is seen. Changing stateKey starts a new paid baseline.

Worked example (a hypothetical site with 5,000 URLs, run daily):

Run	Records	Cost
Run 1 (baseline)	5,000 baseline = 5,000 url events	5 × $0.30 = $1.50
Run 2	12 new + 3 removed + 5 modified = 20 change events	20 / 1,000 × $2.00 = $0.04

On this example the daily cost after the first run is about $0.04 per day. The site sizes and change counts here are invented for arithmetic, not measured results.

includeExisting caveat: with includeExisting: true, the same run 2 would also output 4,992 unchanged URLs as existing, billed as url on every run: 4,992 / 1,000 × $0.30 ≈ $1.50 plus $0.04 of changes, so about $1.54 per day instead of $0.04. Leave it off for scheduled runs unless you need the full list each time.

Extract mode: 5,000 URLs cost the same $1.50 per run, because every record is a url event.

Scheduling and state

Save the input as a Task and add a daily Schedule. Keep the same stateKey between runs; use a different stateKey per independent list. Concurrent runs on one stateKey can bill the same changes twice, so do not overlap them. Delivery is at-least-once: if a run crashes between pushing records and writing state, the next run can repeat those records.

How it protects you from false changes

• Held removals: while any sitemap of a site is not fully read (failed, disallowed, truncated, cut off, or recently dropped from its index), no sitemap of that site emits removals, because a URL may have moved into the unread one. A sitemap treated as gone (dead or dropped for good) no longer holds the others.
• Shrink guard: if a complete read drops by more than shrinkGuardPercent (default 50%) of the last confirmed count, removals are held until a second consistent run confirms them.
• Rolling sitemaps (news sitemaps, or any sitemap whose path or query, not its host, contains news or one of your rollingSitemaps substrings) are exempt from removals: URLs that age out emit nothing.
• Dead sitemaps count as gone only after 3 runs answering 404 or 410, or after 10 runs failing any other way (server errors, rate limits, timeouts, network errors, oversized or malformed bodies); a full read resets both counts. Only then are their URLs emitted as removed, once. A sitemap that was never read in full drops its URLs silently when it goes.
• Deploy-stamp guard: a mass lastmod update to one identical value (a redeploy stamp) is not billed as thousands of modified records; it is reported in SUMMARY as stampedLastmod.
• Host-migration guard: when the same sitemap file starts listing its pages on a new host, and every removed URL matches a new one once scheme and a leading www. are ignored, neither set is billed; SUMMARY reports hostMigrated. A host change that also moves the sitemap file to a new URL is not covered: the new sitemap starts a new paid baseline, and the old sitemap's URLs are reported removed once it is treated as gone.
• Profile pages: profile pages — a URL whose path is exactly /users/<name>, /profile/<name> or the like (users, user, people, profile, profiles, author, authors, member, members, case-insensitive) — and child sitemaps with one of those words as a whole token of a path segment (each segment is percent-decoded and split on any non-letter, dots and digits included, e.g. sitemap.users.xml, members.xml.gz, sitemap-authors1.xml or author-sitemap.xml; a word fused to other letters, like userprofiles.xml, does not count) are skipped by default (turn on includeProfileUrls to include them). Switching it either way never reports profile pages as removed or new: skipped ones stay remembered, and ones included for the first time come out as baseline.

Limits

• Up to 50 sites per run; 10,000 URLs per site by default — raise maxUrlsPerSite (up to 50,000) if SUMMARY shows truncated.
• Truncation: while a site is truncated, its cut-off and unread sitemaps report no new, modified or removed pages. In a cut-off sitemap that has never been read in full, newly seen URLs come out as baseline; in one that has, they are held and come out as new (billed as a change) once it is read in full. Raise maxUrlsPerSite for a real change feed.
• 20 MB decoded per sitemap file; XML (urlset and sitemap index), gzip and plain-text sitemaps only — no RSS/Atom or HTML sitemaps.
• Sitemap indexes are followed to depth 3, with at most 500 child sitemaps per item.
• Very large sites can exceed the per-sitemap state limit (8 MB); such a sitemap is marked failed with reason state too large and bills nothing. Use 1024 MB of memory for very large sites or many large sites.
• Memory: runs need at least 512 MB (the default). Sites are read one at a time, and the peak per sitemap file is its body (up to 20 MB decoded, held once) plus its parsed URL list.
• No moved events; no image, video or news extension fields; lastmod is the only change signal for modified pages.

Politeness and robots.txt

Each sitemap's path is checked against the robots.txt of its host before fetching, and so is every redirect it follows, each on its own host; a sitemap that is disallowed, or that redirects somewhere disallowed, is skipped and named in SUMMARY. At most 1 request per second per host (one in flight), at most 4 hosts at once, retries with backoff on 429/5xx and network errors, a 20 second timeout, and a host that keeps returning 429 is given up on for the rest of the run. Identifying User-Agent: makemoney-sitemap-monitor/1.0 (+https://apify.com/frostpine456/sitemap-monitor).

Responsibility

You are responsible for having the right to monitor the sites you name.

Not affiliated

This Actor is not affiliated with, endorsed by or sponsored by any website, company or search engine whose sitemaps you monitor. Site names are used only to describe what you chose to monitor.

Data and privacy

Public sitemap files only. No login, no personal data, and no page content is stored or output: only robots.txt and sitemap URLs are requested, and an HTML response is abandoned unread. URLs that look like people's profile pages are skipped unless you turn on includeProfileUrls.

Takedown

Site owners can request exclusion via the Issues tab; we honour all requests.

FAQ

• Why was my first run billed for every URL? The first monitor run of a sitemap is a paid baseline: every URL is output once as baseline (a url event). Later runs bill only changes.
• How do I get only new pages? Use monitor mode (the default) and keep the same stateKey; after the baseline you see only new, removed and modified.
• Why did a page not show as modified? A page is modified only when its lastmod changes. Sites without lastmod can show new and removed pages, never modified.
• Why are removals missing from a news sitemap? News sitemaps are rolling: old articles age out, and rolling sitemaps are exempt from removals.
• What does truncated mean? The site has more URLs than maxUrlsPerSite; reading stopped at the cap, the site's status is truncated, and no removals are reported for any of its sitemaps. Raise maxUrlsPerSite.
• Does it work without a sitemap? It reads robots.txt Sitemap: lines first, then tries /sitemap.xml and /sitemap_index.xml. If none parse, the site is reported failed and billed nothing.