Sitemap Change Monitor – New, Removed & Updated Pages Tracker

Watch any website's XML sitemap and get told what changed since the last run: pages that were added, pages that were removed, and pages whose lastmod date changed. Track competitors' new blog posts, product pages, landing pages and docs without crawling their sites.

Run it once to save a baseline, then schedule it (for example daily). Each later run compares every site with the previous run and outputs only what changed.

Who it is for

• SEOs and content marketers following what competitors publish, update and take down.
• E-commerce teams watching for new product and category pages on competing stores.
• Agencies reporting site changes for clients, or checking that a migration did not drop pages.

What it does

1. Finds each site's sitemaps: the Sitemap: lines in robots.txt, or else /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml. You can also give a sitemap URL directly.
2. Reads them all: sitemap indexes (nested up to 3 levels, up to 500 sitemap files per site), gzipped .xml.gz files, and plain-text sitemaps (one URL per line). It reads each page's <loc> and <lastmod>.
3. Compares the URLs with the snapshot saved by the previous run for that site and writes one change row per added, removed or updated page.
4. Optionally writes every URL as a url row.
5. Writes one free site status row per site, and a run summary to the OUTPUT record.

URLs are compared after light normalization: surrounding spaces trimmed, the #fragment dropped, the query string kept, the host lower-cased. A URL listed twice counts once, with the latest of its lastmod values.

Input

Field	Default	What it does
sites	["https://astro.build"]	Up to 20 sites or sitemap URLs. astro.build or https://astro.build finds the sitemaps itself; https://example.com/post-sitemap.xml (ends in .xml, .xml.gz or .txt, or contains "sitemap") is read as given. For a site, any path is ignored.
outputMode	changes_only	changes_only (cheapest for daily monitoring), urls_and_changes, or urls_only (no change tracking).
maxUrlsPerSite	5000	Stop reading a site after this many unique URLs (max 200,000).
detectLastmodChanges	true	Report pages whose <lastmod> changed.

Example:

{
    "sites": ["https://astro.build", "nodejs.org", "https://www.fsf.org/sitemap.xml.gz"],
    "outputMode": "changes_only",
    "maxUrlsPerSite": 20000
}

Output

All rows go to the run's dataset. The type field tells them apart. The dataset has three views: Changes, URLs and Site status.

Change row (type: "change")

{
    "type": "change",
    "site": "https://www.fsf.org/sitemap.xml.gz",
    "url": "https://www.fsf.org/blogs/community/2025-free-software-bulletin-issue-47-now-available",
    "changeType": "lastmod_changed",
    "lastmodBefore": "2025-12-18T10:02:11-05:00",
    "lastmodAfter": "2026-01-05T16:31:35-05:00",
    "sitemap": "https://www.fsf.org/sitemap.xml.gz",
    "detectedAt": "2026-10-12T11:40:03.112Z"
}

changeType	Meaning	lastmodBefore → lastmodAfter
added	The URL is listed now and was not listed in the previous run.	null → current lastmod (or null)
removed	The URL was listed in the previous run and is no longer listed in any of the site's sitemaps.	previous lastmod (or null) → null
lastmod_changed	The URL is listed in both runs and its <lastmod> changed.	previous → current

Lastmod values are shown exactly as the site writes them. Two values that are the same moment in different notations (2026-10-01 and 2026-10-01T00:00:00+00:00) are not a change. A page that gains or loses its <lastmod> is not reported as a change.

URL row (type: "url", only in urls_and_changes and urls_only)

{
    "type": "url",
    "site": "https://astro.build",
    "url": "https://astro.build/blog/astro-5/",
    "lastmod": null,
    "sitemap": "https://astro.build/sitemap-0.xml",
    "scrapedAt": "2026-10-12T11:49:24.120Z"
}

Site status row (type: "site", free, one per site in every mode)

{
    "type": "site",
    "site": "https://astro.build",
    "input": "https://astro.build",
    "status": "ok",
    "baseline": "compared",
    "urlsFound": 338,
    "sitemapsRead": 2,
    "sitemapsFailed": 0,
    "complete": true,
    "removalsDetected": true,
    "lastmodUniform": false,
    "added": 3,
    "removed": 2,
    "lastmodChanged": 0,
    "changesDetected": 5,
    "changesPushed": 5,
    "urlsPushed": 0,
    "stopReason": null,
    "errorType": null,
    "error": null,
    "checkedAt": "2026-10-12T11:49:24.549Z"
}

• status: ok; partial (some sitemap files could not be read, or the charge limit or a dataset error stopped the rows); failed (no sitemap with URLs could be read, for example none exists, the site blocks automated access, or robots.txt could not be read); skipped (not processed: the charge limit was already reached, or another input already covers the same sitemap).
• baseline: created (first run for this site: snapshot saved, no changes reported), compared, not-tracked (urls_only), unknown (rare: a run restarted after a platform migration whose comparison was already saved), or null when nothing could be read.
• complete: every sitemap file was read in full (none failed, came back empty or was cut off) and no limit was hit. Removed pages are reported only when this is true.
• stopReason: max-urls (more URLs than maxUrlsPerSite) or max-sitemaps (more than 500 sitemap files).

The OUTPUT record in the default key-value store holds the same per-site details plus the sitemap roots used, how they were found (robots, fallback or direct), the first sitemap errors, and run totals. The run fails only when every site failed.

How change tracking works

• First run for a site = baseline. It saves a snapshot and reports no changes. Schedule the actor (daily is typical); changes are measured between consecutive runs.
• Snapshots live in your account, in a named key-value store called sitemap-change-monitor-state-erok7ac472yj0bycy (the name ends with this actor's ID in lowercase, so it does not clash with the state store of another actor that has the same name). It holds URLs and lastmod values only, gzipped and split into several records for large sites. Delete a site's records there to start it over with a new baseline. All runs of this actor in your account share these snapshots, so two schedules watching the same site each see only the changes since the other one ran.
• Removed pages are reported only after a complete read. If a sitemap file fails (server error, blocked, timeout, an empty body, or a file cut off before its closing tag), added and updated pages are still reported, removed pages are not, and the URLs of the failed file are kept in the snapshot so nothing is reported twice when it comes back.
• A sitemap that suddenly loses half of its URLs (at least 50 URLs) is treated as a possible glitch: those removals are held back for one run and reported only if the next complete read confirms them.
• Site-wide lastmod stamps are ignored. Some sites write their build time into every URL's <lastmod>, either as one value or as values a few seconds apart. When at least 90% of a site's URLs (and at least 20) share one lastmod value or fall within one 60-second window, in the previous run or in this one, lastmod changes are not reported for that run (lastmodUniform: true); added and removed pages still are.
• Large sites. When a site has more URLs than maxUrlsPerSite, removed pages cannot be detected for it, and only URLs whose lastmod is later than the previous run (or that could not be delivered last time) count as added.
• Undelivered changes come back, delivered ones do not. If the charge limit (or a dataset error) stops a run while change rows are being written, the snapshot is updated only for the changes that were delivered. The next run reports exactly the rest. A run that the platform moves to another server (migration) or that you abort waits up to 20 seconds for this bookkeeping before it stops; see Limitations for the case it cannot cover.

Pricing

Pay per event. Apart from a tiny fixed fee per run, you pay only for rows actually written to your dataset:

Event	When	Price
apify-actor-start	Once per run	$0.00005
url-listed	One URL row written (only in urls_and_changes and urls_only)	$0.0003 ($0.30 per 1,000 URLs)
change-detected	One change row written (added, removed or lastmod_changed)	$0.002 ($2.00 per 1,000 changes)

Site status rows are free. With the default changes_only mode, a first run (baseline) costs only the $0.00005 run fee, and later runs cost the run fee plus the changes found.

If you set a maximum cost per run, the actor stops cleanly when it is reached and says so in the status message and in OUTPUT. Sites cut short are marked partial, sites not started are marked skipped, and changes that could not be delivered are reported on the next run. Delivered changes are not reported or charged again, also across a platform migration or an abort; the exception is a hard stop described under Limitations. When you abort a run, no further charged rows are written; changes not yet written are reported on the next run.

Limitations

• Only what the sitemaps say. A page that exists but is not in the sitemap is invisible; a page removed from the sitemap is reported as removed even if it is still online. Pages are not fetched, so title or content changes are not detected; lastmod_changed is only as reliable as the site's own <lastmod> values.
• Reads <loc> and <lastmod> only. Image, video and news sitemap extensions, hreflang alternates, <changefreq> and <priority> are ignored. RSS and Atom feeds used as sitemaps are not supported (the site is reported with that reason).
• Respects robots.txt: sitemap files the site disallows for crawlers are not read, a Crawl-delay up to 30 seconds is honored for that host, and a site whose robots.txt cannot be read (server error) is not read at all. robots.txt is checked for the site's own host only; sitemap files hosted elsewhere (for example on a CDN) are read without a robots check.
• Sites behind bot protection (Cloudflare challenges and similar) usually answer with an HTML page instead of the sitemap; they are reported as failed with that reason. No proxies, no CAPTCHA solving.
• Only public internet addresses on the standard ports (80/443) are contacted, also when a sitemap index or a redirect points elsewhere. Sitemap files are limited to 50 MB (after unpacking .gz).
• Duplicate inputs (the same site twice, or a site and its sitemap URL, also when one of them redirects to the other) are processed once, the first in input order. A site and one of its child sitemaps given as separate inputs are not detected as duplicates and are both read.
• Changes are recorded as delivered when the run saves the site's snapshot, right after writing its rows. If the run is killed hard in between (out of memory, or a forced stop that skips the abort handling), the next run reports those changes again and they are charged again. Migrations and normal aborts wait up to 20 seconds for the save.
• Plain-text sitemaps have no closing tag, so a text sitemap cut off in transfer cannot be told from a shorter one.
• Memory: the default (1 GB) covers three sites reading 50 MB sitemap files at the same moment. If you lower the memory for sites with very large sitemap files, the run can run out of memory.
• Up to 20 sites per run, 200,000 URLs per site, 500 sitemap files per site.

FAQ

How do I check that a site works? Open https://<site>/robots.txt and look for Sitemap: lines, or open https://<site>/sitemap.xml. If you see XML with <url> or <sitemap> entries, the actor can read it.

Why did the first run report no changes? There was nothing to compare with yet. The first run saves the baseline; changes appear from the second run on.

How fast is it? At most one request every 0.5 seconds per host, shared by all sites on that host (slower if the site sets a Crawl-delay), 3 sites in parallel. A site with one sitemap index and a few sitemap files takes a few seconds.

Do I need proxies? No. Sitemaps are published for crawlers; the actor reads them directly with a clear user agent (SitemapChangeMonitor).