Sitemap Change Monitor — New, Updated & Removed URLs

Monitor public XML sitemaps over time and emit only meaningful URL changes.

This Actor is built for scheduled monitoring, SEO audits, content inventories, migration checks, and lightweight website-change pipelines. It stores a persistent snapshot for each source and compares a new complete observation with the last good snapshot.

Why the fail-safe behavior matters

A temporary 404, timeout, broken child sitemap, response-size cap, URL cap, or sitemap-depth cap must not look like thousands of pages disappeared. This Actor therefore refuses to replace prior state when a scan is incomplete. Failed or incomplete checks are reported in RUN_SUMMARY, produce no false removed rows, and are not charged as completed site-check events.

What it does

• accepts domains, website URLs, direct sitemap URLs, and .xml.gz sitemaps;
• discovers sitemap URLs from robots.txt or common sitemap locations;
• follows sitemap indexes recursively with depth/file/URL safety caps;
• supports gzip-compressed sitemaps;
• stores the last complete snapshot in a named Apify key-value store;
• emits new, updated, and removed URL events on later runs;
• emits one summary row when a run produces no URL-change rows, so a successful no-change check is visible instead of returning an empty dataset;
• can optionally emit baseline rows on the first run;
• treats changes to sitemap lastmod, changefreq, or priority as updated;
• supports include/exclude URL regex filters;
• blocks localhost/private-network fetch targets and re-checks redirect destinations;
• uses direct HTTP only — no browser or proxy is required.

Example input

{
  "sources": ["https://example.com"],
  "stateKey": "production",
  "emitBaseline": false,
  "changeTypes": ["new", "updated", "removed"],
  "maxUrlsPerSource": 20000
}

Run the Actor again later with the same sources and stateKey. The second and later complete runs compare against the previous complete snapshot.

Output

Each emitted dataset row includes:

• event: baseline, new, updated, removed, or summary;
• url;
• current and previous lastmod, changefreq, and priority where applicable;
• source and sourceSitemap provenance;
• observedAt in UTC.

When no URL-change rows are emitted, the default dataset contains one summary row with run-level counts and status. RUN_SUMMARY also reports every requested source as complete, incomplete, or failed, plus sitemap-file count, URL count, emitted changes, response bytes, and truncation/error diagnostics.

State model

Snapshots are isolated by stateKey and source URL. A snapshot is replaced only after a complete scan. This is intentionally conservative: preserving a known-good baseline is more useful than emitting a destructive false delta from a partial observation.

If you intentionally change a source's include/exclude filters or want a clean baseline, use a new stateKey.

Pricing

Pay per completed source check, not per discovered URL. Failed and incomplete source observations are not charged as completed site-check events.

Limitations

• This monitors sitemap metadata, not page-body content. A page can change without its sitemap metadata changing.
• A site's sitemap can omit URLs that still exist; removed means removed from the monitored sitemap snapshot, not necessarily deleted from the web.
• Very large sitemap estates can exceed configured safety caps; those checks remain incomplete and preserve prior state until limits are intentionally raised within the Actor's supported bounds.
• DNS/IP validation materially reduces SSRF risk for user-supplied fetch targets, but this Actor is not a general-purpose network scanner.