Sitemap Change Monitor: New, Removed & Broken URLs avatar

Sitemap Change Monitor: New, Removed & Broken URLs

Pricing

from $0.10 / 1,000 url rows

Go to Apify Store
Sitemap Change Monitor: New, Removed & Broken URLs

Sitemap Change Monitor: New, Removed & Broken URLs

Sitemap change monitor: compare any XML sitemap, sitemap index or robots.txt with the last snapshot. Get new URLs, removed URLs, changed lastmod and optional HTTP status checks for broken links. Built for scheduled monitoring of sitemap changes.

Pricing

from $0.10 / 1,000 url rows

Rating

0.0

(0)

Developer

Ferrolark

Ferrolark

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

4 days ago

Last modified

Features

Limited permissions

Share

Sitemap Delta: New, Removed & Broken URLs Since Last Run

Point it at a sitemap or a website. Each run reads every sitemap (sitemap indexes, .xml.gz and plain-text sitemaps included), compares the URL list with the snapshot saved by the previous run and writes only what changed:

  • added: URLs that are new in the sitemap
  • removed: URLs that disappeared from the sitemap
  • lastmod-changed: URLs whose <lastmod> date changed
  • optional HTTP status checks to find new URLs that are already broken (4xx/5xx) or redirecting

Schedule it daily or weekly to watch your own site after deploys, track a competitor's new pages, or feed new URLs into indexing, translation or content workflows. The first run saves a baseline, and after that you pay only for changes.

Real example

Run 1 (no snapshot yet), with this input:

{
"urls": ["https://www.sitemaps.org/"],
"checkStatus": "added",
"maxStatusChecks": 20
}

The Actor found the sitemap through robots.txt, read 84 URLs and saved the baseline. It wrote 84 baseline rows and status-checked 20 of them (all returned 200).

  • Run time: 4.5 s on Apify at 512 MB.
  • Your cost: 1 sitemap + 84 rows + 20 checks = $0.0194.

Run 2, same site, a few minutes later: status No changes. 84 URLs in 1 sitemap(s); 0 rows.

  • Run time: 4.9 s.
  • Your cost: $0.001 (one sitemap).

A dataset row:

{
"url": "https://www.sitemaps.org/da/",
"change": "baseline",
"lastmod": "2016-11-21",
"sitemap": "https://www.sitemaps.org/sitemap.xml",
"statusCode": 200,
"broken": false,
"snapshotName": "www.sitemaps.org",
"checkedAt": "2026-11-08T23:16:05Z"
}

Rows for the other change types look like this:

{"url": "https://example.com/pricing", "change": "added", "lastmod": null, "statusCode": 404, "broken": true}
{"url": "https://example.com/about", "change": "removed", "previousLastmod": null}
{"url": "https://example.com/docs", "change": "lastmod-changed", "lastmod": "2026-02-01", "previousLastmod": "2026-01-01", "statusCode": 200, "broken": false}

Every run also writes a summary to the OUTPUT record of its key-value store:

{
"snapshotName": "www.sitemaps.org",
"snapshotSaved": true,
"previous": {"source": "stored snapshot \"www.sitemaps.org.json.gz\"", "createdAt": "2026-11-08T23:16:05Z", "urlCount": 84},
"sitemapsFetched": 1,
"sitemapErrors": [],
"urlsInSitemaps": 84,
"changes": {},
"urlsEmitted": 0,
"statusChecked": 0,
"broken": 0,
"partial": false,
"notes": []
}

Price and limits

Pay per event, with no monthly fee:

EventPrice
Sitemap file downloaded and parsed (including index files)$0.001
URL row written to the dataset$0.0001
HTTP status check (optional)$0.0005

Examples:

  • A daily check of a 2,000-URL site with one sitemap and 5 new pages costs about $0.0015 per day.
  • The baseline run for the same site costs about $0.20, or $0.001 with firstRunOutput: "silent".

Sitemaps that fail to download are not charged. Set Max total charge on the run to cap the cost: the Actor stops writing rows when it reaches the cap.

Limits:

  • Up to 500,000 URLs per run (default 50,000).
  • Up to 1,000 sitemap files (default 200).
  • Up to 50 MB per sitemap (gzip expansion is capped too).
  • Index nesting is followed up to 4 levels.
  • Up to 5,000 status checks per run (default 500).

Input

FieldDefaultWhat it does
urlsrequiredSitemap URLs, or website URLs. For a website, sitemaps come from robots.txt, with /sitemap.xml as fallback.
snapshotNamehost of the first URLWhich saved snapshot to compare with and update. Use different names to track sections separately.
compareWithStoredSnapshottrueCompare with the last snapshot saved under that name.
previousSnapshotUrlnoneCompare with a JSON file instead: a snapshot from this Actor, a JSON array of URLs, or [{"url", "lastmod"}]. Gzip is fine.
previousDatasetIdnoneCompare with the dataset of an earlier inventory run (baseline or unchanged rows).
saveSnapshottrueSave this run as the new snapshot.
firstRunOutputinventoryWith no previous snapshot, inventory writes every URL as a baseline row; silent only saves the snapshot.
includeUnchangedfalseAlso write unchanged URLs, for a full inventory every run.
checkStatusnoneadded, changes (added, changed and removed) or all rows.
maxStatusChecks500Cap on status checks.
includePattern / excludePatternnoneRegular expressions to keep or drop URLs, for example /blog/.
maxUrls / maxSitemaps50,000 / 200Safety limits.

Snapshots are stored in your own Apify account, in the named key-value store sitemap-delta-snapshots, as gzipped JSON. You can download them or delete them there.

How it decides what changed

  • URLs are compared exactly as written in <loc>. https://a.com/x and https://a.com/x/ are different URLs, as they are for search engines.
  • lastmod-changed compares the raw <lastmod> text.
  • Partial runs are safe. If any sitemap fails or a limit is hit:
    • removed rows are not reported, because they could be false positives;
    • the stored snapshot is not overwritten;
    • the OUTPUT record lists the errors.
  • Status checks send a HEAD request, falling back to GET on 403, 405 and 501, with a 20 s timeout.
    • Redirects are reported in redirectTo, not followed.
    • broken is true for 4xx, 5xx and network errors.
  • Only public addresses are fetched; private and local IPs are refused.

Call it from code, n8n or an AI agent

API (synchronous, returns the changed URLs):

curl -X POST "https://api.apify.com/v2/acts/ferrolark~sitemap-delta/run-sync-get-dataset-items?maxTotalChargeUsd=0.5" \
-H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
-d '{"urls": ["https://www.example.com/"], "checkStatus": "added"}'

Scheduled monitoring in Apify:

  1. Create a task with your input.
  2. Add a schedule, for example daily at 07:00.
  3. Add an integration (Slack, email, webhook) that fires when a run succeeds.

The task keeps its snapshotName, so every run compares with the previous one.

n8n: paste this workflow on the canvas (Ctrl+V). Then add a Header Auth credential with the name Authorization and the value Bearer <your Apify token>. It runs every morning and passes only changed rows on, to a Slack, Google Sheets or email node.

{
"name": "Sitemap Delta: daily new and removed URLs",
"nodes": [
{"parameters": {"rule": {"interval": [{"triggerAtHour": 7}]}}, "name": "Every day 07:00", "type": "n8n-nodes-base.scheduleTrigger", "typeVersion": 1.2, "position": [0, 0]},
{"parameters": {"method": "POST", "url": "https://api.apify.com/v2/acts/ferrolark~sitemap-delta/run-sync-get-dataset-items?maxTotalChargeUsd=0.5", "authentication": "genericCredentialType", "genericAuthType": "httpHeaderAuth", "sendBody": true, "specifyBody": "json", "jsonBody": "{\"urls\": [\"https://www.example.com/\"], \"firstRunOutput\": \"silent\", \"checkStatus\": \"added\"}", "options": {"timeout": 300000}}, "name": "Run Sitemap Delta", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.2, "position": [220, 0]},
{"parameters": {"conditions": {"string": [{"value1": "={{ $json.change }}", "operation": "notEqual", "value2": "unchanged"}]}}, "name": "Only changes", "type": "n8n-nodes-base.if", "typeVersion": 1, "position": [440, 0]}
],
"connections": {
"Every day 07:00": {"main": [[{"node": "Run Sitemap Delta", "type": "main", "index": 0}]]},
"Run Sitemap Delta": {"main": [[{"node": "Only changes", "type": "main", "index": 0}]]}
}
}

Make / Zapier: use the Apify app's "Run Actor" action with ferrolark/sitemap-delta and the JSON input above, then "Get dataset items".

AI agents (MCP): add ferrolark/sitemap-delta to the Apify MCP server (https://mcp.apify.com?actors=ferrolark/sitemap-delta). The agent can then ask "what pages did example.com add this week?"

Use cases

  • After every deploy:
    • check that no important URL dropped out of the sitemap;
    • check that new URLs return 200 (checkStatus: "changes").
  • Competitor watch: new product, pricing, landing or blog pages as soon as they reach the sitemap.
  • SEO and indexing: send added URLs to an indexing or translation queue, and removed URLs to a redirect review.
  • Content inventory: includeUnchanged: true gives a full, dated URL inventory you can export to CSV or Excel.

Troubleshooting

  • "no sitemap could be read": the site has no Sitemap: line in robots.txt and no /sitemap.xml. Pass the sitemap URL directly. You were not charged.
  • "Got an HTML page, not a sitemap": the URL returns a web page, often a soft 404. Check the URL in a browser.
  • Everything shows as baseline again: the snapshotName changed, or the snapshot store was deleted. Set snapshotName explicitly in scheduled tasks.
  • No removed rows although pages were deleted: the run was partial (see notes in OUTPUT). Fix the failing sitemap, or raise maxUrls and maxSitemaps.
  • Many 403 statuses: some sites block automated HEAD and GET requests. The sitemap diff is still correct; only the status columns are affected.

Limitations

  • Reads sitemaps only. It does not crawl pages to discover URLs that are missing from the sitemap.
  • Does not render JavaScript. Sitemaps are static files, so this is rarely an issue.
  • <lastmod> is only as reliable as the site's CMS. Some sites update it on every build.
  • News, image and video sitemap extensions are read for <loc> and <lastmod> only.

Changelog

  • 1.0 (November 2026): first release. Sitemap index, gzip and text sitemaps, robots.txt discovery, stored snapshots, partial-run protection and optional status checks.

More Ferrolark tools

  • Brand Name Availability Checker: domains, GitHub, npm, PyPI and more in one report.
  • Domain Intelligence Report: WHOIS, DNS, email security, SSL and hosting for any domain.
  • Launch Asset Pack: favicons, PWA icons, OG image and head tags from one logo.
  • Live Deploy QA: meta tags, canonical, robots, OG image and favicon of your deployed routes, with diffs between deploys.
  • RSS Delta: only new RSS, Atom and JSON Feed items, de-duplicated.
  • PDF Merge API: merge PDFs from URLs or base64 with page ranges and a SHA-256 manifest.
  • PDF Split API: split a PDF by page ranges or every N pages, or extract pages, with a ZIP and manifest.
  • Image Convert: PNG, JPG, WebP, AVIF and HEIC conversion (SVG input too) from URLs or base64.
  • Image Compress & Resize: resize images and compress them to a target size in KB, WebP or AVIF.
  • PDF to Image: PDF pages to PNG, JPG or WebP at the DPI you choose.
  • Image to PDF: JPG, PNG, HEIC and WebP images to one PDF, JPEGs embedded without recompression.
  • PDF Compress: smaller PDFs, lossless or with image downsampling; charged only when smaller.
  • HTML to PDF & URL to PDF: web pages or raw HTML to PDF with headless Chrome.
  • Markdown to PDF: Markdown with tables, code and footnotes to a clean PDF.
  • DOCX to PDF: Word, Excel, PowerPoint and ODT to PDF with LibreOffice.
  • Data Convert: CSV, JSON, NDJSON and Excel XLSX in any direction.

Built by Ferrolark.