# Sitemap Change Monitor: New, Removed & Broken URLs (`ferrolark/sitemap-delta`) Actor

Sitemap change monitor: compare any XML sitemap, sitemap index or robots.txt with the last snapshot. Get new URLs, removed URLs, changed lastmod and optional HTTP status checks for broken links. Built for scheduled monitoring of sitemap changes.

- **URL**: https://apify.com/ferrolark/sitemap-delta.md
- **Developed by:** [Ferrolark](https://apify.com/ferrolark) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.10 / 1,000 url rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Delta: New, Removed & Broken URLs Since Last Run

Point it at a sitemap or a website. Each run reads every sitemap (sitemap indexes, `.xml.gz` and plain-text sitemaps included), compares the URL list with the snapshot saved by the previous run and writes only what changed:

- **added**: URLs that are new in the sitemap
- **removed**: URLs that disappeared from the sitemap
- **lastmod-changed**: URLs whose `<lastmod>` date changed
- optional **HTTP status checks** to find new URLs that are already broken (4xx/5xx) or redirecting

Schedule it daily or weekly to watch your own site after deploys, track a competitor's new pages, or feed new URLs into indexing, translation or content workflows. The first run saves a baseline, and after that you pay only for changes.

### Real example

**Run 1** (no snapshot yet), with this input:

```json
{
  "urls": ["https://www.sitemaps.org/"],
  "checkStatus": "added",
  "maxStatusChecks": 20
}
```

The Actor found the sitemap through `robots.txt`, read 84 URLs and saved the baseline. It wrote 84 `baseline` rows and status-checked 20 of them (all returned 200).

- Run time: 4.5 s on Apify at 512 MB.
- Your cost: 1 sitemap + 84 rows + 20 checks = **$0.0194**.

**Run 2**, same site, a few minutes later: status `No changes. 84 URLs in 1 sitemap(s); 0 rows`.

- Run time: 4.9 s.
- Your cost: **$0.001** (one sitemap).

A dataset row:

```json
{
  "url": "https://www.sitemaps.org/da/",
  "change": "baseline",
  "lastmod": "2016-11-21",
  "sitemap": "https://www.sitemaps.org/sitemap.xml",
  "statusCode": 200,
  "broken": false,
  "snapshotName": "www.sitemaps.org",
  "checkedAt": "2026-11-08T23:16:05Z"
}
```

Rows for the other change types look like this:

```json
{"url": "https://example.com/pricing", "change": "added", "lastmod": null, "statusCode": 404, "broken": true}
{"url": "https://example.com/about", "change": "removed", "previousLastmod": null}
{"url": "https://example.com/docs", "change": "lastmod-changed", "lastmod": "2026-02-01", "previousLastmod": "2026-01-01", "statusCode": 200, "broken": false}
```

Every run also writes a summary to the `OUTPUT` record of its key-value store:

```json
{
  "snapshotName": "www.sitemaps.org",
  "snapshotSaved": true,
  "previous": {"source": "stored snapshot \"www.sitemaps.org.json.gz\"", "createdAt": "2026-11-08T23:16:05Z", "urlCount": 84},
  "sitemapsFetched": 1,
  "sitemapErrors": [],
  "urlsInSitemaps": 84,
  "changes": {},
  "urlsEmitted": 0,
  "statusChecked": 0,
  "broken": 0,
  "partial": false,
  "notes": []
}
```

### Price and limits

Pay per event, with no monthly fee:

| Event | Price |
|---|---:|
| Sitemap file downloaded and parsed (including index files) | $0.001 |
| URL row written to the dataset | $0.0001 |
| HTTP status check (optional) | $0.0005 |

Examples:

- A daily check of a 2,000-URL site with one sitemap and 5 new pages costs about $0.0015 per day.
- The baseline run for the same site costs about $0.20, or $0.001 with `firstRunOutput: "silent"`.

Sitemaps that fail to download are not charged. Set **Max total charge** on the run to cap the cost: the Actor stops writing rows when it reaches the cap.

Limits:

- Up to 500,000 URLs per run (default 50,000).
- Up to 1,000 sitemap files (default 200).
- Up to 50 MB per sitemap (gzip expansion is capped too).
- Index nesting is followed up to 4 levels.
- Up to 5,000 status checks per run (default 500).

### Input

| Field | Default | What it does |
|---|---|---|
| `urls` | required | Sitemap URLs, or website URLs. For a website, sitemaps come from `robots.txt`, with `/sitemap.xml` as fallback. |
| `snapshotName` | host of the first URL | Which saved snapshot to compare with and update. Use different names to track sections separately. |
| `compareWithStoredSnapshot` | `true` | Compare with the last snapshot saved under that name. |
| `previousSnapshotUrl` | none | Compare with a JSON file instead: a snapshot from this Actor, a JSON array of URLs, or `[{"url", "lastmod"}]`. Gzip is fine. |
| `previousDatasetId` | none | Compare with the dataset of an earlier inventory run (`baseline` or `unchanged` rows). |
| `saveSnapshot` | `true` | Save this run as the new snapshot. |
| `firstRunOutput` | `inventory` | With no previous snapshot, `inventory` writes every URL as a `baseline` row; `silent` only saves the snapshot. |
| `includeUnchanged` | `false` | Also write unchanged URLs, for a full inventory every run. |
| `checkStatus` | `none` | `added`, `changes` (added, changed and removed) or `all` rows. |
| `maxStatusChecks` | 500 | Cap on status checks. |
| `includePattern` / `excludePattern` | none | Regular expressions to keep or drop URLs, for example `/blog/`. |
| `maxUrls` / `maxSitemaps` | 50,000 / 200 | Safety limits. |

Snapshots are stored in your own Apify account, in the named key-value store `sitemap-delta-snapshots`, as gzipped JSON. You can download them or delete them there.

### How it decides what changed

- URLs are compared exactly as written in `<loc>`. `https://a.com/x` and `https://a.com/x/` are different URLs, as they are for search engines.
- `lastmod-changed` compares the raw `<lastmod>` text.
- **Partial runs are safe.** If any sitemap fails or a limit is hit:
  - `removed` rows are not reported, because they could be false positives;
  - the stored snapshot is not overwritten;
  - the `OUTPUT` record lists the errors.
- Status checks send a HEAD request, falling back to GET on 403, 405 and 501, with a 20 s timeout.
  - Redirects are reported in `redirectTo`, not followed.
  - `broken` is true for 4xx, 5xx and network errors.
- Only public addresses are fetched; private and local IPs are refused.

### Call it from code, n8n or an AI agent

**API (synchronous, returns the changed URLs):**

```bash
curl -X POST "https://api.apify.com/v2/acts/ferrolark~sitemap-delta/run-sync-get-dataset-items?maxTotalChargeUsd=0.5" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"urls": ["https://www.example.com/"], "checkStatus": "added"}'
```

**Scheduled monitoring in Apify:**

1. Create a task with your input.
2. Add a schedule, for example daily at 07:00.
3. Add an integration (Slack, email, webhook) that fires when a run succeeds.

The task keeps its `snapshotName`, so every run compares with the previous one.

**n8n:** paste this workflow on the canvas (Ctrl+V). Then add a Header Auth credential with the name `Authorization` and the value `Bearer <your Apify token>`. It runs every morning and passes only changed rows on, to a Slack, Google Sheets or email node.

```json
{
  "name": "Sitemap Delta: daily new and removed URLs",
  "nodes": [
    {"parameters": {"rule": {"interval": [{"triggerAtHour": 7}]}}, "name": "Every day 07:00", "type": "n8n-nodes-base.scheduleTrigger", "typeVersion": 1.2, "position": [0, 0]},
    {"parameters": {"method": "POST", "url": "https://api.apify.com/v2/acts/ferrolark~sitemap-delta/run-sync-get-dataset-items?maxTotalChargeUsd=0.5", "authentication": "genericCredentialType", "genericAuthType": "httpHeaderAuth", "sendBody": true, "specifyBody": "json", "jsonBody": "{\"urls\": [\"https://www.example.com/\"], \"firstRunOutput\": \"silent\", \"checkStatus\": \"added\"}", "options": {"timeout": 300000}}, "name": "Run Sitemap Delta", "type": "n8n-nodes-base.httpRequest", "typeVersion": 4.2, "position": [220, 0]},
    {"parameters": {"conditions": {"string": [{"value1": "={{ $json.change }}", "operation": "notEqual", "value2": "unchanged"}]}}, "name": "Only changes", "type": "n8n-nodes-base.if", "typeVersion": 1, "position": [440, 0]}
  ],
  "connections": {
    "Every day 07:00": {"main": [[{"node": "Run Sitemap Delta", "type": "main", "index": 0}]]},
    "Run Sitemap Delta": {"main": [[{"node": "Only changes", "type": "main", "index": 0}]]}
  }
}
```

**Make / Zapier:** use the Apify app's "Run Actor" action with `ferrolark/sitemap-delta` and the JSON input above, then "Get dataset items".

**AI agents (MCP):** add `ferrolark/sitemap-delta` to the Apify MCP server (`https://mcp.apify.com?actors=ferrolark/sitemap-delta`). The agent can then ask "what pages did example.com add this week?"

### Use cases

- **After every deploy:**
  - check that no important URL dropped out of the sitemap;
  - check that new URLs return 200 (`checkStatus: "changes"`).
- **Competitor watch:** new product, pricing, landing or blog pages as soon as they reach the sitemap.
- **SEO and indexing:** send `added` URLs to an indexing or translation queue, and `removed` URLs to a redirect review.
- **Content inventory:** `includeUnchanged: true` gives a full, dated URL inventory you can export to CSV or Excel.

### Troubleshooting

- **"no sitemap could be read"**: the site has no `Sitemap:` line in `robots.txt` and no `/sitemap.xml`. Pass the sitemap URL directly. You were not charged.
- **"Got an HTML page, not a sitemap"**: the URL returns a web page, often a soft 404. Check the URL in a browser.
- **Everything shows as `baseline` again**: the `snapshotName` changed, or the snapshot store was deleted. Set `snapshotName` explicitly in scheduled tasks.
- **No `removed` rows although pages were deleted**: the run was partial (see `notes` in `OUTPUT`). Fix the failing sitemap, or raise `maxUrls` and `maxSitemaps`.
- **Many `403` statuses**: some sites block automated HEAD and GET requests. The sitemap diff is still correct; only the status columns are affected.

### Limitations

- Reads sitemaps only. It does not crawl pages to discover URLs that are missing from the sitemap.
- Does not render JavaScript. Sitemaps are static files, so this is rarely an issue.
- `<lastmod>` is only as reliable as the site's CMS. Some sites update it on every build.
- News, image and video sitemap extensions are read for `<loc>` and `<lastmod>` only.

### Changelog

- **1.0** (November 2026): first release. Sitemap index, gzip and text sitemaps, `robots.txt` discovery, stored snapshots, partial-run protection and optional status checks.

### More Ferrolark tools

- [Brand Name Availability Checker](https://apify.com/ferrolark/brand-name-checker): domains, GitHub, npm, PyPI and more in one report.
- [Domain Intelligence Report](https://apify.com/ferrolark/domain-intelligence-report): WHOIS, DNS, email security, SSL and hosting for any domain.
- [Launch Asset Pack](https://apify.com/ferrolark/launch-asset-pack): favicons, PWA icons, OG image and head tags from one logo.
- [Live Deploy QA](https://apify.com/ferrolark/live-deploy-qa): meta tags, canonical, robots, OG image and favicon of your deployed routes, with diffs between deploys.
- [RSS Delta](https://apify.com/ferrolark/rss-delta): only new RSS, Atom and JSON Feed items, de-duplicated.
- [PDF Merge API](https://apify.com/ferrolark/pdf-merge-api): merge PDFs from URLs or base64 with page ranges and a SHA-256 manifest.
- [PDF Split API](https://apify.com/ferrolark/pdf-split-api): split a PDF by page ranges or every N pages, or extract pages, with a ZIP and manifest.
- [Image Convert](https://apify.com/ferrolark/image-convert): PNG, JPG, WebP, AVIF and HEIC conversion (SVG input too) from URLs or base64.
- [Image Compress & Resize](https://apify.com/ferrolark/image-compress-resize): resize images and compress them to a target size in KB, WebP or AVIF.
- [PDF to Image](https://apify.com/ferrolark/pdf-to-image): PDF pages to PNG, JPG or WebP at the DPI you choose.
- [Image to PDF](https://apify.com/ferrolark/image-to-pdf): JPG, PNG, HEIC and WebP images to one PDF, JPEGs embedded without recompression.
- [PDF Compress](https://apify.com/ferrolark/pdf-compress): smaller PDFs, lossless or with image downsampling; charged only when smaller.
- [HTML to PDF & URL to PDF](https://apify.com/ferrolark/html-to-pdf): web pages or raw HTML to PDF with headless Chrome.
- [Markdown to PDF](https://apify.com/ferrolark/markdown-to-pdf): Markdown with tables, code and footnotes to a clean PDF.
- [DOCX to PDF](https://apify.com/ferrolark/docx-to-pdf): Word, Excel, PowerPoint and ODT to PDF with LibreOffice.
- [Data Convert](https://apify.com/ferrolark/data-convert): CSV, JSON, NDJSON and Excel XLSX in any direction.

***

*Built by Ferrolark.*

# Actor input Schema

## `urls` (type: `array`):

Sitemap URLs (XML, sitemap index, .xml.gz or plain-text) or website URLs. For a website URL the sitemaps are discovered from robots.txt, with /sitemap.xml as fallback.

## `snapshotName` (type: `string`):

Name of the saved snapshot to compare against and update. Defaults to the host of the first URL. Use different names to track different sections of the same site.

## `compareWithStoredSnapshot` (type: `boolean`):

Compare with the snapshot saved by the previous run under the same name (stored in your account, key-value store "sitemap-delta-snapshots").

## `previousSnapshotUrl` (type: `string`):

Compare against a JSON file instead: a snapshot from this Actor, a JSON array of URLs, or an array of {url, lastmod} objects. Gzip is fine.

## `previousDatasetId` (type: `string`):

Compare against the dataset of an earlier run (baseline/unchanged/added rows are used as the previous URL list).

## `saveSnapshot` (type: `boolean`):

Store the current URL list for the next run. Partial runs (sitemap errors or limits) never overwrite the snapshot.

## `firstRunOutput` (type: `string`):

When there is no previous snapshot: 'inventory' writes every URL as a 'baseline' row (charged per row); 'silent' only saves the snapshot (you pay only for sitemaps).

## `includeUnchanged` (type: `boolean`):

Also output URLs that did not change (charged per row).

## `checkStatus` (type: `string`):

Send a HEAD request (GET fallback) to find broken URLs. Redirects are reported, not followed. Charged per checked URL.

## `maxStatusChecks` (type: `integer`):

Upper bound on HTTP status checks per run.

## `includePattern` (type: `string`):

Only keep URLs matching this regular expression, for example /blog/.

## `excludePattern` (type: `string`):

Drop URLs matching this regular expression, for example ?page=.

## `maxUrls` (type: `integer`):

Stop collecting after this many unique URLs. Hitting the limit makes the run partial.

## `maxSitemaps` (type: `integer`):

Upper bound on sitemap files downloaded (index files count too).

## Actor input object example

```json
{
  "urls": [
    "https://www.sitemaps.org/sitemap.xml"
  ],
  "compareWithStoredSnapshot": true,
  "saveSnapshot": true,
  "firstRunOutput": "inventory",
  "includeUnchanged": false,
  "checkStatus": "none",
  "maxStatusChecks": 500,
  "maxUrls": 50000,
  "maxSitemaps": 200
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.sitemaps.org/sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("ferrolark/sitemap-delta").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://www.sitemaps.org/sitemap.xml"] }

# Run the Actor and wait for it to finish
run = client.actor("ferrolark/sitemap-delta").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.sitemaps.org/sitemap.xml"
  ]
}' |
apify call ferrolark/sitemap-delta --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ferrolark/sitemap-delta"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/LxZRkLdEfjN6X0d8v/builds/chABhe3849QhkDFj2/openapi.json
