# Sitemap Change Monitor – New, Updated & Removed Pages (`quietjoy/sitemap-change-monitor`) Actor

Watch any websites' XML sitemaps and get only the pages that were added, updated (lastmod) or removed since the last run. Auto-discovers sitemaps via robots.txt, follows sitemap indexes and .gz files.

- **URL**: https://apify.com/quietjoy/sitemap-change-monitor.md
- **Developed by:** [Moduvant Systems](https://apify.com/quietjoy) (community)
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 changed urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Change Monitor – new, updated & removed pages

**Sitemap Change Monitor** watches the XML sitemaps of any websites and tells you which pages were **added**, **updated** (their `<lastmod>` changed) or **removed** since the last run. Schedule it daily or weekly to get a clean feed of what competitors, publishers, docs sites or your own site actually changed – without crawling every page.

- 🔎 **Auto-discovers sitemaps** from `robots.txt`, then `/sitemap.xml` and `/sitemap_index.xml`
- 🗂️ Follows **sitemap indexes**, gzipped `.xml.gz` sitemaps, news/image sitemaps and plain-text sitemaps
- 🆕 **Only changes mode** – remembers every site's URLs in a named key-value store and outputs only the difference
- 🎯 Include / exclude URL patterns (e.g. only `/blog/`)
- 🛡️ No false "removed" alerts: removal detection is skipped when a sitemap fails to load or the URL limit is hit
- No login, no API keys, no browser – just HTTP requests to public sitemap files

### Input

| Field | Description |
|---|---|
| `sites` | Homepages (`https://example.com`) or direct sitemap URLs. Empty = demo (`https://crawlee.dev`). |
| `onlyChanges` | Output only new / updated / removed URLs (default **on**). Off = every URL with a `changeType`. |
| `outputBaselineUrls` | On the first run of a site, output all URLs as `new` (default on). Turn off to record the baseline silently. |
| `reportRemoved` | Report URLs that disappeared from the sitemaps (default on). |
| `trackLastmod` | Report `updated` when `<lastmod>` changes (default on). |
| `includeUrlRegex` / `excludeUrlRegex` | Optional regex filters on page URLs. |
| `maxUrlsPerSite` | Default 10,000 (max 100,000). |
| `maxSitemapsPerSite` | Default 200 sitemap files. |
| `stateStoreName` | Named key-value store with each site's previous URLs – use one name per monitor. |

```json
{
  "sites": ["https://competitor.com", "https://example.com/sitemap_index.xml"],
  "onlyChanges": true,
  "includeUrlRegex": "/blog/|/pricing",
  "maxUrlsPerSite": 20000
}
```

### Output

One item per changed URL:

```json
{
  "site": "https://www.gov.uk",
  "url": "https://www.gov.uk/hmrc-internal-manuals/advance-tax-certainty-service/atcs02100",
  "changeType": "updated",
  "lastmod": "2026-09-30T10:48:14+01:00",
  "previousLastmod": "2026-08-12T09:00:00+01:00",
  "sitemapUrl": "https://www.gov.uk/sitemaps/sitemap_1.xml",
  "isBaseline": false,
  "detectedAt": "2026-11-02T17:40:01.000Z"
}
```

`changeType` is `new`, `updated`, `removed` (or `unchanged` when *Only output changes* is off). A `RUN_SUMMARY` record in the default key-value store has per-site counts and sitemap errors.

### Use cases

- **Competitor monitoring** – new product, landing, pricing or blog pages the day they go live
- **SEO** – check your own sitemaps after each release: pages accidentally dropped or added
- **Content & news tracking** – feed new articles from publishers' news sitemaps into Slack, email, Google Sheets or an LLM summariser (via Apify integrations / webhooks)
- **Docs & changelog watching** – know when a vendor publishes new documentation
- **Lead generation** – new location, job or listing pages on directory sites

### Tips

- Run on a **schedule** (Apify Schedules) with the same `stateStoreName`; connect a Slack / email integration to the run's dataset.
- Very large sites: raise `maxUrlsPerSite` and memory, or point the Actor at one specific child sitemap.
- Sites that rewrite every `<lastmod>` on each deploy: turn off `trackLastmod`.

### Limitations

- Only sees pages that the site lists in its sitemaps; it does not crawl links or compare page content.
- `updated` relies on the site's own `<lastmod>` values, which some sites omit or set inaccurately.
- Sites that block bots or require JavaScript to serve sitemaps are not supported.

### Legal note

The Actor downloads only **publicly published sitemap files** (which sites publish specifically for automated discovery) and respects their URLs as given; it does not log in, bypass protection or collect personal data. You are responsible for using the results in line with the target sites' terms and applicable law.

# Actor input Schema

## `sites` (type: `array`):

Website homepages (e.g. `https://crawlee.dev` – sitemaps are discovered from robots.txt, then /sitemap.xml) or direct sitemap / sitemap-index URLs (e.g. `https://example.com/sitemap_index.xml`). If empty, a demo site is used.

## `onlyChanges` (type: `boolean`):

Output only new, updated and removed URLs compared with the previous run of the same site (stored in the named key-value store below). The first run of a site is a baseline and outputs all URLs as `new` with `isBaseline: true`. Turn off to output every URL each run (unchanged ones get `changeType: unchanged`).

## `outputBaselineUrls` (type: `boolean`):

When a site is checked for the first time, output all its URLs as `new`. Turn off to record the baseline silently, so the first run outputs nothing and later runs output only real changes (used with Only output changes).

## `reportRemoved` (type: `boolean`):

Output URLs that were in the previous run's sitemaps but are gone now. Skipped automatically when a sitemap failed to load or the URL limit was hit, to avoid false alarms.

## `trackLastmod` (type: `boolean`):

Report a URL as `updated` when its `<lastmod>` date changes. Turn off for sites whose lastmod changes on every build.

## `includeUrlRegex` (type: `string`):

Optional case-insensitive regular expression; only matching page URLs are tracked, e.g. `/blog/|/news/`.

## `excludeUrlRegex` (type: `string`):

Optional case-insensitive regular expression for page URLs to ignore, e.g. `/tag/|/author/`.

## `maxUrlsPerSite` (type: `integer`):

Stop collecting after this many page URLs per site. When the limit is hit, removed-URL detection is skipped for that site.

## `maxSitemapsPerSite` (type: `integer`):

Maximum number of sitemap / sitemap-index files to download per site.

## `stateStoreName` (type: `string`):

Name of the key-value store that remembers each site's URLs between runs. Use a different name for separate monitors.

## `requestDelayMs` (type: `integer`):

Polite pause between sitemap downloads.

## Actor input object example

```json
{
  "sites": [
    "https://crawlee.dev"
  ],
  "onlyChanges": true,
  "outputBaselineUrls": true,
  "reportRemoved": true,
  "trackLastmod": true,
  "includeUrlRegex": "",
  "excludeUrlRegex": "",
  "maxUrlsPerSite": 10000,
  "maxSitemapsPerSite": 200,
  "stateStoreName": "sitemap-change-monitor-state",
  "requestDelayMs": 300
}
```

# Actor output Schema

## `results` (type: `string`):

New, updated and removed sitemap URLs (one item per URL).

## `runSummary` (type: `string`):

Per-site counts (new / updated / removed / unchanged) and sitemap errors (RUN_SUMMARY record in the default key-value store).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "https://crawlee.dev"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("quietjoy/sitemap-change-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sites": ["https://crawlee.dev"] }

# Run the Actor and wait for it to finish
run = client.actor("quietjoy/sitemap-change-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "https://crawlee.dev"
  ]
}' |
apify call quietjoy/sitemap-change-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,quietjoy/sitemap-change-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0TSGwZNCzRX2PoXJR/builds/QvUzXs1SDEs6SvOGb/openapi.json