# Sitemap Change Monitor: New, Removed and Changed URLs (`frostpine456/sitemap-monitor`) Actor

Read a site's robots.txt and sitemaps. Extract every page URL, or monitor and get only the new, removed and modified (by lastmod) pages since your last run. No page content is fetched, only sitemap files. Pay per result: $0.30 per 1,000 URLs, $2 per 1,000 changes.

- **URL**: https://apify.com/frostpine456/sitemap-monitor.md
- **Developed by:** [M J](https://apify.com/frostpine456) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Change Monitor: detect new, removed and modified pages on any website

Monitor a website's sitemap and get only the **new, removed and modified pages** since your last run, or use it as a sitemap URL extractor that lists every page URL a site publishes. It reads the site's robots.txt and sitemaps, requests only robots.txt and sitemap URLs, abandons any HTML response unread, and **no page content is stored or output**. Pay per result, with no charge for starting a run: $0.30 per 1,000 `url` records and $2.00 per 1,000 `change` records.

### When to use it, and when not to

**Use it when you want to:**

- watch a competitor's site, docs site, shop or news site for new pages on a daily schedule;
- extract every URL from a sitemap (with `lastmod`, `changefreq` and `priority`) for SEO audits or crawl lists;
- track pages removed from a site without false alarms from failed, truncated or rolling sitemaps.

**Pick something else when you need:**

- the text, prices or other content of pages: this Actor never reads page content, and `lastmod` is its only signal for modified pages;
- sites with no XML, gzip or plain-text sitemap (RSS/Atom and HTML sitemaps are not read, and a site with no sitemap is reported `failed` and billed nothing);
- anything behind a login, or built-in alerts: schedule it and connect Apify's own notifications instead.

### What it does

- **Extract mode:** every page URL listed, with `lastmod`, `changefreq`, `priority` and the sitemap it came from.
- **Monitor mode:** remembers each sitemap between runs and outputs `new`, `removed` and `modified` pages.
- Built for messy real sitemaps: an unchanged site gives zero changes on its second run; a failed, truncated, dead or rolling sitemap never produces false `removed` records; deploy stamps and host migrations are not billed as whole-site changes.
- Only robots.txt and sitemap URLs are requested. If one of them answers with an HTML page, it is abandoned unread (the sitemap is reported as `unsupported format`), and no page content is stored or output.

### Use cases

- **Competitor new-page alerts:** `{ "sites": ["competitor.example"], "mode": "monitor", "maxChanges": 500 }` — schedule daily and connect Apify's own notifications (email, Slack, webhook). The Actor has no alerts of its own.
- **Docs change feed:** `{ "sites": ["https://docs.example.com/sitemap.xml"], "mode": "monitor" }` — a page counts as modified only when its `lastmod` changes; if a site lists no lastmod, its pages can be new or removed but never modified.
- **E-commerce new-product watch:** `{ "sites": ["https://shop.example.com/sitemap_products.xml"], "mode": "monitor", "stateKey": "shop-products" }` — product URLs only; no page content, prices or stock are read.
- **News sitemap monitor:** `{ "sites": ["https://news.example.com/news-sitemap.xml"], "mode": "monitor", "rollingSitemaps": ["news-sitemap"] }` — news sitemaps are rolling, so articles that age out are not billed as removed.
- **Plain sitemap URL extractor:** `{ "sites": ["example.com"], "mode": "extract" }` — extract mode keeps no state and bills each URL as a `url` event on every run.

### Use it from Claude, Cursor or any AI agent

Add Sitemap Change Monitor as a tool through Apify's MCP server: connect your MCP client to https://mcp.apify.com?tools=frostpine456/sitemap-monitor with your Apify API token. Then ask things like "List every URL in this site's sitemap" (extract mode) or "What pages did this site add since last week?" (monitor mode with the same `stateKey`). For quick answers keep `maxUrlsPerSite` small; the same prices apply.

### Input

| Field | Type | Default | Meaning |
|---|---|---|---|
| `sites` | string\[], 1–50 | required | Each item is a **domain** (`example.com`), a **site URL** reduced to its host (`https://example.com/any/path`), or a **direct sitemap URL** whose path ends in `.xml`, `.xml.gz`, `.gz` or `.txt` |
| `mode` | `"monitor"` | `"extract"` | `"monitor"` | See above |
| `maxUrlsPerSite` | integer 1–50,000 | 10,000 | URLs read per site per run (de-duplicated canonical URLs); a larger site is `truncated` |
| `maxChanges` | integer 1–20,000 | 2,000 | `change`-billed records per run across all sites; the rest are deferred to the next run, and sites after the cap is reached still get their `baseline` (and `existing`) records |
| `includeExisting` | boolean | false | Monitor mode: also output unchanged URLs, billed as `url` **on every run** |
| `includeProfileUrls` | boolean | false | Include sitemaps and URLs that look like people's profile pages |
| `rollingSitemaps` | string\[] | `[]` | Extra substrings that mark a sitemap as rolling when found in its URL's path or query |
| `stateKey` | string `[a-z0-9-]{1,40}` | `"default"` | Names the saved state; changing it starts a new **paid baseline** |
| `shrinkGuardPercent` | integer 10–100 | 50 | Per-sitemap drop allowed before removals are held |

The Console example input runs **extract** mode on `https://docs.apify.com` with `maxUrlsPerSite` 100, so every try returns URLs (about $0.03 per run). The default mode is still `monitor`: pick it, or leave `mode` unset in the API, to track changes between runs.

### Output

One dataset record per URL: `site`, `url`, `lastmod`, `changefreq`, `priority`, `sitemapUrl`, `fetchedAt`, and in monitor mode `changeType` (`baseline`, `new`, `removed`, `modified`, `existing`), `previousLastmod` (for `modified`) and `firstSeenAt`. No person data and no page content.

The free `SUMMARY` key-value record reports each site's status (`ok`, `partial`, `truncated`, `failed`, `skipped`), each sitemap's status and reason, per-site counts, flags (for example `shrinkSuspected`, `stateWasInvalid`) and per-section change counts (first path segment, top 50).

### Pricing

Pay per event, no start event:

| Event | Price | Records |
|---|---|---|
| `url` | $0.30 / 1,000 | extract-mode records; `baseline` records; `existing` records |
| `change` | $2.00 / 1,000 | `new`, `removed`, `modified` |

Items that fail before any record is output are free, and so are failed sitemaps. An item that fails while writing records or state may have billed records, which the next run can repeat (at-least-once). `SUMMARY` is free.

**The first monitor run is a paid baseline, later runs bill only changes.** Every URL of a sitemap is output as `baseline` and billed as `url` the first time it is seen. Changing `stateKey` starts a new paid baseline.

Worked example (a hypothetical site with 5,000 URLs, run daily):

| Run | Records | Cost |
|---|---|---|
| Run 1 (baseline) | 5,000 `baseline` = 5,000 `url` events | 5 × $0.30 = **$1.50** |
| Run 2 | 12 `new` + 3 `removed` + 5 `modified` = 20 `change` events | 20 / 1,000 × $2.00 = **$0.04** |

On this example the daily cost after the first run is about **$0.04 per day**. The site sizes and change counts here are invented for arithmetic, not measured results.

`includeExisting` caveat: with `includeExisting: true`, the same run 2 would also output 4,992 unchanged URLs as `existing`, billed as `url` **on every run**: 4,992 / 1,000 × $0.30 ≈ $1.50 plus $0.04 of changes, so about **$1.54 per day instead of $0.04**. Leave it off for scheduled runs unless you need the full list each time.

Extract mode: 5,000 URLs cost the same $1.50 **per run**, because every record is a `url` event.

### Scheduling and state

Save the input as a Task and add a daily Schedule. Keep the same `stateKey` between runs; use a different `stateKey` per independent list. **Concurrent runs on one `stateKey` can bill the same changes twice**, so do not overlap them. Delivery is at-least-once: if a run crashes between pushing records and writing state, the next run can repeat those records.

### How it protects you from false changes

- **Held removals:** while any sitemap of a site is not fully read (failed, disallowed, truncated, cut off, or recently dropped from its index), no sitemap of that site emits removals, because a URL may have moved into the unread one. A sitemap treated as gone (dead or dropped for good) no longer holds the others.
- **Shrink guard:** if a complete read drops by more than `shrinkGuardPercent` (default 50%) of the last confirmed count, removals are held until a second consistent run confirms them.
- **Rolling sitemaps** (news sitemaps, or any sitemap whose path or query, not its host, contains `news` or one of your `rollingSitemaps` substrings) are exempt from removals: URLs that age out emit nothing.
- **Dead sitemaps** count as gone only after 3 runs answering 404 or 410, or after 10 runs failing any other way (server errors, rate limits, timeouts, network errors, oversized or malformed bodies); a full read resets both counts. Only then are their URLs emitted as `removed`, once. A sitemap that was never read in full drops its URLs silently when it goes.
- **Deploy-stamp guard:** a mass lastmod update to one identical value (a redeploy stamp) is not billed as thousands of `modified` records; it is reported in `SUMMARY` as `stampedLastmod`.
- **Host-migration guard:** when the same sitemap file starts listing its pages on a new host, and every removed URL matches a new one once scheme and a leading `www.` are ignored, neither set is billed; `SUMMARY` reports `hostMigrated`. A host change that also moves the sitemap file to a new URL is not covered: the new sitemap starts a new paid baseline, and the old sitemap's URLs are reported `removed` once it is treated as gone.
- **Profile pages:** profile pages — a URL whose path is exactly `/users/<name>`, `/profile/<name>` or the like (`users`, `user`, `people`, `profile`, `profiles`, `author`, `authors`, `member`, `members`, case-insensitive) — and child sitemaps with one of those words as a whole token of a path segment (each segment is percent-decoded and split on any non-letter, dots and digits included, e.g. `sitemap.users.xml`, `members.xml.gz`, `sitemap-authors1.xml` or `author-sitemap.xml`; a word fused to other letters, like `userprofiles.xml`, does not count) are skipped by default (turn on `includeProfileUrls` to include them). Switching it either way never reports profile pages as removed or new: skipped ones stay remembered, and ones included for the first time come out as `baseline`.

### Limits

- Up to 50 sites per run; 10,000 URLs per site by default — raise `maxUrlsPerSite` (up to 50,000) if `SUMMARY` shows `truncated`.
- **Truncation:** while a site is truncated, its cut-off and unread sitemaps report no new, modified or removed pages. In a cut-off sitemap that has never been read in full, newly seen URLs come out as `baseline`; in one that has, they are held and come out as `new` (billed as a change) once it is read in full. Raise `maxUrlsPerSite` for a real change feed.
- 20 MB decoded per sitemap file; XML (urlset and sitemap index), gzip and plain-text sitemaps only — no RSS/Atom or HTML sitemaps.
- Sitemap indexes are followed to depth 3, with at most 500 child sitemaps per item.
- Very large sites can exceed the per-sitemap state limit (8 MB); such a sitemap is marked failed with reason `state too large` and bills nothing. Use 1024 MB of memory for very large sites or many large sites.
- Memory: runs need at least 512 MB (the default). Sites are read one at a time, and the peak per sitemap file is its body (up to 20 MB decoded, held once) plus its parsed URL list.
- No `moved` events; no image, video or news extension fields; `lastmod` is the only change signal for modified pages.

### Politeness and robots.txt

Each sitemap's path is checked against the robots.txt of its host before fetching, and so is every redirect it follows, each on its own host; a sitemap that is disallowed, or that redirects somewhere disallowed, is skipped and named in `SUMMARY`. At most 1 request per second per host (one in flight), at most 4 hosts at once, retries with backoff on 429/5xx and network errors, a 20 second timeout, and a host that keeps returning 429 is given up on for the rest of the run. Identifying User-Agent: `makemoney-sitemap-monitor/1.0 (+https://apify.com/frostpine456/sitemap-monitor)`.

### Responsibility

**You are responsible for having the right to monitor the sites you name.**

### Not affiliated

This Actor is not affiliated with, endorsed by or sponsored by any website, company or search engine whose sitemaps you monitor. Site names are used only to describe what you chose to monitor.

### Data and privacy

Public sitemap files only. No login, no personal data, and no page content is stored or output: only robots.txt and sitemap URLs are requested, and an HTML response is abandoned unread. URLs that look like people's profile pages are skipped unless you turn on `includeProfileUrls`.

### Takedown

Site owners can request exclusion via the Issues tab; we honour all requests.

### FAQ

- **Why was my first run billed for every URL?** The first monitor run of a sitemap is a paid baseline: every URL is output once as `baseline` (a `url` event). Later runs bill only changes.
- **How do I get only new pages?** Use monitor mode (the default) and keep the same `stateKey`; after the baseline you see only `new`, `removed` and `modified`.
- **Why did a page not show as modified?** A page is `modified` only when its `lastmod` changes. Sites without lastmod can show new and removed pages, never modified.
- **Why are removals missing from a news sitemap?** News sitemaps are rolling: old articles age out, and rolling sitemaps are exempt from removals.
- **What does `truncated` mean?** The site has more URLs than `maxUrlsPerSite`; reading stopped at the cap, the site's status is `truncated`, and no removals are reported for any of its sitemaps. Raise `maxUrlsPerSite`.
- **Does it work without a sitemap?** It reads robots.txt `Sitemap:` lines first, then tries `/sitemap.xml` and `/sitemap_index.xml`. If none parse, the site is reported `failed` and billed nothing.

# Actor input Schema

## `sites` (type: `array`):

The sites to read, 1 to 50 items. Each item is a domain (example.com), a site URL reduced to its host (https://example.com/any/path), or a direct sitemap URL whose path ends in .xml, .xml.gz, .gz or .txt. Identical sites are de-duplicated; example.com and example.com/sitemap.xml are separate sites with separate state. The first monitor run bills every URL as a paid baseline.

## `mode` (type: `string`):

What each run returns. 'monitor' remembers each sitemap between runs and outputs only new, removed and modified pages (plus a paid baseline on the first run). 'extract' outputs every page URL listed, keeps no state, and bills each URL on every run.

## `maxUrlsPerSite` (type: `integer`):

The most URLs read per site per run, counted as de-duplicated canonical URLs, 1 to 50,000. A larger site is marked truncated in SUMMARY and its removals are held; raise this if SUMMARY shows truncated. The prefill keeps a first try small and cheap.

## `maxChanges` (type: `integer`):

The most change-billed records (new, removed, modified) emitted per run across all sites, 1 to 20,000. Baseline and existing records are not counted. Records past the cap are deferred to the next run, so nothing is lost.

## `includeExisting` (type: `boolean`):

Monitor mode only: also output every unchanged URL as an existing record, billed as a url event on EVERY run, not just the first. Leave it off for scheduled runs unless you need the full list each time.

## `includeProfileUrls` (type: `boolean`):

Include sitemaps and URLs whose path looks like a person's profile page (/user/, /profile/, /author/ and similar). They are skipped by default; skipped sitemaps are named and skipped URLs counted in SUMMARY.

## `rollingSitemaps` (type: `array`):

Extra substrings that mark a sitemap as rolling when found in its URL's path or query (never the host), e.g. news-sitemap. A sitemap is also rolling when it uses the news: namespace or its path or query contains 'news'. Rolling sitemaps are exempt from removals: URLs that age out emit nothing and are not billed.

## `stateKey` (type: `string`):

A label that separates change-tracking state when you run several independent schedules. Changing it starts a new paid baseline, so keep the same stateKey between runs of one schedule. 1 to 40 characters of lowercase a-z, 0-9 and '-'.

## `shrinkGuardPercent` (type: `integer`):

How far a sitemap may shrink in one run, in percent, before its removals are held for a confirming second run (10 to 100, default 50). Protects you from a truncated or half-written sitemap billing every missing URL as removed.

## Actor input object example

```json
{
  "sites": [
    "https://docs.apify.com"
  ],
  "mode": "extract",
  "maxUrlsPerSite": 100,
  "maxChanges": 2000,
  "includeExisting": false,
  "includeProfileUrls": false,
  "rollingSitemaps": [],
  "stateKey": "default",
  "shrinkGuardPercent": 50
}
```

# Actor output Schema

## `records` (type: `string`):

One record per URL: every listed URL in extract mode, or baseline/new/removed/modified/existing records in monitor mode.

## `summary` (type: `string`):

Per-site status, sitemap statuses, counts, flags and sections for this run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "https://docs.apify.com"
    ],
    "mode": "extract",
    "maxUrlsPerSite": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("frostpine456/sitemap-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sites": ["https://docs.apify.com"],
    "mode": "extract",
    "maxUrlsPerSite": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("frostpine456/sitemap-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "https://docs.apify.com"
  ],
  "mode": "extract",
  "maxUrlsPerSite": 100
}' |
apify call frostpine456/sitemap-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,frostpine456/sitemap-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/HgCLMPHhuaATRmlx7/builds/R6fZkF2J4JGkjTmkW/openapi.json