# Sitemap Change Monitor: New, Updated & Removed Pages (`jobdata_labs/sitemap-change-monitor`) Actor

Extract every URL from a website's XML sitemaps and track which pages were added, updated or removed since the last run. Competitor content tracking, SEO audits and site migrations.

- **URL**: https://apify.com/jobdata_labs/sitemap-change-monitor.md
- **Developed by:** [Momi](https://apify.com/jobdata_labs) (community)
- **Features:** Limited permissions
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.35 / 1,000 sitemap urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Change Monitor: New, Updated & Removed Pages

Extract every URL from a website's XML sitemaps and find out **which pages were added, updated or removed since the last check**. Point it at a competitor, a client site or your own site, schedule it, and get a clean change list instead of a full crawl.

### What it does

- **Finds sitemaps automatically**: give it a website and it reads the `Sitemap:` lines in robots.txt, falling back to `/sitemap.xml`. Or give it a sitemap URL directly.
- **Follows sitemap indexes and `.xml.gz` files**, so large sites with nested sitemaps work out of the box.
- **Change tracking**: with "Only changes since last run" on, each run returns only pages that are `new`, `updated` (newer `lastmod`) or `removed`.
- **Filters**: keep only `/blog/` or `/product/` URLs, exclude tag pages, or ignore pages not modified since a date.
- **Lightweight and polite**: reads only sitemap files, never the pages themselves. Identifies itself with its own user agent and respects robots.txt.

### Typical uses

- **Competitor content tracking**: see every new blog post, landing page or product a competitor publishes.
- **SEO monitoring**: catch pages that silently disappear from your sitemap, or a sudden wave of new URLs.
- **Site migrations**: compare the URL inventory before and after a relaunch.
- **Content pipelines**: feed only new or updated URLs into a scraper, an LLM or a RAG index instead of re-crawling everything.
- **Client reporting**: weekly list of what changed on a client's site.

### How change tracking works

1. The first run with a given **State name** stores a snapshot of all URLs and their `lastmod` dates and returns all of them (`changeType: null`).
2. Each later run compares the sitemap with the snapshot and returns only:
   - `new`: the URL was not in the previous snapshot,
   - `updated`: its `lastmod` is different from the previous one,
   - `removed`: it was in the previous snapshot and is gone now.
3. The snapshot is then replaced with the current state.

`removed` is only reported when the whole sitemap was read without errors, limits or filters, so a partial read never produces false removals.

Schedule the Actor from the **Schedules** tab (daily is typical) and connect an integration (Slack, email, Google Sheets, webhook) to receive the changes.

### Input

| Field | Type | Description |
| :--- | :--- | :--- |
| `startUrls` | Array | Websites (`example.com`) or sitemap URLs. Required. |
| `onlyChanges` | Boolean | Return only new, updated and removed pages. Default `false`. |
| `stateKey` | String | Name of the stored snapshot. Default `default`. |
| `includePatterns` | Array | Keep only URLs containing one of these fragments. |
| `excludePatterns` | Array | Drop URLs containing any of these fragments. |
| `lastmodAfter` | String | Keep only URLs modified on or after this date (`2026-01-31`). |
| `maxUrlsPerSite` | Integer | Stop after this many URLs per site. `0` = no limit. Default `5000`. |
| `respectRobotsTxt` | Boolean | Skip sitemap files disallowed by robots.txt. Default `true`. |

#### Example: weekly competitor blog tracker

```json
{
  "startUrls": ["https://competitor.example"],
  "onlyChanges": true,
  "stateKey": "competitor-blog",
  "includePatterns": ["/blog/"]
}
```

### Output

One dataset row per URL:

```json
{
  "site": "https://crawlee.dev",
  "url": "https://crawlee.dev/blog/crawlee-v3-16",
  "lastmod": "2026-11-05T00:00:00.000Z",
  "changefreq": "weekly",
  "priority": 0.5,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "changeType": "updated",
  "previousLastmod": "2026-10-01T00:00:00.000Z",
  "checkedAt": "2026-11-12T14:34:03.058Z"
}
```

A per-site summary (sitemap files read, URLs found and saved, counts of new/updated/removed, errors) is stored in the key-value store under `SUMMARY` and printed in the log.

### Pricing

You pay per URL saved to the dataset; the current rate is on the pricing tab. With change tracking on, unchanged pages are not saved and not charged, so a scheduled monitor on a stable site costs very little. Use `maxUrlsPerSite` to cap the cost of a run.

### FAQ

**The site has no sitemap. What happens?**
The run fails with a clear message. This Actor only reads sitemaps; it does not crawl pages.

**Why are some pages never marked as updated?**
Updates are detected from the sitemap's `lastmod` field. If a site does not publish `lastmod`, only new and removed pages can be detected.

**A sitemap shows "Disallowed by robots.txt".**
The website asks automated clients not to fetch that file, and this Actor respects that by default.

**Can I track several sites in one run?**
Yes. Each site has its own snapshot inside the same state name.

# Actor input Schema

## `startUrls` (type: `array`):

Websites (example.com) or direct sitemap URLs (example.com/sitemap.xml). For a website, sitemaps are found through robots.txt, falling back to /sitemap.xml. Sitemap indexes and .xml.gz files are followed automatically.

## `onlyChanges` (type: `boolean`):

Return only pages that are new, have a newer lastmod, or disappeared since the previous run with the same state name. The first run stores a baseline and returns all URLs. Turn this on for scheduled monitoring.

## `stateKey` (type: `string`):

Name under which the previous snapshot is remembered. Runs that share a state name compare against each other; use different names for independent monitors.

## `includePatterns` (type: `array`):

Keep only URLs that contain at least one of these text fragments, for example /blog/ or /product/. Leave empty to keep everything.

## `excludePatterns` (type: `array`):

Drop URLs that contain any of these text fragments, for example /tag/ or ?page=.

## `lastmodAfter` (type: `string`):

Keep only URLs whose sitemap lastmod is on or after this date, for example 2026-01-31. URLs without lastmod are dropped when this is set.

## `maxUrlsPerSite` (type: `integer`):

Stop reading a site after this many URLs. Set to 0 for no limit. Removed pages are only reported when the whole sitemap fits within the limit.

## `respectRobotsTxt` (type: `boolean`):

Skip sitemap files that the website's robots.txt disallows. Skipped files are listed in the run summary.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "onlyChanges": false,
  "stateKey": "default",
  "maxUrlsPerSite": 100,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ],
    "maxUrlsPerSite": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("jobdata_labs/sitemap-change-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev"],
    "maxUrlsPerSite": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("jobdata_labs/sitemap-change-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrlsPerSite": 100
}' |
apify call jobdata_labs/sitemap-change-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,jobdata_labs/sitemap-change-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/t1Nm7mXQF93iGLU82/builds/defrX9gdmsmaoUsbA/openapi.json