# Sitemap Change Monitor: New Competitor Pages (`singletonops/sitemap-change-monitor`) Actor

Track new competitor pages: monitor public XML sitemaps (indexes and .xml.gz) and robots.txt and get only the URLs that were added, removed or whose lastmod changed since the last run. Often the first sign of a new landing page or campaign. Optional webhook. No browser, no proxy.

- **URL**: https://apify.com/singletonops/sitemap-change-monitor.md
- **Developed by:** [SingletonOps](https://apify.com/singletonops) (community)
- **Categories:** Marketing, SEO tools, Automation
- **Stats:** 2 total users, 1 monthly users, 20.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Sitemap Change Monitor do?

**Sitemap Change Monitor** watches public XML sitemaps (including sitemap indexes and `.xml.gz` files) and **robots.txt** `Sitemap:` directives, then outputs **only the URLs that changed** since the previous run — **added**, **removed**, or **`<lastmod>` changed**. It is a lightweight, browser-free **sitemap change detection API** alternative for SEO and content ops teams.

It does **not** crawl page HTML, bypass logins, or use proxies. Public sitemap data only.

***

### Why use Sitemap Change Monitor?

| Use case | How it helps |
| --- | --- |
| **Competitor new-page tracking** | Spot new URLs in a competitor's sitemap as they publish |
| **Content / launch detection** | Catch product, blog, or docs launches the moment they appear in the sitemap |
| **SEO monitoring** | Track lastmod churn and removals without scraping every page |
| **Scheduled automation** | Pair with Apify Schedules + webhooks / Zapier / Make / Slack |

**Benefits**

- **Cheap compute** — plain HTTP + fast XML parse (no browser)
- **Stateful diffs** — named key-value store baseline across scheduled runs
- **Index-aware** — recursively expands sitemap indexes with configurable caps
- **Gzip + robots.txt** — handles `.xml.gz` and discovers sitemaps from `robots.txt`
- **Optional filters** — include/exclude URL regexes
- **Optional webhook** — JSON POST when changes are found

***

### What data can Sitemap Change Monitor extract?

| Field | Type | Description |
| --- | --- | --- |
| `changeType` | string | `added` | `removed` | `lastmodChanged` |
| `url` | string | Page URL from `<loc>` |
| `lastmod` | string | null | Current `<lastmod>` (null if omitted or removed) |
| `previousLastmod` | string | null | Prior baseline lastmod |
| `sourceSitemap` | string | Concrete sitemap that listed the URL |
| `detectedAt` | string | ISO-8601 detection time |

You can download the dataset as JSON, CSV, Excel, or HTML from the Apify Console, or via the Apify API.

***

### How to monitor sitemap changes

1. Open **Sitemap Change Monitor** on Apify Store and click **Try for free**
2. Paste one or more public `sitemap.xml`, `.xml.gz`, or `robots.txt` URLs
3. Keep **Emit all URLs on first run** unchecked to record a quiet baseline (recommended)
4. (Optional) Set include/exclude regexes and a webhook URL
5. Click **Start**, or create a **Schedule** (e.g. hourly or daily)
6. On later runs, read the **Dataset** for changes and the **OUTPUT** key for a summary

#### Scheduling tips

- Use the **same** `stateStoreName` and `stateKey` on every scheduled run so diffs stay continuous
- Start with hourly or daily cadence; aggressive polling is rarely needed for sitemaps
- Set `maxSitemaps` / `maxUrls` to bound cost on large sites

***

### How much will it cost?

Designed for **pay-per-result** style pricing (each dataset row = one meaningful change). Compute stays low because the Actor only does HTTP fetches and XML parsing — no headless browser. Exact Store pricing is configured by the publisher; free-tier compute applies while you try it.

***

### Input

See the **Input** tab for the full form. Main fields:

| Field | Required | Default | Description |
| --- | --- | --- | --- |
| `sitemapUrls` | ✅ | — | Public sitemap / index / `.xml.gz` / `robots.txt` URLs |
| `maxSitemaps` | ❌ | `50` | Cap on sitemap documents fetched while expanding indexes |
| `maxUrls` | ❌ | `10000` | Cap on page URLs tracked per run |
| `stateStoreName` | ❌ | `sitemap-change-monitor-state` | Named KVS for baselines across runs |
| `stateKey` | ❌ | `sitemap-state` | Record key inside that store |
| `emitAllOnFirstRun` | ❌ | `false` | If false, first run baselines only (no dataset rows) |
| `includeUrlRegex` | ❌ | — | Keep only matching page URLs |
| `excludeUrlRegex` | ❌ | — | Drop matching page URLs |
| `webhookUrl` | ❌ | — | POST JSON summary when changes exist |
| `requestTimeoutSecs` | ❌ | `30` | Per-request timeout |
| `maxRetries` | ❌ | `3` | Retries with exponential backoff |

#### Example input

```json
{
  "sitemapUrls": [
    "https://www.apify.com/sitemap.xml",
    "https://example.com/robots.txt"
  ],
  "maxSitemaps": 50,
  "maxUrls": 10000,
  "stateStoreName": "sitemap-change-monitor-state",
  "stateKey": "sitemap-state",
  "emitAllOnFirstRun": false,
  "includeUrlRegex": "",
  "excludeUrlRegex": ".*\\.(pdf|jpg|png)$"
}
```

***

### Output

#### Dataset example

```json
[
  {
    "changeType": "added",
    "url": "https://example.com/blog/new-post",
    "lastmod": "2026-10-08",
    "previousLastmod": null,
    "sourceSitemap": "https://example.com/sitemap-posts.xml",
    "detectedAt": "2026-10-08T15:30:00.000Z"
  },
  {
    "changeType": "lastmodChanged",
    "url": "https://example.com/pricing",
    "lastmod": "2026-10-07",
    "previousLastmod": "2026-09-01",
    "sourceSitemap": "https://example.com/sitemap.xml",
    "detectedAt": "2026-10-08T15:30:00.000Z"
  },
  {
    "changeType": "removed",
    "url": "https://example.com/old-landing",
    "lastmod": null,
    "previousLastmod": "2026-01-01",
    "sourceSitemap": "https://example.com/sitemap.xml",
    "detectedAt": "2026-10-08T15:30:00.000Z"
  }
]
```

#### Key-value store

| Location | Key | Contents |
| --- | --- | --- |
| Default store | `OUTPUT` | Run summary (totals + per-root stats) |
| Named store (`stateStoreName`) | `stateKey` | Baseline map of URL → lastmod per input root |

#### Webhook payload (when configured and changes exist)

Includes `totalChanges`, `added` / `removed` / `lastmodChanged`, per-root summaries, and a `sample` of up to 50 change rows. Full rows are always in the dataset.

***

### FAQ

#### Does the first run flood my dataset?

No — by default `emitAllOnFirstRun` is `false`, so the first run only writes a baseline and an `OUTPUT` summary. Set it to `true` if you want every URL emitted as `added` once.

#### Can I pass robots.txt instead of a sitemap?

Yes. If a URL path ends with `robots.txt`, the Actor reads `Sitemap:` lines and expands those sitemaps (still public HTTP only).

#### Are gzipped sitemaps supported?

Yes. `.xml.gz` URLs and gzip-encoded responses are decompressed automatically.

#### Does this respect robots.txt beyond sitemap discovery?

The Actor uses robots.txt only to **discover** listed sitemaps. It sends an honest User-Agent, uses timeouts and retries with backoff, and does not scrape page bodies. You are responsible for complying with each site's terms and for polite scheduling.

#### How do scheduled runs keep state?

Use a **named** key-value store (`stateStoreName`) and a stable `stateKey`. The default run store is not shared across runs; the named store is.

#### Local development

```bash
cd sitemap-change-monitor
## Edit storage/key_value_stores/default/INPUT.json
apify run
## Inspect storage/datasets/default/ and storage/key_value_stores/
```

***

### Legal & acceptable use

- **Public sitemaps and robots.txt only.** Do not use this Actor on URLs that require login, cookies, API keys, or other authentication.
- Respect publisher terms of service and rate limits. Prefer reasonable schedules.
- You are responsible for how you use the output (attribution, redistribution, downstream automation).
- This Actor does not bypass paywalls or crawl HTML page content.

Our Actors are ethical and do not extract private user data. They only read what site owners publish in public sitemaps / robots.txt. Results may still include URLs that identify people or businesses; comply with GDPR and other applicable laws, and scrape only when you have a legitimate purpose.

***

### Limitations

- Very large indexes are truncated by `maxSitemaps` / `maxUrls`
- Sites that omit `<lastmod>` can only surface **added** / **removed** (not lastmod changes)
- Fetch failures are logged per root; other roots still process
- Duplicate `<loc>` values: first occurrence wins within a run

***

### Changelog

#### 0.1

- Initial release: sitemap + index + gzip + robots.txt discovery, named-store baselines, dataset diffs, OUTPUT summary, optional webhook and URL filters

***

### Support

Open an issue on the Actor’s Apify page or contact the author through Apify Console messaging. For programmatic access, see the **API** tab.

# Actor input Schema

## `sitemapUrls` (type: `array`):

Public sitemap.xml, sitemap index, gzipped .xml.gz, or robots.txt URLs. For robots.txt the Actor reads Sitemap: directives and expands those sitemaps. Private or login-walled URLs are not supported.

## `maxSitemaps` (type: `integer`):

Maximum number of sitemap documents to fetch while expanding sitemap indexes (including the root). Protects against huge indexes.

## `maxUrls` (type: `integer`):

Maximum number of page URLs to collect across all expanded sitemaps for this run. Bounds memory and key-value store size.

## `stateStoreName` (type: `string`):

Named Apify key-value store used to persist URL/lastmod baselines across scheduled runs. Use the same name for recurring monitoring.

## `stateKey` (type: `string`):

Record key inside the named store that holds the baseline map. Change this to keep separate baselines (e.g. per project).

## `emitAllOnFirstRun` (type: `boolean`):

When false (default), the first run only records a baseline and writes a summary — dataset stays empty. When true, every URL is emitted as changeType=added.

## `includeUrlRegex` (type: `string`):

Optional JavaScript regex. When set, only page URLs matching this pattern are tracked (e.g. ".*/blog/.*").

## `excludeUrlRegex` (type: `string`):

Optional JavaScript regex. Matching page URLs are ignored (e.g. ".\*\\.(pdf|jpg)$").

## `webhookUrl` (type: `string`):

Optional HTTPS endpoint that receives a JSON POST when changes are found. Payload includes totals, per-root summaries, and a sample of up to 50 change rows.

## `requestTimeoutSecs` (type: `integer`):

Per-request HTTP timeout.

## `maxRetries` (type: `integer`):

Retries with exponential backoff for timeouts, 429, and 5xx responses.

## Actor input object example

```json
{
  "sitemapUrls": [
    "https://apify.com/sitemap.xml"
  ],
  "maxSitemaps": 50,
  "maxUrls": 10000,
  "stateStoreName": "sitemap-change-monitor-state",
  "stateKey": "sitemap-state",
  "emitAllOnFirstRun": false,
  "requestTimeoutSecs": 30,
  "maxRetries": 3
}
```

# Actor output Schema

## `changes` (type: `string`):

Dataset of added, removed, and lastmod-changed page URLs.

## `summary` (type: `string`):

JSON summary written to the default key-value store under key OUTPUT.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sitemapUrls": [
        "https://apify.com/sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("singletonops/sitemap-change-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sitemapUrls": ["https://apify.com/sitemap.xml"] }

# Run the Actor and wait for it to finish
run = client.actor("singletonops/sitemap-change-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sitemapUrls": [
    "https://apify.com/sitemap.xml"
  ]
}' |
apify call singletonops/sitemap-change-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,singletonops/sitemap-change-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/R2FJSMfWevbi62CcC/builds/LABj5cuRbD6icHGtc/openapi.json