# Sitemap Change Monitor — Added, Removed & Updated URLs (`skyfishdev/sitemap-catalog-change-monitor`) Actor

Track catalog and website URL changes from XML sitemaps and sitemap indexes. Return baseline, added, removed, or lastmod-changed records. Stores a private per-user baseline, supports scheduled runs, and protects it when a crawl is incomplete.

- **URL**: https://apify.com/skyfishdev/sitemap-catalog-change-monitor.md
- **Developed by:** [Ibrahim](https://apify.com/skyfishdev) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.75 / 1,000 url records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Change Monitor — Added, Removed & Updated URLs

Track catalog and website URL changes from XML sitemaps and sitemap indexes. Return baseline, added, removed, or lastmod-changed records. Stores a private per-user baseline, supports scheduled runs, and protects it when a crawl is incomplete.

### Quick start

Try the small public demonstration fixture first, then replace its URL with your own public source:

```json
{
  "urls": [
    "https://commerce-actor-demos.vercel.app/sitemap.xml"
  ],
  "maxResults": 10
}
```

The demo source contains synthetic catalog data labelled as a test fixture. Results go straight into the default Dataset. Export JSON, CSV, or Excel in Apify Console. Diagnostics are saved as RUN_STATS.

### What it does

Follows same-origin sitemap index links, supports URL-set XML and gzip content, and deduplicates URLs. First complete run returns baseline records. Later runs compare URL presence and published lastmod; this does not compare page contents or detect price changes. Baselines live in a private named key-value store scoped to your Apify user and source URL. Use an Apify schedule to run again. Partial snapshots never overwrite the baseline and never label unseen URLs as removed. Run the same source sequentially; overlapping scheduled runs can race.

### Output

Main dataset fields: `change`, `url`, `lastmod`, `previousLastmod`, `snapshotComplete`, `sourceUrl`. A stable recordKey is included for retry deduplication. Inspect the demo run's dataset for a complete working output example.

#### Verified demo output

This sample was returned by a successful cloud run against the public synthetic fixture.

```json
[
  {
    "sourceUrl": "https://commerce-actor-demos.vercel.app/sitemap.xml",
    "url": "https://example.com/products/demo-one",
    "lastmod": "2026-10-08",
    "change": "unchanged",
    "previousLastmod": "2026-10-08",
    "snapshotComplete": true
  },
  {
    "sourceUrl": "https://commerce-actor-demos.vercel.app/sitemap.xml",
    "url": "https://example.com/products/demo-two",
    "lastmod": "2026-10-07",
    "change": "unchanged",
    "previousLastmod": "2026-10-07",
    "snapshotComplete": true
  }
]
```

### Pricing

Configured launch price: **$0.75 per 1,000 url record records**, plus a $0.00005 Actor-start event (memory-based, minimum one event). Check the live Pricing tab before running; discounted tiers may differ. Platform usage is included in pay-per-event pricing. Each dataset record is one event, including a validation result that reports issues. RUN_STATS diagnostics are not dataset items. Spending-limit checks stop new work before the next record.

### Limits and honest failures

Default maxResults is 100. This is a preview cap, not a promise to export everything. Set a larger cap only after checking a sample. Up to 100 source URLs, 8 MB per document, 50,000 rows per feed, 30-second request timeouts, 512 KB image probes, conservative shared request spacing, and at most five redirects. Public HTTP(S) only. Private/reserved addresses and credential URLs are rejected; DNS destinations are checked and pinned. robots.txt is respected. 429/503 stops the source for a later retry. Login/CAPTCHA pages, browser-only markup, and denied requests are reported rather than bypassed.

All-source failure ends FAILED after saving diagnostics. Useful partial output remains available with a partial status. Truncation and spending-limit stops are marked separately. Migration/retry checkpoints and existing dataset record IDs prevent repeated output in the same run. New independent runs are charged normally.

### Input

- **urls**: 1–100 public HTTP(S) URLs. No credentials, private networks, custom ports, or access-control bypass.
- **maxResults**: Global output cap. RUN_STATS reports truncation. Start with a small sample. Default: 100.
- **requestDelayMs**: Shared host spacing. Requests stop on 429/503 instead of ignoring cooldowns. Default: 500.
- **maxSitemaps**: Includes root and child XML documents. Reaching this limit prevents baseline replacement and removal detection. Default: 20.
- **onlyChanges**: Only changes after the first snapshot. Default: false.
- **updateBaseline**: Update private baseline after a complete crawl. Default: true.

### API

Use your Apify token in the Authorization header. For small jobs:

```sh
curl -X POST \
  'https://api.apify.com/v2/acts/skyfishdev~sitemap-catalog-change-monitor/run-sync-get-dataset-items' \
  -H 'Authorization: Bearer YOUR_APIFY_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"urls":["https://commerce-actor-demos.vercel.app/sitemap.xml"],"maxResults":10}'
```

For larger jobs, use the asynchronous runs endpoint, poll the run status, and fetch its default dataset.

### Local development

```sh
npm ci
npm test
apify run --input-file examples/demo.json
```

No LLM bill, image-host subscription, or additional API key is required. Only process public sources you are authorized to use. This tool does not promise customer demand, Google approval, or complete data on every website.

# Actor input Schema

## `urls` (type: `array`):

1–100 public HTTP(S) URLs. No credentials, private networks, custom ports, or access-control bypass.

## `maxResults` (type: `integer`):

Global output cap. RUN_STATS reports truncation. Start with a small sample.

## `requestDelayMs` (type: `integer`):

Shared host spacing. Requests stop on 429/503 instead of ignoring cooldowns.

## `maxSitemaps` (type: `integer`):

Includes root and child XML documents. Reaching this limit prevents baseline replacement and removal detection.

## `onlyChanges` (type: `boolean`):

Only changes after the first snapshot.

## `updateBaseline` (type: `boolean`):

Update private baseline after a complete crawl.

## Actor input object example

```json
{
  "urls": [
    "https://commerce-actor-demos.vercel.app/sitemap.xml"
  ],
  "maxResults": 100,
  "requestDelayMs": 500,
  "maxSitemaps": 20,
  "onlyChanges": false,
  "updateBaseline": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `runStats` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://commerce-actor-demos.vercel.app/sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("skyfishdev/sitemap-catalog-change-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://commerce-actor-demos.vercel.app/sitemap.xml"] }

# Run the Actor and wait for it to finish
run = client.actor("skyfishdev/sitemap-catalog-change-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://commerce-actor-demos.vercel.app/sitemap.xml"
  ]
}' |
apify call skyfishdev/sitemap-catalog-change-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,skyfishdev/sitemap-catalog-change-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/zwPSe6JF780SwkgiP/builds/kYtomEYPLDiyZILh2/openapi.json