# Sitemap Change Detector (`tri_angle/sitemap-change-detector`) Actor

Identify and monitor sitemaps for specified websites. Retrieve only the new, updated, or removed URLs since the last crawl.

- **URL**: https://apify.com/tri_angle/sitemap-change-detector.md
- **Developed by:** [Tri⟁angle](https://apify.com/tri_angle) (Apify)
- **Categories:** AI, Developer tools, Integrations
- **Stats:** 72 total users, 1 monthly users, 100.0% runs succeeded, 4 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Change Detector

Monitor changes in website sitemaps by comparing current entries against a previous snapshot stored in a named key-value store. Detect new, updated, removed, or unchanged URLs and react to changes automatically.

### What is Sitemap Change Detector?

This Apify actor fetches sitemap URLs (or discovers them via robots.txt), parses `<loc>` and `<lastmod>` entries, and stores a snapshot of URLs in a key-value store named `sitemap-change-detector-snapshots`. On subsequent runs, it retrieves the previous snapshot and compares the lists to identify changes.

### How it Detects Changes

- **NEW**: URLs present in the current sitemap but missing from the previous snapshot.
- **REMOVED**: URLs present in the previous snapshot but missing from the current sitemap.
- **UPDATED**: URLs present in both snapshots where the current `<lastmod>` timestamp is later than the previous one.
- **SAME**: URLs present in both snapshots with identical `<lastmod>` timestamps.

### How to Use

1. Go to the Sitemap Change Detector actor on the Apify Store.
2. Enter one or more Start URLs (direct sitemap URLs or root domains).
3. Enable Discover sitemaps to fetch robots.txt and enqueue sitemap URLs.
4. Select which Change Types to detect: NEW, UPDATED, REMOVED, SAME.
5. (Optional) Provide a URL filter regex to restrict monitored URLs.
6. (Optional) Enable Add removed URLs to KVS to log deleted URLs separately.
7. Save the input and click Run.
8. View the results in the default dataset or fetch via API or webhooks.

#### Example Input

```json
{
    "startUrls": [
        {
            "url": "https://www.apify.com",
            "method": "GET"
        }
    ],
    "changeTypes": ["NEW", "UPDATED"],
    "addRemovedUrlsToKvs": false,
    "discoverSitemaps": true,
    "snapshotKeyPrefix": "APIFY"
}
```

### Output

Each record in the output dataset represents a URL change:

```json
{
    "url": "https://example.com/page",
    "changeType": "UPDATED"
}
```

If Add removed URLs to KVS is enabled, check the `REMOVED-URLS` KVS record for an array of removed URLs.

### FAQ

#### Can I export data using API?

Yes, you can access this actor using your own applications through the Apify API. Click on the API tab for code examples or check out the Apify API reference docs at <https://docs.apify.com/api/v2> for full details.

#### Can I use Sitemap Change Detector through an MCP Server?

This actor, like all Apify actors, works on the Apify MCP server. For more information and instructions, read the Apify MCP server integration guide at <https://docs.apify.com/platform/integrations/mcp>.

#### Can I integrate data from Sitemap Change Detector with other apps?

Yes. Sitemap Change Detector can be connected with almost any cloud service or web app. Read more about the possibilities on our integrations page at <https://apify.com/integrations>.

#### Is it legal to scrape data using Sitemap Change Detector?

This actor only extracts publicly available data. It does not collect private user data. However, you should ensure your reason for scraping is legitimate. Consult legal counsel if unsure. For more on scraping legality and ethics, see:

- <https://blog.apify.com/is-web-scraping-legal/>
- <https://blog.apify.com/what-is-ethical-web-scraping-and-how-do-you-do-it/>

#### Your feedback

We welcome feedback to improve this actor. If you encounter issues or have suggestions, please create an issue on the actor’s Issues tab.

# Actor input Schema

## `startUrls` (type: `array`):

List of start URLs to scrape. These can be direct sitemap urls or website on which the sitemaps are going to be found if the `discoverSitemaps` is enabled.

## `discoverSitemaps` (type: `boolean`):

If enabled, the actor will fetch each start URL's robots.txt and enqueue any sitemap URL it finds. This is useful if you don't want to enter direct sitemap URLs. Please note that this will only work if the website has robots.txt.

## `changeTypes` (type: `array`):

Maximum number of URLs the Website Content Crawler will process in each run. Increasing this decreases the total number of runs needed to process all URLs from the Sitemap Change Detector. Note that the run datasets will be merged and output afterwards.

## `snapshotKeyPrefix` (type: `string`):

Prefix for the snapshot record key stored in the snapshots key-value store, to separate runs by website or project.

## `urlFilterRegex` (type: `string`):

Regex pattern to filter which URLs are included in the output and snapshot. This filter applies only to the final URLs and not to intermediate sitemap URLs.

## `addRemovedUrlsToKvs` (type: `boolean`):

If enabled, the actor will always also include URLs that were removed compared to the previous snapshot to the key-value store.

## `proxyConfiguration` (type: `object`):

Proxy configuration used for crawling.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "discoverSitemaps": true,
  "changeTypes": [
    "NEW",
    "UPDATED"
  ],
  "snapshotKeyPrefix": "DEFAULT",
  "addRemovedUrlsToKvs": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tri_angle/sitemap-change-detector").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://apify.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("tri_angle/sitemap-change-detector").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ]
}' |
apify call tri_angle/sitemap-change-detector --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tri_angle/sitemap-change-detector"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5kzBz8eWYP2tKADin/builds/KTPZsKw8XSXb1O4uH/openapi.json