# Sitemap Extractor & Change Tracker (`codemagnet/sitemap-change-tracker`) Actor

Extract every URL from a website's sitemaps (finds them even when they're hidden) and track new, updated and removed pages between runs. Reports blocked or broken sitemaps instead of failing.

- **URL**: https://apify.com/codemagnet/sitemap-change-tracker.md
- **Developed by:** [Kirby Bookhart-Tsai](https://apify.com/codemagnet) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.25 / 1,000 url saveds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor & Change Tracker

Get every URL from a website's sitemaps, and see which pages are **new, updated or removed** since your last run. Enter a domain and the Actor finds the sitemaps itself, reads nested and compressed ones, and tells you exactly what it could and couldn't read.

- **Finds sitemaps other tools miss.** It checks robots.txt first, then the usual locations (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemap.txt` and more), then the home page's `<link rel="sitemap">`. If robots.txt lists sitemaps that no longer exist, it falls back to the usual locations.
- **Reads what's actually out there:** sitemap indexes nested several levels deep, `.gz` files (even when the server sends the wrong content type or compresses them twice), plain-text sitemaps, RSS and Atom feeds, namespaced and slightly broken XML.
- **Tracks changes between runs.** Each URL is marked `new`, `updated` (its last-modified date changed), `unchanged` or `removed`. Schedule it weekly and save **only the changes**: new product pages, new blog posts, deleted pages.
- **No false alarms.** If a sitemap fails to load this time, its pages are *not* reported as removed. They're remembered and compared again next run.
- **Honest reports, not silent failures.** A web page served instead of a sitemap, a corrupt `.gz` file, a timeout or bot protection: each problem is named per sitemap file in the report. One broken sitemap doesn't fail the run; the URLs from the others are kept.
- **Polite by design.** One request at a time per website, at least half a second apart, slower if robots.txt sets a `Crawl-delay`. Obeys robots.txt and identifies itself honestly. Never bypasses bot protection: if a site blocks automated access, you're told so.
- **You don't pay for nothing.** Sites where no sitemap could be read cost nothing beyond the tiny start fee.

### Quick start

1. Enter one or more websites (`example.com`) or sitemap addresses under **Websites or sitemap URLs**.
2. Run it. The first run lists every URL and remembers them.
3. To monitor a site, set **What to save** to **Only changes** and add a [schedule](https://docs.apify.com/platform/schedules) (for example every Monday). Each run then saves only the pages that are new, updated or removed since the previous run.
4. Optional: connect an [integration](https://docs.apify.com/platform/integrations) (Slack, email, Google Sheets, Zapier, Make, a webhook) to get the changes delivered.

**Tip:** enter a section such as `example.com/blog/` to get only that section's URLs. The Actor checks for the section's own sitemap first, which is much faster on big sites. Enter the bare domain to get the whole site.

### Output

One row per URL:

```json
{
  "url": "https://crawlee.dev/blog/scrapy-vs-crawlee",
  "changeType": "updated",
  "lastmod": "2026-09-30",
  "previousLastmod": "2026-06-12",
  "firstSeenAt": "2026-08-04",
  "changefreq": "weekly",
  "priority": 0.5,
  "site": "https://crawlee.dev/",
  "sitemapUrl": "https://crawlee.dev/sitemap.xml"
}
```

| Field | Meaning |
|---|---|
| `changeType` | `new`, `updated`, `unchanged` or `removed` compared with the previous run. Empty when change tracking is off. |
| `lastmod` / `previousLastmod` | The page's last-modified date from the sitemap, now and last time. |
| `firstSeenAt` | The date this tracker first saw the URL. |
| `changefreq`, `priority` | As given in the sitemap, if present. |
| `sitemapUrl` | The sitemap file the URL came from. |

Rows are ordered new, updated, removed, then unchanged. Download them as JSON, CSV, Excel or HTML from the run's **Storage** tab, or via the API.

The **OUTPUT** record in the key-value store has a report per site: how its sitemaps were found, how many were read, which ones failed and why, and the counts of new, updated and removed pages.

### How change tracking works

- After each run the Actor stores each site's URLs (with their last-modified dates) in a storage on **your own Apify account**, named `sitemap-change-tracker` by default. Nothing is shared with anyone else.
- The next run compares against it. A URL is `updated` only when its sitemap date changes. Pages whose sitemap has no dates can show up as new or removed, but not as updated.
- **Only changes** on a first run saves no rows; it just records the starting point. Use **All URLs** if you want the full list on the first run.
- Histories are kept per site and per filter setting, so changing **Only URLs containing** or **Skip URLs containing** starts a fresh history for that combination. Use different **History name**s to keep separate histories for separate projects, or **Start history fresh** to reset.
- If any of a site's sitemaps couldn't be read, or a URL, memory or cost limit cut the scan short, missing pages are **not** reported as removed in that run.

### Filters

- **Only URLs containing** / **Skip URLs containing:** text that must, or must not, appear in the URL. `*` is a wildcard (`/products/*/reviews`). Start with `re:` for a regular expression (`re:/20(25|26)/`).
- **Modified since:** a date (`2026-09-01`) or a period (`7 days`, `2 weeks`, `3 months`). Keeps URLs whose sitemap date is on or after it; URLs without a date are left out. With change tracking on, all URLs are still remembered so the comparison stays correct. Removed pages are always reported.
- **Max URLs per site:** stop reading a site after this many URLs.

### Pricing

You pay only for what the Actor does, with no monthly fee:

| Event | Price | When |
|---|---|---|
| Actor start | $0.00005 | Each run (Apify's standard start event) |
| Site scanned | $0.002 | Per website where at least one sitemap was read |
| URL found | $0.05 per 1,000 | Each unique URL read from the sitemaps (after your filters) |
| URL saved | $0.25 per 1,000 | Each row saved to the dataset |

Examples:

- Full list of a 5,000-page site: 0.002 + 5,000 × (0.00005 + 0.00025) ≈ **$1.50**.
- Weekly monitoring of that site with **Only changes**, 40 changes a week: 0.002 + 5,000 × 0.00005 + 40 × 0.00025 ≈ **$0.26 per week**.
- A site with no readable sitemap: **$0.00005**.

Set a **maximum cost per run** in the run options. The Actor stops cleanly at that limit. Changes it couldn't save are not lost: they're reported again on the next run.

### Good to know

- **Big sites:** the default 1 GB of memory holds about 400,000 URLs for one site. Up to three sites are read at the same time and share that memory equally: about 200,000 URLs each for two sites, 135,000 each for three or more. If a site is bigger, the Actor stops cleanly at that point and the report says so. Raise the memory in the run options to read it all (each extra GB adds about 400,000 URLs). The cut-off is the same every run, so it never shows up as false "new" or "removed" pages.
- **Speed:** a site with one sitemap takes a few seconds. Sites with hundreds of sitemap files take longer, because the Actor waits between requests to the same website (and longer if robots.txt asks for it).
- **Not found?** If no sitemap is found, enter the sitemap's address directly. Some sites only mention it in Google Search Console.
- **Blocked?** Some sites block all automated requests (for example, with a Cloudflare challenge). The report says so. The Actor does not try to get around it.
- This Actor reads sitemaps, files that websites publish specifically for automated crawlers. It doesn't visit or download the pages themselves, and it never collects personal data.

### FAQ

**Why are some sites slow?** Each website gets one request at a time, half a second apart or more. That keeps the Actor welcome on the sites you monitor. Different websites are read in parallel.

**Why was a URL I can see on the site not found?** It isn't in the site's sitemaps. Sitemaps list what the site owner chose to list. This Actor reads sitemaps rather than crawling links.

**Can I use it from an AI agent or the API?** Yes. Call it with a list of URLs and read the dataset. The OUTPUT record gives a compact summary per site.

**Something wrong?** Open an issue on the **Issues** tab with the site and the run link. Fixes for real sitemaps that fail are a priority.

# Actor input Schema

## `startUrls` (type: `array`):

Websites (like <code>example.com</code>) or direct sitemap addresses. For a website, the Actor finds the sitemaps itself: robots.txt first, then common locations like /sitemap.xml and /wp-sitemap.xml, then the home page.

## `output` (type: `string`):

<b>All URLs</b>: every URL, each marked new, updated, unchanged or removed. <b>Only changes</b>: just new, updated and removed pages since the last run (the first run only records a starting point). Best for scheduled monitoring.

## `trackChanges` (type: `boolean`):

Remember each site's URLs (in a storage on your own account) and compare with the previous run. Turn off for a one-off list.

## `includePatterns` (type: `array`):

Keep only URLs matching at least one pattern. A pattern matches anywhere in the URL; <code>*</code> is a wildcard. Examples: <code>/blog/</code>, <code>/products/*/reviews</code>. Start with <code>re:</code> for a regular expression, e.g. <code>re:/20(25|26)/</code>.

## `excludePatterns` (type: `array`):

Drop URLs matching any of these patterns (same syntax as above). Example: <code>/tag/</code>, <code>?page=</code>.

## `modifiedSince` (type: `string`):

Only save URLs whose sitemap date (lastmod) is on or after this date, e.g. <code>2026-09-01</code>, or a period like <code>7 days</code>. URLs without a date are left out. Removed pages are always reported.

## `maxUrlsPerSite` (type: `integer`):

Stop reading a site's sitemaps after this many URLs. Leave empty for no limit. (When a scan stops early, removed pages are not reported for that run.)

## `historyName` (type: `string`):

Name of the storage (on your account) that remembers URLs between runs. Use a different name to keep separate histories, for example one per project.

## `resetHistory` (type: `boolean`):

Ignore what was remembered before and treat this run as the first one.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "output": "all",
  "trackChanges": true,
  "historyName": "sitemap-change-tracker",
  "resetHistory": false
}
```

# Actor output Schema

## `urls` (type: `string`):

One row per URL with its change type (new, updated, removed, unchanged), last-modified dates and the sitemap it came from.

## `report` (type: `string`):

How each site's sitemaps were found, which ones could not be read and why, and the change counts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("codemagnet/sitemap-change-tracker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://crawlee.dev"] }

# Run the Actor and wait for it to finish
run = client.actor("codemagnet/sitemap-change-tracker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ]
}' |
apify call codemagnet/sitemap-change-tracker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,codemagnet/sitemap-change-tracker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/2nslxxnL7Cxpm3tph/builds/lQb6s42zpXnvBIJLe/openapi.json