# Sitemap URL Extractor with Change Tracking (`opendatalabs/sitemap-change-tracker`) Actor

Extracts every URL from a site's sitemaps (indexes, gzip, text and feed formats included) and reports which URLs were added, removed or re-dated since the previous run.

- **URL**: https://apify.com/opendatalabs/sitemap-change-tracker.md
- **Developed by:** [Shyheim Ulrickson](https://apify.com/opendatalabs) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.21 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**Use it to** get every URL a website lists in its sitemaps, and see which pages were added, changed or removed since your last run.

**Cost:** $0.25 per 1,000 URLs, plus $0.001 per sitemap file read. A 100-URL test run costs about $0.03.

**Why this one**

- **Change tracking built in.** Most sitemap extractors give you a one-off list. Change mode keeps a baseline between runs and labels every URL `added`, `changed` or `removed`, so a daily schedule returns only what moved.
- **Built not to fail.** A sitemap that times out or returns 403 becomes a data row and a counter, not a failed run. Transient errors are retried with backoff, and a bad night at the origin cannot wipe your baseline.
- **Finds the sitemaps for you** through `robots.txt` and common locations, including nested indexes and `.xml.gz` files.

**Sample output** (real run on `crawlee.dev`, Oct 20, 2026):

| url | changefreq | priority | sourceSitemap | discoveredVia |
|---|---|---|---|---|
| https://crawlee.dev/blog | weekly | 0.5 | https://crawlee.dev/sitemap.xml | robots.txt |
| https://crawlee.dev/blog/archive | weekly | 0.5 | https://crawlee.dev/sitemap.xml | robots.txt |
| https://crawlee.dev/blog/authors | weekly | 0.5 | https://crawlee.dev/sitemap.xml | robots.txt |

[![Sitemap change tracker output: new blog posts on blog.apify.com labelled added, with lastmod dates](https://api.apify.com/v2/key-value-stores/CHfh24KK6KF51EHqU/records/sitemap-change-tracker-new-blog-posts.png)](https://apify.com/opendatalabs/sitemap-change-tracker/examples/track-new-blog-posts)

*Real change-tracking output from the [Track new blog posts](https://apify.com/opendatalabs/sitemap-change-tracker/examples/track-new-blog-posts) example task (Oct 21, 2026).*

Pull every URL out of a website's sitemaps, then find out what changed since the last time you looked.

Give it a domain and it discovers the sitemaps through `robots.txt`, walks sitemap indexes, unpacks gzip, and returns one row per URL with `lastmod`, `changefreq`, `priority` and the sitemap each URL came from. Switch on change tracking and it also tells you which URLs are **new**, which have been **removed**, and which have a **changed `lastmod`** since the previous run.

### What can this sitemap extractor do?

- **Finds sitemaps for you.** Hand it `example.com` and it reads `robots.txt` first, which is the only discovery mechanism the sitemap protocol actually specifies. If `robots.txt` declares nothing, it probes twelve common locations (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, the `.gz` variants, and more).
- **Handles every sitemap shape.** Sitemap indexes nested several levels deep, `.xml.gz` files, plain-text URL lists, and RSS/Atom feeds. Namespaced XML, hreflang alternates, image/video extensions and news sitemaps are all parsed.
- **Tracks changes between runs.** Change mode keeps the previous URL set in a **named** key-value store, which survives from run to run, and labels each URL `added`, `changed`, `removed` or `unchanged`.
- **Checks for broken links, optionally.** A cheap `HEAD` per URL (falling back to `GET` where a server rejects `HEAD`) adds `statusCode`, `statusOk` and `redirectedTo`.
- **Finishes the run.** A sitemap that times out or 403s is reported as a row of data and a run-summary counter, not as a failed run. A target only fails the run if *every* target fails.

### How to extract all URLs from a sitemap

1. Enter one or more domains (`example.com`) or direct sitemap URLs in **Domains or sitemap URLs**. The Actor finds the sitemaps through `robots.txt` for you.
2. Pick a **Run mode**: **Snapshot** lists every URL; **Change tracking** returns only URLs added, changed or removed since the previous run.
3. Set **Max URLs per target** to cap cost, then click **Start**.
4. Download the results as CSV, Excel or JSON from the **Output** tab, or read them through the Apify API.
5. To monitor a site, save the input as a task and add a schedule (Console > Schedules). Each run then returns only what moved.

### Example tasks: monitor competitors and track new pages

Ready-made configurations you can copy and run in one click:

- [Monitor a competitor's new pages](https://apify.com/opendatalabs/sitemap-change-tracker/examples/monitor-competitor-new-pages): added and removed URLs on a schedule.
- [Track new blog posts on a website](https://apify.com/opendatalabs/sitemap-change-tracker/examples/track-new-blog-posts): new posts from a blog's post sitemap, with dates.
- [Export every URL in a website's sitemap](https://apify.com/opendatalabs/sitemap-change-tracker/examples/export-website-urls-from-sitemap): a full URL list ready for CSV or Excel.

### Why is this sitemap scraper reliable?

Reliability is the point of this Actor, so a few things work differently from the usual sitemap scraper:

| Problem | What this Actor does |
|---|---|
| CDN returns 403/503 for some sitemaps | Retries only transient errors, with exponential backoff and jitter, honouring `Retry-After`. A 404 fails immediately instead of burning four timeouts. |
| A bad day at the origin wipes your baseline | Change mode refuses to overwrite the baseline when a run extracts zero URLs, or when more than a configurable share of sitemaps failed. Without that guard, one bad night makes the next run report your entire site as "added". |
| `lastmod` reformatting creates fake changes | `lastmod` is compared on a normalised ISO-8601 value, so `2026-10-13`, `2026-10-13T00:00:00Z` and `2026-10-13T00:00:00+00:00` are the same date, not three changes. |
| Hitting a URL limit returns one section of the site | Sitemaps are walked **breadth-first**, so a truncated run still spans the whole site. |
| A sitemap served as `.xml.gz` with no encoding header | Gzip is detected from the file's magic number, not from a header the server may not send. |

### Sitemap extractor input examples

Minimum viable input:

```json
{
  "targets": ["example.com"]
}
```

A scheduled competitor monitor:

```json
{
  "targets": ["competitor.example", "https://other.example/sitemap_index.xml"],
  "mode": "changes",
  "stateStoreName": "competitor-watch",
  "outputChangeTypes": ["added", "changed", "removed"],
  "checkUrlStatus": false,
  "requestsPerSecond": 3
}
```

A sitemap-wide link check:

```json
{
  "targets": ["your-site.example"],
  "mode": "snapshot",
  "checkUrlStatus": true,
  "statusCheckLimit": 2000,
  "concurrency": 10
}
```

#### Key settings

| Field | Default | Notes |
|---|---|---|
| `targets` | — | Bare domains or direct sitemap URLs, mixed freely. Left empty, the run is a 100-URL demo on `crawlee.dev` (about $0.03) instead of failing. |
| `mode` | `snapshot` | `changes` enables run-to-run diffing. |
| `stateStoreName` | `sitemap-change-tracker-state` | Named key-value store holding the baseline. One store per monitoring job; targets are separate keys inside it. |
| `outputChangeTypes` | `added`, `changed`, `removed` | Add `unchanged` for a full labelled snapshot. |
| `checkUrlStatus` | `false` | Adds a `HEAD` request per URL. |
| `maxUrlsPerTarget` | `1000` | Breadth-first, so truncation still spans the site. The default keeps a run around $0.25 per target at $0.00025 per URL, and the Console form is prefilled with 100 so a first test run costs cents. Raise it (up to 5,000,000) for a full inventory. |
| `maxSitemapsPerTarget` | `500` | Large stores publish hundreds of child sitemaps. |
| `respectRobotsTxt` | `true` | Checks the sitemap path against `robots.txt` before fetching. |
| `requestsPerSecond` | `5` | Global token-bucket ceiling. |
| `onlyUrlsModifiedSince` | — | `YYYY-MM-DD`. URLs with no `lastmod` are kept. |

Full descriptions for every field are in the Input tab.

### What data can you extract from a sitemap?

One row per URL. Snapshot mode:

```json
{
  "url": "https://wordpress.org/news/2026/10/wordpress-6-9-release-candidate/",
  "domain": "wordpress.org",
  "lastmod": "2026-10-14T19:02:40Z",
  "lastmodIso": "2026-10-14T19:02:40.000Z",
  "changefreq": "monthly",
  "priority": 0.5,
  "sourceSitemap": "https://wordpress.org/sitemap-1.xml",
  "discoveredVia": "robots.txt",
  "scrapedAt": "2026-10-19T01:39:22.118Z"
}
```

Change mode adds the diff fields:

```json
{
  "url": "https://competitor.example/pricing/enterprise",
  "domain": "competitor.example",
  "lastmodIso": "2026-10-18T09:12:00.000Z",
  "sourceSitemap": "https://competitor.example/sitemap-pages.xml",
  "changeType": "added",
  "firstSeenAt": "2026-10-19T01:39:22.118Z",
  "scrapedAt": "2026-10-19T01:39:22.118Z"
}
```

With `checkUrlStatus` on, every row also carries `statusCode`, `statusOk` and, where relevant, `redirectedTo` or `statusError`.

The dataset ships three views: **URL overview**, **Changes since last run**, and **Link check**.

A `RUN_SUMMARY` record is written to the default key-value store with per-target counts, the sitemap failure rate, discovery notes, change counts, and whether the baseline was updated.

### Use cases for sitemap change tracking

- **Competitive content intelligence.** Schedule it daily in change mode against a competitor's domain and get a feed of every page they published, retired or re-dated.
- **SEO audits and migrations.** Inventory every indexable URL, spot sitemap entries that 404, and diff a staging sitemap against production before cutover.
- **RAG and LLM ingestion.** Use the URL list as a seed set, then re-run in change mode so you only re-embed what actually changed.
- **E-commerce monitoring.** Discover new product URLs the moment they enter the sitemap.
- **Feeding a page audit.** Chain the output into a Core Web Vitals or accessibility Actor and only audit the URLs that changed.

### Limitations and known issues

- **Only what the sitemap says.** This Actor reads sitemaps; it does not crawl pages. A page missing from the sitemap is invisible to it, and `lastmod` is a publisher-supplied hint that some CMSs set to the build time of every page. When `lastmod` is unreliable on a site, `changeType: "changed"` will be noisy — rely on `added` and `removed` there.
- **First run in change mode reports everything as `added`.** There is no baseline yet. That is expected; the comparison starts on run two.
- **Change tracking needs a named key-value store.** The default store is scoped to a single run. If you rename `stateStoreName`, you start from a fresh baseline.
- **Sites behind aggressive bot protection may refuse.** Those sitemaps come back as `blocked` in the run summary rather than being retried indefinitely. The Actor does not attempt to evade bot protection.
- **`maxUrlsPerTarget` and `maxSitemapsPerTarget` are real ceilings.** When a run is truncated, the run summary says so.
- **Public data only.** Sitemaps are published for crawlers. No login, no personal data, and `robots.txt` is honoured by default.

### How much does it cost to extract sitemap URLs?

| Event | Price | Charged when |
|---|---|---|
| Actor start | $0.00005 | Once per run (per GB of memory). |
| `sitemap-file` | $0.001 | One sitemap document was fetched and parsed successfully. |
| `url-record` | $0.00025 ($0.25 per 1,000) | One URL was written to the dataset. |

Apify subscribers on the Bronze, Silver and Gold plans pay 5%, 10% and 15% less per event; the Pricing tab shows the price for your plan.

A sitemap that 404s, times out or is blocked costs a request but produces nothing, so it is not charged. In change-tracking mode `url-record` bills only for the rows you actually receive — the unchanged URLs the Actor read and deliberately did not emit are free.

When the run reaches your maximum total charge it stops cleanly: it writes the rows it could pay for, skips any remaining targets, and records the reason in `RUN_SUMMARY` rather than failing. `RUN_SUMMARY.charging` shows the units of work performed next to the units actually billed.

### FAQ

#### How do I get only new or removed pages every day?

Set **Run mode** to change tracking and schedule the Actor (Console > Schedules). The first run saves a baseline and reports every URL as `added`; from the second run on you get only `added`, `changed` and `removed` rows, and pay only for those.

#### Can I export the sitemap to CSV or Excel?

Yes. Every run's dataset downloads as CSV, Excel, JSON or XML from the Output tab, or through the Apify API.

#### Why are some URLs marked `changed` on every run?

Some CMSs set `lastmod` to the build time of every page. On those sites rely on `added` and `removed`, which do not depend on `lastmod`.

#### Does it crawl my pages?

No. It reads only the sitemaps a site publishes (found through `robots.txt` and common locations), which is why it is fast and cheap. Pages missing from the sitemap are not discovered.

#### Can I send the results to Google Sheets, Slack or my own app?

Yes. Use Apify's integrations (Google Sheets, Slack, Zapier, Make, n8n, webhooks) or call the Actor from the API, so a scheduled run can push new URLs wherever you need them.

### Related Actors from opendatalabs

- [Broken Link Checker & 404 Finder for Websites](https://apify.com/opendatalabs/broken-link-site-qa-checker): crawl a site and find broken links, images, redirect chains and mixed content. Give it your sitemap with **Sitemap URL or domain** to check every page it lists.
- [Grants.gov Grant Opportunity Monitor](https://apify.com/opendatalabs/grants-gov-monitor): a daily feed of new US federal grants and deadline changes.

### Feedback and support

This Actor is new and actively maintained. If a run fails, a field is missing, or a result looks wrong, please open an issue on the **Issues** tab with the run ID. We read and reply to every issue, and suggestions for new filters or fields are very welcome.

# Actor input Schema

## `targets` (type: `array`):

What to extract. Give a bare domain (example.com) and the Actor finds the sitemaps through robots.txt, falling back to common paths such as /sitemap.xml and /wp-sitemap.xml. Give a direct sitemap URL and it is used as-is. Sitemap indexes, .xml.gz files, plain-text URL lists and RSS/Atom feeds are all handled. Leave empty to run a small demo on crawlee.dev (100 URLs).

## `mode` (type: `string`):

Snapshot returns every URL found. Change tracking compares this run against the previous run's baseline for the same target and labels each URL added, changed, removed or unchanged. The first run in change mode has no baseline, so everything is reported as added and the baseline is written for next time.

## `outputChangeTypes` (type: `array`):

Only used in change-tracking mode. Leave the default to get just the differences, which is what a scheduled monitor usually wants. Add 'unchanged' to get a full snapshot with change labels attached.

## `stateStoreName` (type: `string`):

Named key-value store that holds the previous run's URL set. A named store survives between runs, unlike the default store, which is what makes change tracking work. Use one store name per monitoring job; separate targets are kept under separate keys inside it.

## `maxSitemapFailurePercentForBaselineUpdate` (type: `integer`):

Safety valve for change tracking. If more than this percentage of a target's sitemaps fail to load, the baseline is left untouched, so a bad day at the origin server does not make the next run report the whole site as 'added'. A run that extracts zero URLs never overwrites the baseline, regardless of this setting.

## `checkUrlStatus` (type: `boolean`):

Sends a cheap HEAD request per URL (falling back to GET when a server rejects HEAD) and adds statusCode, statusOk and redirectedTo to every row. Useful for finding sitemap entries that 404. Off by default because it multiplies the number of requests.

## `statusCheckLimit` (type: `integer`):

Caps the HEAD checks so a 300,000-URL sitemap cannot turn into a 300,000-request run by accident.

## `onlyUrlsModifiedSince` (type: `string`):

YYYY-MM-DD. Drops URLs whose lastmod is older than this date. URLs with no lastmod at all are kept, because a missing date is not evidence that a page is old.

## `maxUrlsPerTarget` (type: `integer`):

Hard ceiling on URLs collected per domain. Sitemaps are walked breadth-first, so hitting this limit still returns a spread across the whole site rather than everything from one section. The default of 1,000 caps a run at about $0.25 per target; raise it for a full inventory.

## `maxSitemapsPerTarget` (type: `integer`):

Hard ceiling on sitemap documents fetched per domain. Large e-commerce sites routinely publish hundreds of child sitemaps behind one index.

## `maxSitemapDepth` (type: `integer`):

How many levels of nested sitemap index to follow. 0 means only the sitemaps given or discovered directly.

## `includeAlternates` (type: `boolean`):

Adds the xhtml:link hreflang alternates to each row. Snapshot mode always includes them when present; this switch controls the (usually much larger) change-mode rows.

## `respectRobotsTxt` (type: `boolean`):

Checks the sitemap path against robots.txt before fetching it. Leave this on. Sitemaps are published for crawlers, but a few sites still disallow specific sitemap paths.

## `discoverViaCommonPaths` (type: `boolean`):

When robots.txt declares no Sitemap: directive, try /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and nine other well-known locations. Turn this off to rely on robots.txt only.

## `requestsPerSecond` (type: `integer`):

Global rate ceiling across all targets. Lower this for small or slow origin servers.

## `concurrency` (type: `integer`):

How many sitemap or status requests are in flight at once. The rate limit above still applies.

## `maxRetries` (type: `integer`):

Total attempts, including the first. Retries use exponential backoff with jitter and honour Retry-After. Only transient failures (timeouts, 5xx, 429) are retried; a 404 fails immediately instead of burning four timeouts.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for a single sitemap or status request before giving up on that attempt. Large gzipped sitemap indexes on slow origins can legitimately take 30 seconds or more, so lowering this below the default will start costing you URLs.

## `userAgentContact` (type: `string`):

Appended to the User-Agent header, for example a URL or an email address, so site owners can see who is crawling them. Optional here; the Actor sends a descriptive User-Agent either way.

## Actor input object example

```json
{
  "targets": [
    "example.com",
    "https://www.example.org/sitemap_index.xml"
  ],
  "mode": "snapshot",
  "outputChangeTypes": [
    "added",
    "changed",
    "removed"
  ],
  "stateStoreName": "sitemap-change-tracker-state",
  "maxSitemapFailurePercentForBaselineUpdate": 25,
  "checkUrlStatus": false,
  "statusCheckLimit": 500,
  "maxUrlsPerTarget": 100,
  "maxSitemapsPerTarget": 5,
  "maxSitemapDepth": 5,
  "includeAlternates": false,
  "respectRobotsTxt": true,
  "discoverViaCommonPaths": true,
  "requestsPerSecond": 5,
  "concurrency": 10,
  "maxRetries": 4,
  "requestTimeoutSecs": 60,
  "userAgentContact": "+https://your-company.example/bot"
}
```

# Actor output Schema

## `results` (type: `string`):

Sitemap URL rows (snapshot mode) or added/changed/removed URL rows (change mode).

## `runSummary` (type: `string`):

Run summary: URLs found, rows written, per-target results (sitemaps, failures) and charged events.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "targets": [
        "https://crawlee.dev"
    ],
    "maxUrlsPerTarget": 100,
    "maxSitemapsPerTarget": 5
};

// Run the Actor and wait for it to finish
const run = await client.actor("opendatalabs/sitemap-change-tracker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "targets": ["https://crawlee.dev"],
    "maxUrlsPerTarget": 100,
    "maxSitemapsPerTarget": 5,
}

# Run the Actor and wait for it to finish
run = client.actor("opendatalabs/sitemap-change-tracker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "targets": [
    "https://crawlee.dev"
  ],
  "maxUrlsPerTarget": 100,
  "maxSitemapsPerTarget": 5
}' |
apify call opendatalabs/sitemap-change-tracker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,opendatalabs/sitemap-change-tracker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/kf5kh3od46Li0IBxU/builds/5yFXRh8wWEHuTTguq/openapi.json