# Sitemap Extractor & Change Monitor (new/removed/updated URLs) (`datlab/sitemap-change-monitor`) Actor

Extract every URL from any website's sitemaps (robots.txt, sitemap index, nested, .gz, news/image/video, RSS/Atom fallback) with lastmod, changefreq and priority. Schedule it to get only added, removed and updated URLs, and optional page-content change checks.

- **URL**: https://apify.com/datlab/sitemap-change-monitor.md
- **Developed by:** [DatLab](https://apify.com/datlab) (community)
- **Features:** Limited permissions
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 url results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor & Change Monitor: all URLs, lastmod, new/removed pages

An **XML sitemap parser** and URL change monitor: give it a domain or a `sitemap.xml` URL and get **every URL the site publishes** in its XML sitemaps (sitemap indexes, gzip sitemaps, and Google News, image and video sitemaps), with `lastmod`, `changefreq` and `priority`. Run it on a schedule for an SEO sitemap audit, and it reports **only what changed** since the last run: **new pages, removed pages and pages whose `lastmod` moved**. You can also check a list of key pages for **content change detection**, comparing a hash of the main text so ads, scripts, navigation and footers don't trigger false alarms.

- ✅ **Automatic discovery**: robots.txt `Sitemap:` lines, then common paths (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, `.gz`, `.txt` …), then the site's **RSS/Atom feed** as a fallback
- ✅ **Every sitemap format**: sitemap indexes (nested up to 5 levels), **gzip** (`.xml.gz`, or gzip served without a `.gz` name), plain-text sitemaps, **Google News / image / video** extensions, RSS 2.0 and Atom
- ✅ **Change monitoring** between runs, with state kept in your own key-value store: `added`, `removed`, `lastmodChanged`
- ✅ **Page content checks**: main-text hash plus HTTP status, title and meta description, with a CSS selector and ignore patterns
- ✅ **Polite and robust**: obeys robots.txt (RFC 9309) and `Crawl-delay`, applies a per-host delay, retries 429/5xx, and never reports false "removed" URLs after a partial read
- ✅ **No browser, no proxy needed**: plain HTTP, so it is fast and cheap (tens of thousands of URLs per minute)

### Use cases

- **SEO**: audit what a site submits to Google, find URLs missing a `lastmod`, and catch sitemap regressions after a deploy
- **Competitor monitoring**: get alerted when a competitor publishes new product, pricing, landing or blog pages, or removes some
- **Content and news tracking**: new articles from news sitemaps (title, publication date) as a structured feed
- **Crawl seeding**: a clean URL list to feed other scrapers, filtered with include/exclude patterns
- **Change detection on key pages**: pricing pages, terms and policies, docs

### Input

| Field | Description |
|---|---|
| `startUrls` | Domains or any page URLs (`example.com`, `https://www.example.com/blog`). Sitemaps are discovered automatically. A URL that is itself a sitemap or feed is read directly. |
| `sitemapUrls` | Exact sitemap, index, `.gz`, RSS or Atom URLs, read as-is with no discovery |
| `includePatterns` / `excludePatterns` | `blog` or `/blog/` = substring (case-insensitive), `https://example.com/blog/*` = wildcard, `/\/20(25\|26)\//i` = regex |
| `maxUrls` | Cap **per site** (default 10,000; `0` = no limit). When capped, the newest child sitemaps are read first. |
| `maxSitemapsPerSite` | Safety cap on sitemap files per site (default 500) |
| `discoverSitemaps`, `rssFallback` | Turn off path probing or the RSS/Atom fallback |
| `includeExtensions` | Add news, image and video fields |
| `trackChanges` | Compare with the previous run and mark each URL `added` / `lastmodChanged` / `unchanged`, plus `removed` rows |
| `onlyChanges` | Output only the changes (this turns monitoring on). The first run saves a baseline and outputs nothing, unless `outputBaseline` is set. |
| `stateKey` | Monitor name. Use a different name for each schedule or task. |
| `pageCheckUrls` | Pages whose content should be compared between runs (max 1,000) |
| `pageCheckOnlyChanged` | Fetch only those pages whose sitemap entry is new or changed, plus pages never checked before |
| `pageCheckDetails`, `contentSelector`, `ignoreTextPatterns` | Also compare title and meta; hash only one part of the page; ignore noisy text such as view counters |
| `maxConcurrency`, `perHostDelaySecs`, `maxRequestsPerMinute` | Politeness controls (defaults 5 / 1 s / 120) |

Example: monitor a competitor's blog and pricing page every day:

```json
{
  "startUrls": ["competitor.com"],
  "includePatterns": ["/blog/", "/pricing"],
  "onlyChanges": true,
  "stateKey": "competitor-daily",
  "pageCheckUrls": ["https://competitor.com/pricing"]
}
```

Schedule it in Apify (for example daily) and connect a webhook, Slack, email or Google Sheets integration. Each run then delivers only the changes.

### Output

**URL rows** (`recordType: "url"`):

```json
{
  "recordType": "url",
  "url": "https://example.com/blog/new-post",
  "site": "example.com",
  "lastmod": "2026-10-29",
  "changefreq": "weekly",
  "priority": 0.8,
  "sourceType": "sitemap",
  "sitemapUrl": "https://example.com/post-sitemap.xml.gz",
  "scrapedAt": "2026-10-29T00:19:36.820Z",
  "changeType": "added",
  "previousLastmod": null
}
```

With `includeExtensions`, rows also carry `title` (news title), `newsPublicationName`, `newsPublicationDate`, `newsLanguage`, `imageUrls`, `imageCount`, `videoTitles` and `videoCount` when the sitemap has them. RSS/Atom rows use the item title (`title`) and date as `lastmod` (`sourceType: "rss"` / `"atom"`).

`changeType` is `added`, `removed`, `lastmodChanged`, `unchanged`, or `baseline` (first run). It is `null` when monitoring is off.

**Page-check rows** (`recordType: "page"`): `changeType` is `new` / `changed` / `unchanged` / `error` / `blockedByRobots`. Each row also has `changedFields` (`text`, `statusCode`, `title`, `metaDescription`), the current and previous `contentHash`, `statusCode`, `title`, `metaDescription`, `textLength`, plus `canonicalUrl`, `metaRobots` and `finalUrl`.

The dataset has three views: **URLs**, **Changes only** and **Page checks**. A `RUN_SUMMARY` record in the default key-value store lists, for each site, how sitemaps were found, how many were read, URL counts, whether the read was complete, and any problems. It also records run time and peak memory.

### Pricing

This Actor uses **pay per event**, so you pay only for what you get:

| Event | Price | When it's charged |
|---|---|---|
| URL result | **$0.30 per 1,000** ($0.0003 each) | Each URL row saved to the dataset |
| Page check | **$0.002 per page** | Each page from `pageCheckUrls` that is actually fetched and compared |

- Full export: 10,000 URLs = $3.00.
- **Monitoring** (`onlyChanges`): you're charged only for change rows (`added`, `removed`, `lastmodChanged`). The first run saves a baseline for free unless you set `outputBaseline`, and a run with no changes produces no URL charges.
- Page checks: 50 pages checked daily ≈ $0.10/day. Pages blocked by robots.txt or that fail to load aren't charged. Use `pageCheckOnlyChanged` to fetch only pages whose sitemap entry changed.
- Set **Maximum cost per run** in the run options to cap spending. The Actor stops cleanly and saves its state.

The live price list is on the **Pricing** tab of this page.

### FAQ

**Does it work on any website?**
Yes, if the site publishes an XML sitemap, a plain-text sitemap or an RSS/Atom feed. Most CMSs (WordPress, Shopify, Wix, Webflow, Next.js sites …) do.

**How do I get alerts for new pages?**
Turn on `onlyChanges` and add a schedule (e.g. daily). The first run stores a baseline. Later runs output only added, removed and updated URLs. Send them to Slack, email, Google Sheets or a webhook with Apify integrations.

**Where is the monitoring state stored?**
In a named key-value store in your own Apify account, so it survives between runs. Nothing is shared with other users.

**Does it render JavaScript?**
No. Sitemaps don't need a browser, and page checks compare the HTML text the server returns, which keeps runs fast and cheap.

**Why are no URLs reported as removed?**
Removals are only reported after two complete reads in a row, to avoid false alarms (see below).

### How change detection stays accurate

- A URL is reported as **removed** only if the previous run *and* this run read the site's sitemaps **completely**: no failed sitemap, no `maxUrls` cap and no spending limit hit. Otherwise the new URLs are merged into the saved state and removals wait for the next complete run.
- If you change the include/exclude patterns, the monitor resets to a fresh baseline for that site instead of reporting the filtered-out URLs as removed.
- Some sites generate their sitemap on every request, so every `lastmod` equals "now". The Actor detects this (`lastmodLooksDynamic` in `RUN_SUMMARY`). From the next run it **ignores lastmod changes for that site**, so you only get real added/removed URLs. Use page checks for content changes there.
- Page checks hash the visible **main text** only. Scripts, styles, hidden elements, header, footer, nav and aside are dropped, and you can narrow further with `contentSelector` and `ignoreTextPatterns`.

### Privacy & politeness

- Only reads public sitemaps, feeds and the pages you list. It does not extract emails, names or other personal data. Page checks store only a hash, the text length, the title and the meta description.
- robots.txt is respected. The Actor's own `User-agent` token is `SitemapChangeMonitor`, falling back to `*`. Sitemaps a site lists in robots.txt are considered invited. Pages disallowed for crawlers are reported as `blockedByRobots` and not fetched.
- `Crawl-delay` is honoured up to 30 s. **Note:** sites with a long delay (e.g. Hacker News: 30 s) make discovery slow, because each probe waits for the delay.

### Limits & tips

- The cap is per site, so one huge site can't starve the others. Raise `maxUrls` (or set `0`) for full exports of big sites. 1M URLs need roughly 1–2 GB memory.
- No sitemap? The Actor falls back to the site's RSS/Atom feed. That usually lists only the latest 10–50 items, which suits "new page" alerts but is not a full URL list.
- Sites that block datacenter IPs can be read through the Apify Proxy (`proxyConfiguration`).

### Legal & data use

- The Actor reads only public sitemaps and feeds, plus the pages you list. It obeys robots.txt and `Crawl-delay` and doesn't log in or bypass access controls.
- It doesn't collect personal data. Output is URLs and page metadata (status, title, meta description, content hash).
- You're responsible for monitoring only sites you're allowed to, and for using the results in line with those sites' terms and applicable law.

### Running on Apify

You can run this Actor on the Apify platform from the Console, the API, a schedule, integrations (webhooks, Slack, Google Sheets, Make, Zapier) or MCP.

### Other DatLab actors

- [Web Page to Markdown](https://apify.com/datlab/web-page-to-markdown) — turn a list of URLs into Markdown for LLMs and RAG
- [Website to Markdown Crawler](https://apify.com/datlab/site-to-markdown-crawler) — crawl a site into Markdown, with sitemaps and llms.txt
- [ATS Job Listings](https://apify.com/datlab/ats-job-listings) — Personio, Teamtailor, Workable and Recruitee, one schema
- [SAM.gov Contract Opportunities](https://apify.com/datlab/sam-gov-contract-opportunities) — US federal opportunities from the official daily extract, no API key

# Actor input Schema

## `startUrls` (type: `array`):

Domains or any page URL (e.g. `example.com`, `https://www.example.com/blog`). Sitemaps are found from robots.txt `Sitemap:` lines, then common paths (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, …), then RSS/Atom feeds. A URL that is itself a sitemap or feed (ends in .xml, .xml.gz, .txt, /feed …) is read directly.

## `sitemapUrls` (type: `array`):

Optional. Exact sitemap, sitemap index, .gz, RSS or Atom URLs to read as-is (no discovery).

## `includePatterns` (type: `array`):

Keep only URLs matching at least one pattern. Plain text or a path like `/blog/` = substring (case-insensitive); `*` = wildcard over the whole URL (e.g. `https://example.com/blog/*`); `/regex/i` = regular expression.

## `excludePatterns` (type: `array`):

Drop URLs matching any of these patterns (same syntax as include).

## `maxUrls` (type: `integer`):

Stop reading a site after this many URLs (after filters). 0 = no limit. When the cap is hit, the newest child sitemaps (by lastmod) are read first and removals are not reported for that run.

## `maxSitemapsPerSite` (type: `integer`):

Safety cap on the number of sitemap files (including nested ones) read per site.

## `discoverSitemaps` (type: `boolean`):

When robots.txt lists no sitemap, try /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt and similar.

## `rssFallback` (type: `boolean`):

When no sitemap exists, use the RSS/Atom feed linked from the home page (or /feed, /rss.xml, /atom.xml …).

## `includeExtensions` (type: `boolean`):

Add Google News (title, publication, date), image URLs and video titles when the sitemap has them.

## `trackChanges` (type: `boolean`):

Save the URL list after each run and mark every URL as added, lastmodChanged or unchanged, plus removed URLs. The first run records the baseline.

## `onlyChanges` (type: `boolean`):

Output only added, removed and lastmod-changed URLs (and changed pages). Turns on monitoring. Ideal for schedules + webhooks/integrations.

## `outputBaseline` (type: `boolean`):

With 'Output only changes', the first run normally outputs nothing and only records the baseline. Turn on to output the full list on that run too.

## `stateKey` (type: `string`):

Name of the saved state. Use a different name for each separate monitor (e.g. per task/schedule). State is kept in your named key-value store `sitemap-change-monitor-state`.

## `pageCheckUrls` (type: `array`):

Optional. These pages are downloaded and their main text is hashed (scripts, navigation, header/footer removed) to detect content changes between runs. Max 1000. Respects robots.txt.

## `pageCheckOnlyChanged` (type: `boolean`):

With monitoring on, fetch only pages from the list that are new or whose sitemap lastmod changed (plus never-checked pages). Saves requests and cost.

## `pageCheckDetails` (type: `boolean`):

Also compare and output title, meta description, canonical URL and meta robots.

## `contentSelector` (type: `string`):

Optional. Hash only the text inside this CSS selector (e.g. `#pricing`, `article .price`). Default: <main>, \[role=main] or <article>, else the body without header/footer/nav.

## `ignoreTextPatterns` (type: `array`):

Text removed before hashing so it doesn't count as a change, e.g. `/\d+ views/` or `Last updated`. Plain text or `/regex/flags`.

## `maxConcurrency` (type: `integer`):

Parallel requests across all sites.

## `perHostDelaySecs` (type: `number`):

Polite per-host delay (min 0.25). A robots.txt Crawl-delay is honoured up to 30 s.

## `maxRequestsPerMinute` (type: `integer`):

Upper limit on requests per minute across all hosts. A host's robots.txt Crawl-delay is always respected on top of this.

## `proxyConfiguration` (type: `object`):

Usually not needed: sitemaps are public. Use a proxy only if a site blocks datacenter IPs.

## Actor input object example

```json
{
  "startUrls": [
    "apify.com",
    "https://crawlee.dev"
  ],
  "maxUrls": 10000,
  "maxSitemapsPerSite": 500,
  "discoverSitemaps": true,
  "rssFallback": true,
  "includeExtensions": true,
  "trackChanges": false,
  "onlyChanges": false,
  "outputBaseline": false,
  "stateKey": "default",
  "pageCheckOnlyChanged": false,
  "pageCheckDetails": true,
  "maxConcurrency": 5,
  "perHostDelaySecs": 1,
  "maxRequestsPerMinute": 120,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All dataset items produced by the run.

## `runSummary` (type: `string`):

RUN_SUMMARY record with per-source status, counts and timing.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "apify.com",
        "https://crawlee.dev"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("datlab/sitemap-change-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        "apify.com",
        "https://crawlee.dev",
    ],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("datlab/sitemap-change-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "apify.com",
    "https://crawlee.dev"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call datlab/sitemap-change-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datlab/sitemap-change-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/vFE8NaBAekbmiU7kK/builds/azCY47AuEPaL90W5j/openapi.json