# Sitemap Extractor & Change Monitor (No Proxy) (`saiberops/sitemap-extractor-change-monitor`) Actor

Every URL in a site's sitemaps with lastmod, changefreq and priority. Finds sitemaps via robots.txt, reads indexes, gzip, RSS and Atom, validates the protocol, checks llms.txt, and monitors added, removed and changed pages.

- **URL**: https://apify.com/saiberops/sitemap-extractor-change-monitor.md
- **Developed by:** [Saiber Operations](https://apify.com/saiberops) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / 1,000 url rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor & Change Monitor: All URLs, lastmod, robots.txt, llms.txt (No Proxy)

Get every URL a website publishes in its sitemaps, as clean rows with `lastmod`, `changefreq`, `priority` and hreflang counts, plus one report per site. Point it at a domain and it finds the sitemaps for you. Run it in **monitor mode** on a schedule and it returns only the pages that were **added, removed or updated** since last time.

**Built for:**

- **SEO teams and agencies.** URL inventories before a migration or audit, index-bloat checks, and protocol errors in the sitemap itself.
- **Competitor watching.** A daily or weekly list of the pages a competitor just published or took down.
- **RAG and AI-agent builders.** The full URL list of a site before you crawl it, in one fast call.
- **Developers.** Sitemap-to-JSON without running a crawler, a browser or a proxy.

### What it does

- **Finds sitemaps automatically:** reads `Sitemap:` lines in robots.txt, then tries common locations (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml` and others).
- **Reads every format:** sitemap indexes (nested), gzip (`.xml.gz`), plain-text sitemaps, and RSS 2.0 / Atom feeds.
- **Returns one row per URL** with `lastmod`, `changefreq`, `priority`, hreflang languages, image and video counts, and the sitemap file it came from. Duplicates across sitemaps are removed.
- **Validates the sitemap protocol:** invalid dates, future dates, bad `changefreq`/`priority`, non-HTTPS URLs, cross-host URLs, files over 50,000 URLs or 50 MB.
- **Site report per domain:** every sitemap file found (status, gzip, size, URL count, issues), totals, newest and oldest `lastmod`, robots.txt summary, and **llms.txt / llms-full.txt** presence.
- **Monitor mode:** saves a snapshot per site and, on the next run, outputs only `added`, `removed` and `changed` URLs, plus a `CHANGES.csv`.
- **Optional status checks:** HTTP status, final URL, content type and `Last-Modified` for each URL.

### Quick start

1. Add a domain like `https://www.example.com` (or a sitemap URL) to **Websites or sitemap URLs**.
2. Click **Start**. Open the **URLs** tab for the URL list and **Site reports** for the summary.
3. **To monitor changes:** set **Mode** to **Monitor changes since last run** and add a schedule (daily or weekly). The first run saves the baseline; later runs return only what changed.

### Sample output

A URL row:

```json
{
  "recordType": "url",
  "site": "https://www.example.com",
  "url": "https://www.example.com/blog/launch",
  "sourceSitemap": "https://www.example.com/sitemap-blog.xml.gz",
  "sitemapKind": "urlset",
  "lastmod": "2026-10-20T10:00:00+00:00",
  "changefreq": "weekly",
  "priority": 0.8,
  "hreflangCount": 2,
  "hreflangLanguages": ["en", "es"],
  "imageCount": 1,
  "videoCount": 0,
  "issues": [],
  "changeType": "added",
  "previousLastmod": null
}
```

A site report (abbreviated):

```json
{
  "recordType": "site",
  "site": "https://www.example.com",
  "discoveredVia": "robots",
  "robotsTxt": { "status": 200, "sitemapLines": ["https://www.example.com/sitemap_index.xml"] },
  "sitemaps": [{ "url": "https://www.example.com/sitemap-blog.xml.gz", "kind": "urlset", "status": 200, "gzip": true, "urlCount": 412, "issues": [] }],
  "totals": { "sitemapFiles": 4, "urlsUnique": 1873, "duplicates": 12, "truncated": false, "newestLastmod": "2026-10-23" },
  "llmsTxt": { "present": true, "h1": "Example", "sectionCount": 4, "llmsFullPresent": false },
  "monitor": { "firstRun": false, "added": 3, "removed": 1, "changed": 7, "unchanged": 1862 }
}
```

### Pricing

Pay per event: you pay only for results.

| Event | Free plan | Bronze | Silver | Gold |
|---|---|---|---|---|
| URL row output | $0.0006 | $0.0005 | $0.00045 | $0.0004 |
| Status check (only with "Check each URL's HTTP status") | $0.0006 | $0.0005 | $0.00045 | $0.0004 |
| Site report (once per site with a definitive answer) | $0.012 | $0.01 | $0.009 | $0.008 |

That's $0.40–$0.60 per 1,000 URLs, plus one cent per site and Apify's standard Actor start fee. In monitor mode, unchanged URLs are free, so a daily check of a site with no changes costs about one cent. **Not charged:** excluded URLs, sites blocked by bot protection or unreachable, and status checks that time out or get blocked. Set **Max cost per run** to cap any run.

### Integrations and API

Schedule it in Apify and send changes to Slack, email, Google Sheets, Make, Zapier or n8n through Apify's integrations, or call it from the Apify API. AI agents can use it through Apify's MCP server: give it a domain, get back the URL list.

### Limits

- Reads sitemaps only. Pages that aren't in any sitemap or feed aren't listed (this is not a crawler).
- Monitor snapshots are kept in a storage named `saiberops-sitemap-snapshots` in your Apify account, so monitor runs must use the same account. Removed-URL detection is skipped when a run hits **Max URLs per site**.
- Sites behind bot protection (for example a Cloudflare challenge on the sitemap) are reported as `POSSIBLY_BOT_BLOCKED` and not charged. No proxies are used.

### FAQ

**Why no proxy?** Sitemaps are published for machines to read. Plain, honest requests are enough for almost every site, and they keep runs fast and cheap.

**What does `SITEMAP_DISALLOWED_BUT_DECLARED` mean?** The site lists the sitemap in robots.txt (inviting crawlers to read it) but a `Disallow` rule also covers it. The Actor reads it because the site declared it, and flags the contradiction so you can fix it. Locations it only guesses at, and per-URL status checks, always obey `Disallow`.

**My own site shows `POSSIBLY_BOT_BLOCKED`.** Allowlist the `SaiberOpsBot` User-Agent in your firewall or CDN and re-run.

**Does it read the content of llms.txt?** Only its structure: presence, size, title (H1), number of sections and links, and whether an `Optional` section exists.

### Responsible use

The Actor identifies itself as `SaiberOpsBot`, respects robots.txt by default (per RFC 9309), never logs in or solves CAPTCHAs, and does not rotate IP addresses. It outputs URLs, dates, counts and HTTP metadata only. Feed titles, descriptions and author names are never read into the output.

### Support

Found a bug or need a feature? Open an issue on this Actor's **Issues** tab and we'll respond there.

*Maintained by Saiber Operations LLC.*

# Changelog

This Actor's version history is a separate document: https://apify.com/saiberops/sitemap-extractor-change-monitor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Domains (example.com), sitemap or sitemap-index URLs (including .xml.gz), robots.txt URLs, or RSS/Atom feed URLs. For a domain, sitemaps are found through robots.txt, then common locations.

## `mode` (type: `string`):

map: every URL found. monitor: compare with the snapshot saved by your previous run and output only URLs that were added, removed or had their lastmod change. The first monitor run saves the baseline.

## `maxUrlsPerSite` (type: `integer`):

Stops reading a site's sitemaps after this many URLs (the site report says truncated). Each output URL is one charged event.

## `checkStatus` (type: `boolean`):

Sends a HEAD request to each output URL and records status code, final URL, content type and Last-Modified. Charged per URL with a definitive answer.

## `includePatterns` (type: `array`):

Substrings, or regular expressions starting with re:, e.g. /blog/ or re:/products?/. Leave empty for all URLs.

## `excludePatterns` (type: `array`):

Substrings or re: regular expressions. Excluded URLs are not output or charged.

## `checkLlmsTxt` (type: `boolean`):

Adds /llms.txt and /llms-full.txt presence, size, title and section count to each site report.

## `probeCommonPaths` (type: `boolean`):

When robots.txt lists no sitemap, try /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and similar.

## `emitBaselineRows` (type: `boolean`):

On the first monitor run for a site, output every URL as a baseline row. Turn off to only save the snapshot (you then pay only for the site report).

## `emitUnchanged` (type: `boolean`):

Unchanged URLs are normally skipped (and free).

## `respectRobotsTxt` (type: `boolean`):

On by default. Sitemaps declared in robots.txt are read even if a Disallow rule covers them (flagged in the output); probed paths and status checks obey Disallow. Turn off only for sites you own.

## `maxIndexDepth` (type: `integer`):

How many levels of sitemap indexes to follow.

## `maxSitemapFilesPerSite` (type: `integer`):

Guards against huge indexes. The site report says when the limit was reached.

## `maxConcurrency` (type: `integer`):

Total parallel connections.

## `maxConcurrencyPerHost` (type: `integer`):

Keeps the load on each website polite.

## `requestTimeoutSecs` (type: `integer`):

Per request; sitemap files get twice this.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com"
  ],
  "mode": "map",
  "maxUrlsPerSite": 1000,
  "checkStatus": false,
  "includePatterns": [],
  "excludePatterns": [],
  "checkLlmsTxt": true,
  "probeCommonPaths": true,
  "emitBaselineRows": true,
  "emitUnchanged": false,
  "respectRobotsTxt": true,
  "maxIndexDepth": 3,
  "maxSitemapFilesPerSite": 500,
  "maxConcurrency": 10,
  "maxConcurrencyPerHost": 4,
  "requestTimeoutSecs": 20
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `changes` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com"
    ],
    "maxUrlsPerSite": 1000,
    "includePatterns": [],
    "excludePatterns": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("saiberops/sitemap-extractor-change-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com"],
    "maxUrlsPerSite": 1000,
    "includePatterns": [],
    "excludePatterns": [],
}

# Run the Actor and wait for it to finish
run = client.actor("saiberops/sitemap-extractor-change-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com"
  ],
  "maxUrlsPerSite": 1000,
  "includePatterns": [],
  "excludePatterns": []
}' |
apify call saiberops/sitemap-extractor-change-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,saiberops/sitemap-extractor-change-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ycJlnJyCz50zKcNTS/builds/KviCs8n6c9vXdDoFH/openapi.json