# Sitemap Extractor & Website URL Change Monitor (`vdb997/sitemap-growth-watch`) Actor

Track sitemap growth for SEO and content pipelines. Find newly listed URLs, lastmod changes and URLs removed from a fully scanned sitemap tree. XML, gzip and nested indexes; persistent history, coverage reports and one flat price per root check.

- **URL**: https://apify.com/vdb997/sitemap-growth-watch.md
- **Developed by:** [Victor Diaz](https://apify.com/vdb997) (community)
- **Categories:** Marketing, Automation, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $200.00 / 1,000 sitemap root checks

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor & Website URL Change Monitor

Track a website's sitemap inventory for SEO research, competitor publishing and content pipelines. Find newly listed URLs, changes to the site's declared `lastmod`, and URLs removed from a completely scanned sitemap tree. Feed the resulting URL deltas into your crawler or knowledge-base refresh workflow.

### Who this is for

**Sitemap monitoring for SEO and content pipelines.** Use the sitemap extractor to find URLs entering or leaving a website's sitemap inventory and to track changes in declared lastmod dates. Pass URL deltas into an SEO review or downstream crawler. A sitemap removal is not proof of an HTTP 404, and lastmod is not proof of a content edit.

### Quick start

Export sitemap URLs immediately:

```json
{"sitemapUrls":["https://docs.apify.com/sitemap.xml"],"maxUrlsPerRoot":1000,"maxSitemapsPerRoot":10,"outputMode":"all"}
```

Monitor changes on a schedule:

```json
{"sitemapUrls":["https://docs.apify.com/sitemap.xml"],"maxUrlsPerRoot":10000,"maxSitemapsPerRoot":20,"urlContains":["/platform/"],"watchId":"docs-growth","outputMode":"changes","trackRemovals":true}
```

Paste the **exact sitemap URL**, not a website home page or article. XML sitemaps, gzip XML and nested sitemap indexes are supported. Indexes are followed on the same HTTPS origin, up to four nested levels. The Actor does not guess sitemap locations, discover them from robots.txt, crawl pages or verify page status codes.

### Change tracking

The first successful changes-mode run establishes a silent baseline and returns a check summary. Later runs with the same Watch ID return newly observed URLs and changed `lastmod` values. The optional URL-contains filter is a case-sensitive OR filter applied after comparison; all scanned URLs are remembered, so changing filters does not replay existing URLs.

**Removal requires two consecutive complete scans.** A capped scan preserves unseen entries and disables removals. After an incomplete scan, the next complete scan re-establishes the removal baseline. A source or parsing failure preserves the previous snapshot and does not generate a check charge. This avoids treating a missing child sitemap, robots restriction or coverage limit as deleted pages.

`removed` means no longer listed in the scanned sitemap tree. It does not establish that a page was deleted or returns HTTP 404. `lastmod` is exactly what the website declares; it is not proof the content changed. Sites that update every `lastmod` on each build may produce many alerts. URLs without dates can still be new or removed, but cannot reveal body edits.

### Output

Each `recordType=url` contains `pageUrl`, `lastmod`, `changeFrequency`, numeric `priority` when valid, source `sitemapUrl`, `rootSitemapUrl`, `changeType`, `changedFields`, `previousLastmod`, `firstSeenAt` and `observedAt`.

`changeType` is `baseline`, `new`, `changed`, `unchanged` or `removed`. Only `lastmod` changes trigger `changed`; priority, change frequency and source-file moves are informational. `new` means new to your Watch ID, not necessarily newly published.

A `check_summary` is always returned per successful root with URL/file counts, the sitemap-file inventory, duplicates, `coverageComplete`, `coverageLimits`, `removalsCompared` and change counts. Inspect coverage before treating an inventory as complete. `source_error` describes failed roots. The key-value store's `SUMMARY` contains the overall run and spending-limit status.

### Pricing

**$0.20 per successful root-sitemap check**, including up to 10,000 unique URLs and 20 sitemap files, plus **$0.00005 per run start** at fixed 512 MB memory. Platform usage is included. No per-URL or per-summary charge.

Each input root is a separate check, even if two roots overlap. Empty, unchanged and successfully capped checks count; their coverage is reported explicitly. Failed checks do not generate the root event; the standard start fee still applies. Ten roots cost $2.00005 per run, or $60.0015 if checked daily for 30 days. These are buyer costs, not developer earnings. Maximum cost per run is checked before starting each root.

### Coverage and source protections

- Up to ten roots per run; 1–10,000 unique URLs per root (default 1,000); 1–20 sitemap files including the root (default ten). Four nested index levels, a two-minute source-check budget, four MB compressed/wire data and 16 MB decoded XML per file. Limits are lower than the sitemap protocol's maximums; large sites may need separate roots.
- Strict valid XML with no DOCTYPE or entity declarations. RSS/Atom, HTML, plain-text maps, image/video detail records and cross-origin child sitemaps are unsupported. Only the main page loc is exported; media-extension locs are not treated as pages.
- Reads robots.txt before sitemap files, using the `PublicWatch` product token. Disallowed maps fail the root. Missing robots.txt (404/410) is allowed; other failures and non-text responses reject the check. At least half a second between sitemap requests; longer declared crawl delays are honored up to five seconds. Longer delays are unsupported within this monitor's time budget.
- Only public HTTPS hostnames can be fetched. Private/reserved IPs, credential URLs, cross-origin sitemap redirects and protection pages are rejected. TLS validation stays enabled; there is no login, proxy or blocking bypass. Page URLs are listed without fetching their contents.
- Conflicting `lastmod` values for duplicate URLs reject the check instead of inventing a change. Sitemap documents can change during pagination; the result is not an atomic site snapshot.
- History retains up to 30,000 URL entries per root, prioritizing recently observed entries; partial scans can accumulate entries up to that cap. Repeated truncation may cause evicted URLs to be observed as new again. Full page bodies are never stored. A new Watch ID resets history. Do not overlap runs using the same Watch ID.

Source format: [sitemaps.org protocol](https://www.sitemaps.org/protocol.html).

Related tools: [WordPress Content Watch](https://apify.com/vdb997/wordpress-content-watch), [Google News Keyword Watch](https://apify.com/vdb997/google-news-keyword-watch), [Steam Review Change Watch](https://apify.com/vdb997/steam-review-change-watch).

### Try one source before scheduling

1. Open the Input tab and use one supported source with `outputMode: "all"` for an immediate export.
2. Check the returned records and the check summary. Confirm the source, coverage and price suit your workflow.
3. To monitor, set a stable Watch ID and `outputMode: "changes"`. The first successful run establishes a silent baseline.
4. Reuse that Watch ID on later runs. A summary without change records means the check completed without matching changes.
5. Add an Apify schedule after the manual checks work. Avoid overlapping runs with the same Watch ID. Use Maximum cost per run to cap event charges.

Public-data monitors by the same developer: [browse the catalog](https://apify.com/vdb997).

# Actor input Schema

## `sitemapUrls` (type: `array`):

Exact HTTPS XML sitemap, XML gzip file or sitemap-index URLs. Nested maps must be on the same origin. This Actor does not guess sitemap locations.

## `maxUrlsPerRoot` (type: `integer`):

Coverage cap. Missing URLs are never classified as removed after a capped scan.

## `maxSitemapsPerRoot` (type: `integer`):

Includes the root and nested indexes. Files are read sequentially; coverage limits are reported.

## `urlContains` (type: `array`):

Case-sensitive OR substring filter applied to output only. All scanned URLs remain in history.

## `trackRemovals` (type: `boolean`):

Only after two complete scans with the same root. A removal means absent from the sitemap, not a verified deleted page.

## `watchId` (type: `string`):

Reuse an ID for history in your account. A new ID resets history. Do not overlap runs with the same ID.

## `outputMode` (type: `string`):

Changes mode begins with a silent baseline; check summaries and source errors are always returned.

## Actor input object example

```json
{
  "sitemapUrls": [
    "https://docs.apify.com/sitemap.xml"
  ],
  "maxUrlsPerRoot": 1000,
  "maxSitemapsPerRoot": 10,
  "urlContains": [],
  "trackRemovals": true,
  "outputMode": "all"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sitemapUrls": [
        "https://docs.apify.com/sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("vdb997/sitemap-growth-watch").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sitemapUrls": ["https://docs.apify.com/sitemap.xml"] }

# Run the Actor and wait for it to finish
run = client.actor("vdb997/sitemap-growth-watch").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sitemapUrls": [
    "https://docs.apify.com/sitemap.xml"
  ]
}' |
apify call vdb997/sitemap-growth-watch --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,vdb997/sitemap-growth-watch"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/kG0U5omAqwe5Q6yjh/builds/G5bMtRXDa8fCPaimo/openapi.json