# Sitemap Extractor & New Page Monitor (`metallic_pastry/sitemap-page-monitor`) Actor

Extract every URL from any website's sitemaps (index, gzip, news, image, hreflang, RSS and text) and get new, updated and removed pages since the last run. Optional HTTP status check and a sitemap health report with SEO issues.

- **URL**: https://apify.com/metallic_pastry/sitemap-page-monitor.md
- **Developed by:** [Simon Liarsgård](https://apify.com/metallic_pastry) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 sitemap urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor & New Page Monitor

Get every page URL a website publishes in its sitemaps, and find out which pages are **new, updated or removed** since your last run. Give it a domain or a sitemap URL. It finds the sitemaps itself (robots.txt, `sitemap.xml`, `sitemap_index.xml`, WordPress, gzip and more), follows nested sitemap indexes and returns clean rows with lastmod, section, images, hreflang and Google News data. You can add an HTTP status check per URL and get a sitemap health report with SEO issues.

No login, no API key, no proxies. It only reads the files the website publishes for crawlers.

### What you can use it for

- **Competitor monitoring**: schedule it daily on competitors' sites and get only the new product pages, blog posts, landing pages or pricing pages they added.
- **SEO audits and migrations**: export the full URL inventory, check that every sitemap URL answers 200 and isn't redirected or noindexed, and compare the URL list before and after a migration.
- **Content tracking**: follow news sitemaps and see new articles with title and publication date within minutes of publishing.
- **Crawl seeding**: feed the URL list into a scraper (for example Website Content Crawler) so you scrape only pages that changed since the last run.
- **Lead and market research**: count how many products, locations or job pages a company has, and how fast that changes.

### Input

| Field | What it does |
|---|---|
| `websites` | Domains, sections or sitemap URLs, one per line. `allbirds.com` finds all sitemaps on the site. `wordpress.org/news` keeps only URLs under `/news`, uses the section's own sitemap when it has one, and keeps its own change history. `https://techcrunch.com/news-sitemap.xml` reads only that sitemap and its children. |
| `outputMode` | `urls` (default): one row per URL, marked `new`, `updated` or `unchanged` since the last run. `changes`: only new, updated and removed pages. `summary`: one health report per website. |
| `includePatterns` / `excludePatterns` | Keep or drop URLs containing a text such as `/blog/`, or matching a `*` pattern against the whole URL. Case-insensitive. |
| `lastmodWithinDays` | Keep only URLs modified (or published, for news) in the last N days. `0` keeps everything. |
| `trackChanges` / `stateKey` | Remember URLs between runs (on by default). Use a tracking key to keep separate histories, for example one per client. |
| `checkStatus` / `maxStatusChecksPerSite` | Add-on: request each output URL and add the status code, redirect chain, final URL, X-Robots-Tag and response time. |
| `maxUrlsPerSite` / `maxSitemapsPerSite` | Safety limits. Default 20,000 URLs and 500 sitemap files per website. |

Example: monitor new product and blog pages on two shops.

```json
{
    "websites": ["allbirds.com", "gymshark.com"],
    "outputMode": "changes",
    "includePatterns": ["/products/", "/blogs/"]
}
```

The first `changes` run saves a baseline for free. Every later run returns only what changed.

### Output

Every URL row (`type: "url"`):

```json
{
    "type": "url",
    "website": "allbirds.com",
    "url": "https://www.allbirds.com/products/mens-tree-runners",
    "changeType": "new",
    "lastmod": "2026-10-30T03:38:19.000Z",
    "previousLastmod": null,
    "changefreq": "daily",
    "priority": null,
    "section": "/products",
    "imageCount": 1,
    "videoCount": 0,
    "hreflangCount": 0,
    "alternates": [],
    "newsTitle": null,
    "newsPublishedAt": null,
    "sitemap": "https://www.allbirds.com/sitemap_products_1.xml?from=1878193471557&to=7369944137808",
    "scannedAt": "2026-10-30T08:39:57.961Z"
}
```

With the status check on, rows also get `statusCode`, `finalUrl`, `redirectCount`, `redirectChain`, `xRobotsTag`, `responseTimeMs` and `checkError`.

`changeType` is `null` on the first run of a website, then `new`, `updated` (lastmod changed), `unchanged` or `removed` (changes mode only).

Each website also gets one row (`type: "site"`) with its status (`ok`, `partial`, `not_found`, `error`), how the sitemaps were found, the number of sitemaps and URLs, and the change counts. In `summary` mode that row is the full report:

- duplicate and invalid URLs, share of URLs with a lastmod, newest and oldest lastmod;
- pages updated in the last 7, 30 and 90 days;
- hreflang, image and news URL counts;
- the top 20 sections (first path segment) by URL count;
- every sitemap file read, with its status and size;
- SEO issues with a code, a count, an explanation and an example.

Issue codes include `SITEMAP_NOT_IN_ROBOTS_TXT`, `SITEMAP_FETCH_FAILED`, `INVALID_SITEMAP`, `SITEMAP_TOO_LARGE`, `EMPTY_SITEMAP`, `DUPLICATE_URLS`, `INVALID_LASTMOD`, `FUTURE_LASTMOD`, `UNIFORM_LASTMOD`, `CROSS_HOST_URLS`, `HTTP_URLS_ON_HTTPS`, `BROKEN_URLS`, `REDIRECTING_URLS`, `NOINDEX_URLS` and `UNREACHABLE_URLS`.

The dataset has views for URLs, Changes, Status checks and Websites. A `RUN_SUMMARY` record in the key-value store has totals for the run.

### Supported formats

- XML sitemaps and sitemap indexes, nested up to 5 levels;
- gzip (`.xml.gz`), plain-text sitemaps and RSS/Atom feeds;
- image, video, hreflang (`xhtml:link`) and Google News extensions;
- WordPress, Yoast, Rank Math, Shopify, Squarespace, Wix, Webflow and custom generators.

Some generators stamp the same build time on every URL in a file. The Actor detects this and ignores those dates, so a regenerated sitemap doesn't mark every page as updated. It reports it as `UNIFORM_LASTMOD` instead.

### Pricing

Pay per event, plus a tiny start fee:

| Event | Price | When |
|---|---|---|
| Sitemap URL | $0.0002 ($0.20 per 1,000) | each URL row in `urls` mode, or each new, updated or removed page in `changes` mode |
| Status-checked URL | $0.0008 ($0.80 per 1,000) | add-on, each URL checked for its HTTP status |
| Website sitemap report | $0.005 | each website in `summary` mode |

You don't pay for:

- the baseline on the first `changes` run, or unchanged pages in later `changes` runs;
- website rows in `urls` and `changes` mode;
- websites without a sitemap, unreachable websites and invalid input;
- URLs skipped by your filters, and status checks blocked by robots.txt.

Monitoring a site daily in `changes` mode usually costs a few cents a month. Set a maximum charge on the run if you want a hard ceiling. The Actor then stops cleanly and doesn't save a partial history.

### How it works and what it respects

The Actor reads robots.txt and the sitemap files a website publishes for search engines. It doesn't crawl pages or follow links. Requests to the same host are paced, and it identifies itself as `SitemapPageMonitor` in the user agent. If robots.txt disallows that agent from a sitemap, the sitemap is skipped. The optional status check sends one HEAD request per URL (GET when HEAD isn't supported) and also respects robots.txt.

### Limits

- Only URLs listed in sitemaps are found. Pages a site leaves out of its sitemap won't appear.
- `updated` depends on the site's lastmod values. Sites without lastmod only report new and removed pages.
- Removed pages are only reported when the whole sitemap was read. If `maxUrlsPerSite` or `maxSitemapsPerSite` cut the read short, removals are skipped and the site row says so.
- Some large sites block requests from cloud servers. Their sitemaps then come back as `not_found` with `SITEMAP_FETCH_FAILED` and the HTTP status, at no charge.
- Change history is kept for up to 300,000 URLs per website and tracking key.

### Support

Found a sitemap the Actor can't read, or need another field? Open an issue on the Actor page with the URL, and it will usually be fixed in the next version.

# Changelog

This Actor's version history is a separate document: https://apify.com/metallic_pastry/sitemap-page-monitor/changelog.md

# Actor input Schema

## `websites` (type: `array`):

One per line. A domain or homepage ("allbirds.com") finds the sitemaps through robots.txt and the usual locations. A section URL ("wordpress.org/news") does the same but keeps only URLs under that path. A sitemap URL ("https://techcrunch.com/news-sitemap.xml") reads only that sitemap and its children. Duplicates are removed.

## `outputMode` (type: `string`):

"urls" outputs one row per page URL. "changes" outputs only pages that were added, updated or removed since the previous run (the first run saves a free baseline). "summary" outputs one report per website with counts, freshness, sections and SEO issues.

## `includePatterns` (type: `array`):

Optional. Keep only URLs containing one of these texts, e.g. "/blog/" or "/products/". Use \* for a wildcard pattern matched against the whole URL, e.g. "https://*.example.com/*/pricing". Case-insensitive.

## `excludePatterns` (type: `array`):

Optional. Drop URLs containing one of these texts or matching a \* pattern, e.g. "/tag/" or "?page=".

## `lastmodWithinDays` (type: `integer`):

0 keeps everything. Otherwise keep only URLs whose sitemap lastmod (or news publication date) is within this many days. URLs without a date are dropped when this is set. Removed pages in changes mode are always reported.

## `trackChanges` (type: `boolean`):

Remember each website's URLs and lastmod dates so the next run can mark pages as new, updated or removed. Always on in changes mode. Schedule the Actor daily or weekly to monitor a site.

## `stateKey` (type: `string`):

Optional. Separate change histories for the same website, for example one per client or per schedule. Letters, digits, - and \_ only.

## `checkStatus` (type: `boolean`):

Request each output URL (HEAD, GET fallback) and add the status code, redirect chain, final URL, X-Robots-Tag and response time. Charged per checked URL. In summary mode a sample is checked for the health report. URLs blocked by robots.txt are skipped and free.

## `maxStatusChecksPerSite` (type: `integer`):

Upper limit on checked URLs per website, to keep cost and load on the site predictable.

## `maxUrlsPerSite` (type: `integer`):

Stop reading sitemaps after this many unique URLs. Removed pages are only reported when the full sitemap was read, so raise this for large sites you monitor.

## `maxSitemapsPerSite` (type: `integer`):

Upper limit on sitemap files (including nested index files) fetched per website.

## `concurrency` (type: `integer`):

How many websites are read at the same time. Requests to the same host are always paced.

## Actor input object example

```json
{
  "websites": [
    "allbirds.com",
    "https://techcrunch.com/news-sitemap.xml"
  ],
  "outputMode": "urls",
  "lastmodWithinDays": 0,
  "trackChanges": true,
  "checkStatus": false,
  "maxStatusChecksPerSite": 1000,
  "maxUrlsPerSite": 20000,
  "maxSitemapsPerSite": 500,
  "concurrency": 3
}
```

# Actor output Schema

## `urls` (type: `string`):

Default dataset, URLs view.

## `changes` (type: `string`):

Default dataset, Changes view: new, updated and removed pages.

## `sites` (type: `string`):

Default dataset, Websites view: one row per website with counts, changes and issues.

## `runSummary` (type: `string`):

RUN_SUMMARY record in the default key-value store: websites scanned, URLs found, rows, changes and charged events.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websites": [
        "allbirds.com",
        "https://techcrunch.com/news-sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("metallic_pastry/sitemap-page-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "websites": [
        "allbirds.com",
        "https://techcrunch.com/news-sitemap.xml",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("metallic_pastry/sitemap-page-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websites": [
    "allbirds.com",
    "https://techcrunch.com/news-sitemap.xml"
  ]
}' |
apify call metallic_pastry/sitemap-page-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,metallic_pastry/sitemap-page-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/BmtSoZga66K09rXei/builds/DI2fyow8cxlOuK40a/openapi.json