# Sitemap Monitor: All URLs + New Pages Since Last Run (`lindenbyte/sitemap-monitor-new-pages`) Actor

Extract every URL from any website's XML sitemaps (robots.txt discovery, indexes, .xml.gz, hreflang, images) and schedule it to get only NEW pages since the last run. Optional HTTP status and redirect check.

- **URL**: https://apify.com/lindenbyte/sitemap-monitor-new-pages.md
- **Developed by:** [Lindenbyte Studio](https://apify.com/lindenbyte) (community)
- **Categories:** SEO tools, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Monitor: Extract All URLs + Get New Pages Since Last Run

Point it at any website and get **every URL from its XML sitemaps**: robots.txt discovery, sitemap indexes, nested indexes, `.xml.gz`, plain-text sitemaps, `lastmod`, `changefreq`, `priority`, hreflang alternates and image counts. Then switch on **monitoring mode** and schedule it: each run outputs **only the pages that are new since the last run**. Optionally check the **HTTP status and redirects** of every URL.

### Why this one

- **New-pages feed, not just a dump**: `onlyNew` remembers every URL per site and returns only the new ones. Schedule it daily or hourly to see when a competitor launches a product, publishes a post or adds a landing page.
- **Cheap to start monitoring**: turn off `emitOnBaseline` and the first run just records the baseline (no URLs output, nothing charged per URL).
- **Finds the sitemaps for you**: give `example.com` and it reads robots.txt `Sitemap:` lines, then falls back to `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`. Or pass sitemap URLs directly.
- **Handles real-world sitemaps**: indexes up to 10 levels deep, gzip, text sitemaps, hreflang (`xhtml:link`), image sitemaps, big sites (hundreds of thousands of URLs).
- **Filter before you pay**: regex `urlPattern` (only `/blog/` or `/products/`) and `changedSinceDays` (by `lastmod`; old child sitemaps are not even downloaded). Filtered URLs are free.
- **Status check built in**: HEAD request (GET fallback) per URL with status code, final URL and redirect count. Finds 404s and redirect chains listed in your own sitemap. Polite: respects robots.txt and Crawl-delay, per-host throttle.
- **Pure HTTP, no browser**: fast and cheap to run. Optional proxy only for sites that block datacenter IPs (403/429): just those sites are retried through it, everything else stays direct.

### Use cases

- **Competitor monitoring**: new product pages, blog posts, landing pages, locations, job pages.
- **SEO audits**: full URL inventory, sitemap URLs that 404 or redirect, stale `lastmod`, hreflang coverage.
- **Content teams and agencies**: weekly list of what clients or competitors published.
- **Crawling pipelines**: feed only new URLs into a scraper instead of re-crawling the whole site.
- **E-commerce**: new SKUs on a competitor's store the day they go live.

### Input

| Field | What it does |
|---|---|
| `sites` | Domains (`example.com`), homepage URLs or sitemap URLs (`.xml`, `.xml.gz`, `.txt`) |
| `onlyNew` | Monitoring mode: output only URLs not seen on the previous run (state per site) |
| `emitOnBaseline` | Monitoring mode, first run: output all URLs (on) or only record the baseline (off) |
| `stateKey` | Name of the monitor; different names keep separate memories |
| `urlPattern` | Case-insensitive regex; only matching URLs are output |
| `changedSinceDays` | Only URLs whose `lastmod` is within the last N days |
| `checkStatus` | Add `statusCode`, `finalUrl`, `redirectCount` per URL |
| `maxUrlsPerSite` | Cost cap: max URLs output per site per run (default 10,000) |
| `maxScannedUrlsPerSite` | Safety cap on URLs read per site (default 200,000; reading is free) |
| `statusConcurrency`, `perHostDelayMs` | Status-check speed and politeness |
| `maxSitemapDepth` | Nested sitemap index levels to follow (default 3) |

#### Example: daily new-pages monitor

```json
{
  "sites": ["competitor.com", "https://other.com/sitemap_index.xml"],
  "onlyNew": true,
  "emitOnBaseline": false,
  "urlPattern": "/(blog|products)/"
}
```

Create a schedule (Console > Schedules) and connect an integration (email, Slack, webhook) to get the new URLs each day.

### Output

One row per URL (real example):

```json
{
  "site": "crawlee.dev",
  "url": "https://crawlee.dev/blog/archive",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "isNew": null,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "alternates": [],
  "alternatesCount": 0,
  "imageCount": 0,
  "statusCode": 200,
  "finalUrl": "https://crawlee.dev/blog/archive",
  "redirectCount": 0,
  "statusError": null,
  "scrapedAt": "2026-11-13T10:25:38.920Z"
}
```

`isNew` is `true` in monitoring mode, `null` otherwise. `alternates` lists hreflang `{ hreflang, href }` pairs. A per-site summary (sitemaps fetched, URLs scanned, new URLs, filtered counts, errors) is saved to the key-value store record `SUMMARY`.

### Pricing

Pay per result: you pay per URL saved to the dataset and per successful status check, plus a small start fee per run. URLs that are filtered out, already seen, or recorded as a silent baseline are free. If you enable the optional proxy, each request actually sent through it is one `proxied-request` event (sites that don't block you are never proxied, so they cost nothing extra). Set a max charge per run in the run options to cap cost.

### Limits (honest)

- Only pages listed in the site's sitemaps are found. Sites without a sitemap return an error for that site (it is not a crawler).
- "New" means "new in the sitemap since the last run". Removed URLs are not reported in this version.
- Some sites block datacenter IPs on their sitemaps (e.g. some news sites answer 403). Enable the optional proxy (residential works best) to retry just those sites; sites with browser-only bot challenges still fail with a clear error in the log and summary.
- Monitoring state lives in your account's named key-value store `sitemap-monitor-state`. Changing `stateKey` starts a fresh baseline.

### Legal

Reads public sitemaps and robots.txt that websites publish for automated discovery; status checks respect robots.txt. Do not use it to overload websites. Not affiliated with any website it reads.

# Actor input Schema

## `proxyConfiguration` (type: `object`):

Optional. Requests go out directly first; if a site answers 401/403/429 (it blocks datacenter IPs), only that site's requests are retried through this proxy. Each request sent through the proxy costs one extra 'proxied-request' event (see Pricing); sites that don't block are never proxied. Residential is the most reliable for news sites.

## `sites` (type: `array`):

Domains (example.com), homepage URLs, or direct sitemap URLs (https://example.com/sitemap.xml, .xml.gz, .txt). For a domain, sitemaps are found via robots.txt 'Sitemap:' lines, then /sitemap.xml, /sitemap_index.xml and /sitemap-index.xml.

## `maxUrlsPerSite` (type: `integer`):

Cost cap: at most this many URLs are saved (and charged) per site per run. In 'only new' mode this is the max NEW URLs per run; any remaining new URLs are output on the next run.

## `maxScannedUrlsPerSite` (type: `integer`):

Safety limit on how many unique URLs are read from a site's sitemaps (reading is free; only output is charged). In 'only new' mode keep it above the site's size, or new pages in unread sitemaps are missed. Above ~300,000 give the run 1 GB+ memory.

## `onlyNew` (type: `boolean`):

Remembers every URL seen per site in a named key-value store ('sitemap-monitor-state') and outputs only URLs that were not there on the previous run. Schedule the Actor (daily/hourly) to get a feed of new pages. You are charged only for URLs that are output.

## `emitOnBaseline` (type: `boolean`):

Only for 'only new' mode. The first run for a site has nothing to compare to. On = output all URLs (marked isNew). Off = silently record the baseline and output nothing (cheapest way to start monitoring).

## `stateKey` (type: `string`):

Only for 'only new' mode. Separate monitors (for example one per schedule or client) keep separate memories when they use different names. Change it to start a fresh baseline.

## `changedSinceDays` (type: `integer`):

Keep only URLs whose sitemap <lastmod> is within the last N days (URLs without lastmod are dropped). Child sitemaps whose own lastmod is older are not even downloaded. 0 or empty = off.

## `urlPattern` (type: `string`):

Optional case-insensitive regular expression; only matching URLs are output and charged. Examples: /blog/ , /products/ , ^https://example.com/(en|de)/ . Note: a homepage URL with a path in 'sites' still reads the whole domain's sitemaps, so use this to narrow to a section.

## `checkStatus` (type: `boolean`):

Send a HEAD request (GET if HEAD is refused) to each output URL and record statusCode, finalUrl and redirectCount (follows up to 5 redirects). Respects the site's robots.txt and Crawl-delay. Charged per successful check.

## `statusConcurrency` (type: `integer`):

Maximum simultaneous status-check requests (the per-host delay below still applies).

## `perHostDelayMs` (type: `integer`):

Politeness throttle for status checks: at most one request per host every N ms (a robots.txt Crawl-delay, if larger, wins, capped at 10 s).

## `maxSitemapDepth` (type: `integer`):

How many levels of nested sitemap indexes to follow (index -> index -> sitemap). 3 covers virtually every site.

## Actor input object example

```json
{
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "sites": [
    "https://docs.python.org",
    "https://crawlee.dev"
  ],
  "maxUrlsPerSite": 10000,
  "maxScannedUrlsPerSite": 200000,
  "onlyNew": false,
  "emitOnBaseline": true,
  "stateKey": "default",
  "checkStatus": false,
  "statusConcurrency": 5,
  "perHostDelayMs": 250,
  "maxSitemapDepth": 3
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "proxyConfiguration": {
        "useApifyProxy": false
    },
    "sites": [
        "https://docs.python.org",
        "https://crawlee.dev"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("lindenbyte/sitemap-monitor-new-pages").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "proxyConfiguration": { "useApifyProxy": False },
    "sites": [
        "https://docs.python.org",
        "https://crawlee.dev",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("lindenbyte/sitemap-monitor-new-pages").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "sites": [
    "https://docs.python.org",
    "https://crawlee.dev"
  ]
}' |
apify call lindenbyte/sitemap-monitor-new-pages --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,lindenbyte/sitemap-monitor-new-pages"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/6pED7CixfdvovbJ1y/builds/3QOzgAnBxt1zuZphp/openapi.json
