# Sitemap URL Extractor & New Page Monitor (`thomson-digital/sitemap-url-monitor`) Actor

Get every URL from any website's sitemap.xml (auto-discovered via robots.txt, .gz and nested indexes) with lastmod, hreflang, image and news data. Filter by date or regex, check status codes, or monitor competitors for new pages. Fast, no browser, no proxy.

- **URL**: https://apify.com/thomson-digital/sitemap-url-monitor.md
- **Developed by:** [Matt](https://apify.com/thomson-digital) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.30 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor & New Page Monitor

Extract **every URL from any website's sitemap** in seconds — just enter a domain. The Actor finds the sitemaps for you (robots.txt, `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, `.xml.gz`, text sitemaps), follows nested sitemap indexes, and returns a clean list of page URLs with `lastmod`, `changefreq`, `priority`, hreflang alternates, image, news and video data.

Turn on **monitor mode** and schedule it to get **only the new pages** a website published since the last run — new products, blog posts, landing pages, job pages or docs from your competitors.

No browser, no proxies, no login: it only reads the public sitemap files that websites publish for crawlers, so it is fast and cheap.

### What can it do?

- 🔎 **Auto-discovers sitemaps** from `robots.txt` and common locations; or give it a sitemap URL directly
- 🗂️ **Follows sitemap indexes** (any depth), gzip (`.xml.gz`), text and RSS/Atom sitemaps
- 🧾 **Rich output**: `lastmod`, `changefreq`, `priority`, hreflang `alternates`, image URLs, Google News title and publication date, video title
- 📅 **Date filter**: only URLs modified in the last N days or after a date — and it **skips old child sitemaps entirely**, so big sites stay fast and cheap
- 🎯 **Regex include / exclude** filters (e.g. only `/blog/`, drop `/tag/`)
- 🆕 **Monitor mode**: remembers URLs between runs and outputs only new ones (perfect with Apify Schedules + email/Slack/webhook integrations)
- ✅ **Optional HTTP status check** (HEAD request) to find 404s and redirects listed in sitemaps
- 🛡️ **Robust**: retries with exponential backoff, honors `Retry-After`, per-site error report, one broken site never fails the whole run
- 🤖 **AI-agent friendly**: simple input, flat JSON output, usable via the Apify API and MCP server

### Use cases

- **SEO audits** — get the full list of indexable URLs, find 404s/redirects in sitemaps, check hreflang coverage
- **Competitor monitoring** — get notified when a competitor publishes new pages, products, pricing or landing pages
- **Content and news tracking** — collect fresh articles from news sitemaps every hour or day
- **Crawl planning / RAG** — feed a clean URL list into Website Content Crawler or your own scraper instead of crawling blindly
- **Site migrations** — export all old URLs to build redirect maps
- **E-commerce** — list every product URL of a store, or only products added this week

### Input

Only `startUrls` is required. Example:

```json
{
  "startUrls": ["apify.com", "https://www.bbc.co.uk/sitemaps/https-index-uk-news.xml"],
  "maxUrlsPerSite": 100,
  "includeUrlPatterns": ["/news/"],
  "modifiedWithinDays": 7,
  "onlyNewUrls": false,
  "checkStatus": false
}
```

| Field | Description |
| --- | --- |
| `startUrls` | Domains, homepage URLs, `robots.txt` URLs or direct sitemap URLs. A homepage/path URL is treated as its whole domain — use `includeUrlPatterns` to narrow it down. |
| `maxUrlsPerSite` | Max URLs output per site (default 100). |
| `includeUrlPatterns` / `excludeUrlPatterns` | Case-insensitive regular expressions. |
| `modifiedWithinDays` / `modifiedAfter` | Keep only URLs with a recent `<lastmod>`. |
| `keepUrlsWithoutLastmod` | Keep undated URLs when a date filter is set. |
| `onlyNewUrls` | Monitor mode: output only URLs not seen in earlier runs. The first run saves a baseline. |
| `emitAllOnFirstRun` | Output everything on the first monitor run too. |
| `checkStatus` | Adds `statusCode`, `finalUrl`, `redirected`. |
| Advanced | `maxSitemapsPerSite`, `maxDepth`, `maxConcurrency`, `requestTimeoutSecs`, `maxRetries`, `proxyConfiguration`. |

### Output

One dataset item per URL. Export as JSON, CSV, Excel, XML or via API. Example:

```json
{
  "site": "techcrunch.com",
  "url": "https://techcrunch.com/2026/10/30/nvidia-weighs-playing-the-cosigner-for-nscales-ai-data-center-in-west-virginia/",
  "lastmod": "2026-10-31",
  "changefreq": null,
  "priority": null,
  "sitemapUrl": "https://techcrunch.com/sitemap-page-122.xml",
  "imageCount": 1,
  "images": ["https://techcrunch.com/wp-content/uploads/2026/03/GettyImages-2235971874.jpg"],
  "extractedAt": "2026-11-04T02:57:47Z"
}
```

Optional fields appear when available: `alternates` (`[{"hreflang": "de", "href": "..."}]`), `newsTitle`, `newsPublicationDate`, `videoTitle`, `isNew` (monitor mode), `statusCode` / `finalUrl` / `redirected` (status check).

A per-site report (sitemaps found, URLs found/filtered/output, new URLs, errors) is saved to the key-value store record **`SUMMARY`**.

### How to monitor a website for new pages

1. Add the sites to `startUrls` and enable **Monitor mode**.
2. Run once — this saves the baseline (no results, no charge for URLs).
3. Create a **Schedule** (e.g. daily) in Apify Console for the saved task.
4. Add an integration (email, Slack, Google Sheets, Zapier, Make, webhook) to receive the new URLs.

The list of seen URLs is stored as hashes in a private named key-value store `sitemap-url-monitor-state` in your own Apify account. Delete a record there to reset the baseline for a site.

### Pricing

Pay per result: you pay only for URLs written to the dataset. Sitemap discovery, sites without sitemaps, filtered-out URLs and the monitor-mode baseline run are not charged per URL. Set a maximum cost per run in the run options — the Actor stops cleanly when it is reached.

### FAQ

**The site has no sitemap — what happens?** The site is reported as `no-sitemap-found` in `SUMMARY`; the run continues with the other sites. This Actor does not crawl HTML pages — use a crawler for sites without sitemaps.

**Does it find every page of a website?** It returns every page the site lists in its sitemaps. Most CMSs (WordPress, Shopify, Wix, Squarespace, Webflow, Next.js, Docusaurus …) publish complete sitemaps, but pages left out of the sitemap won't appear.

**How big can a site be?** Millions of URLs are fine. Raise `maxUrlsPerSite` and `maxSitemapsPerSite`. Use the date filter to make large-site runs much cheaper.

**Is it legal?** Sitemaps are public files that site owners publish specifically so that automated tools can discover their pages. The Actor collects only URLs and sitemap metadata — no personal data. You are responsible for how you use the output and for respecting each website's terms.

**Do I need proxies?** Usually not. If a site blocks requests to its sitemap, enable Apify Proxy in the Advanced section.

**Can I use it from code or an AI agent?** Yes — call it via the Apify API, the Python/JS clients, or the Apify MCP server. Input and output are simple JSON.

**Something doesn't work?** Open an issue on the Issues tab with the site URL — fixes are usually quick.

# Actor input Schema

## `startUrls` (type: `array`):

One per line. Enter a domain (example.com), a homepage URL, a robots.txt URL, or a direct sitemap URL (.xml, .xml.gz or .txt). For domains, sitemaps are discovered from robots.txt and common locations such as /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml.

## `maxUrlsPerSite` (type: `integer`):

Maximum number of URLs to output for each website. Keep it small for testing. Sitemap download stops early once the limit is reached (except in monitor mode, which must read the whole sitemap to detect new URLs).

## `includeUrlPatterns` (type: `array`):

Optional. Keep only URLs matching at least one of these regular expressions (case-insensitive), e.g. /blog/ or /products/.

## `excludeUrlPatterns` (type: `array`):

Optional. Drop URLs matching any of these regular expressions, e.g. /tag/ or ?page=.

## `modifiedWithinDays` (type: `integer`):

Optional. Keep only URLs whose sitemap <lastmod> is within the last N days. Child sitemaps with an older <lastmod> are skipped entirely, which makes runs on big sites much faster and cheaper.

## `modifiedAfter` (type: `string`):

Optional. ISO date such as 2026-01-31. Same effect as the option above, using a fixed date.

## `keepUrlsWithoutLastmod` (type: `boolean`):

Many sitemaps omit <lastmod>. When a date filter is set, such URLs are dropped unless you enable this.

## `onlyNewUrls` (type: `boolean`):

Remembers every URL seen per site (in your private named key-value store 'sitemap-url-monitor-state') and outputs only URLs that were not present in previous runs. Schedule the Actor daily or weekly to get alerts about new pages, products or articles. The first run only saves a baseline.

## `emitAllOnFirstRun` (type: `boolean`):

In monitor mode, the first run normally outputs nothing and just saves the baseline. Enable this to output all URLs on that first run (marked isNew=true).

## `checkStatus` (type: `boolean`):

Sends a lightweight HEAD request to every output URL and adds statusCode, finalUrl and redirected. Useful for SEO audits (find 404s and redirects listed in sitemaps). Slower.

## `maxSitemapsPerSite` (type: `integer`):

Safety cap on the number of sitemap files downloaded per website.

## `maxDepth` (type: `integer`):

How many levels of nested sitemap indexes to follow.

## `maxConcurrency` (type: `integer`):

Number of sitemap files downloaded in parallel for each site. Be polite.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each HTTP request.

## `maxRetries` (type: `integer`):

Retries for network errors, HTTP 429 and 5xx responses (exponential backoff, honors Retry-After).

## `proxyConfiguration` (type: `object`):

Optional. Not needed for most sites. Use Apify Proxy only if a site blocks plain requests to its sitemap.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com"
  ],
  "maxUrlsPerSite": 20,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "keepUrlsWithoutLastmod": false,
  "onlyNewUrls": false,
  "emitAllOnFirstRun": false,
  "checkStatus": false,
  "maxSitemapsPerSite": 200,
  "maxDepth": 5,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 30,
  "maxRetries": 3,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com"
    ],
    "maxUrlsPerSite": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("thomson-digital/sitemap-url-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com"],
    "maxUrlsPerSite": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("thomson-digital/sitemap-url-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com"
  ],
  "maxUrlsPerSite": 20
}' |
apify call thomson-digital/sitemap-url-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,thomson-digital/sitemap-url-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/hbGsXgtQca2m0sFuT/builds/9psDKxbEkrOc7VjKY/openapi.json