# Sitemap Scraper - Find the Sitemap and Watch for New URLs (`neverempty/sitemap-finder-monitor`) Actor

Give it a domain. It reads robots.txt to find where that site really keeps its sitemap, follows nested indexes, unpacks gzip, and returns one row per URL with lastmod, change frequency and priority. Monitoring mode returns only the URLs that are new since the last run.

- **URL**: https://apify.com/neverempty/sitemap-finder-monitor.md
- **Developed by:** [NeverEmpty](https://apify.com/neverempty) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.73 / 1,000 url returneds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Scraper — find the sitemap, follow it, and watch for new URLs

Give it a domain. `stripe.com` is enough.

It reads `robots.txt` to find where that site actually keeps its sitemap, follows nested indexes, unpacks gzip,
and returns **one row per URL** with the last-modified date, change frequency and priority the sitemap states.

### Why finding it matters

Asking for `/sitemap.xml` fails on a lot of real sites. Measured on 2026-09-07 against ten live sites:

| Site | `/sitemap.xml` | Where the sitemap actually is |
|---|---|---|
| stripe.com | **404** | `robots.txt` declares `/sitemap/sitemap.xml` |
| www.nytimes.com | **403** | `robots.txt` declares 25 separate sitemaps |
| www.shopify.com | 200 | `robots.txt` declares `sitemaps_list.xml` (a different name) |
| www.bbc.com | 200 | `robots.txt` declares **39** sitemaps |
| github.com | **406** | declares none — this Actor reports that instead of guessing |
| apify.com | 200 | an index pointing at 12 more files |

So the order is: **read `robots.txt` first, take every `Sitemap:` line, and only guess if there are none.**
Nine common locations are tried as a fallback, and a site with no sitemap is reported as `no-sitemap` with the
reason — not returned as an empty result.

### Monitoring mode: only what is new

Turn on **Monitoring mode** and the Actor remembers every URL it returned.

- The **first run** returns everything and remembers it.
- Every run after that returns **only URLs that were not there before**.
- A quiet day returns one row saying nothing is new, and **nothing is charged**.

That makes a daily watch on a competitor's site, a documentation site or a job board cheap: you pay for new
pages, not for the whole sitemap every day.

### What comes back

| Field | What it is |
|---|---|
| `site`, `url` | The site you asked for, and the URL found in its sitemap |
| `host`, `path`, `depth` | Split out so you can group by section. `depth` is how many path segments deep |
| `extension` | `pdf`, `html`, … or `null` when the URL has none |
| `lastmod`, `lastmodAt` | Exactly as the sitemap wrote it, and the same value as an ISO instant |
| `changefreq`, `priority` | What the sitemap claims. `null` when it says nothing |
| `sitemapUrl`, `sitemapDepth` | Which file this URL came out of, and how many indexes deep it was |
| `foundVia` | `robots.txt`, `common-path` or `given` — how the sitemap was located |
| `sitemapIsGzipped` | Whether that file was served compressed |
| `status`, `reason` | `ok`, or why a site produced nothing |

A field the sitemap does not state comes back as `null`. Nothing is filled in with a zero or a guessed date.

### Filters

All of these run on URLs already fetched — nothing extra is downloaded to apply them.

- **Path contains / does not contain** — keep `/blog/`, drop `/tag/`
- **File extensions** — only `pdf`, only `html`
- **Changed on or after / before** — by the sitemap's own `lastmod`. ⚠️ A URL with **no** date is **kept**, because
  unknown is not the same as old
- **Minimum priority**, **Change frequency**, **Keywords / Exclude keywords**

### Limits you control

`Maximum URLs`, `Maximum sitemap files` and `Maximum index depth`. Large sites split their sitemap into hundreds
of files, so these keep a run bounded and predictable.

### Pricing

Charged per URL returned. Sites that produced nothing — no sitemap, a sitemap that answered 403, a run where
nothing is new in monitoring mode, or filters that matched nothing — come back with the reason and are
**not charged**.

### Notes

- gzip is detected by the file's own first two bytes, not by the `.gz` in the name, because some servers compress
  without saying so in the URL.
- The same sitemap file is never read twice in one run, and the same URL is never returned twice.
- Only `http` and `https` are accepted. Anything else is rejected rather than rewritten into a different host.

# Actor input Schema

## `sites` (type: `array`):

Domains or URLs. A bare domain such as stripe.com is enough: the Actor reads robots.txt to find where that site actually keeps its sitemap, and only falls back to guessing common locations if robots.txt declares none.

## `sitesText` (type: `string`):

The same list as a block you can paste. Both fields are merged and duplicates are dropped.

## `sitemapUrls` (type: `array`):

Give sitemap files directly when you already know them. These are read in addition to whatever the sites above turn up.

## `useCommonPaths` (type: `boolean`):

Tries /sitemap.xml, /sitemap\_index.xml, /sitemap/sitemap.xml and six more. Turn this off to report only what a site officially declares.

## `maxUrls` (type: `integer`):

Stop after this many URLs across all sites in this run.

## `maxUrlsPerSite` (type: `integer`):

Cap each site separately so one large site cannot use up the whole run. 0 = no per-site cap.

## `maxSitemaps` (type: `integer`):

Large sites split their sitemap into many files. This caps how many files are opened.

## `maxSitemapDepth` (type: `integer`):

A sitemap index can point at other indexes. This is how many levels down to follow.

## `timeoutSecs` (type: `integer`):

Give up on a single sitemap file after this long. The other files are still returned.

## `pathIncludes` (type: `array`):

Keep only URLs whose address contains one of these, for example /blog/ or /docs/. Empty keeps everything.

## `pathExcludes` (type: `array`):

Drop URLs whose address contains any of these, for example /tag/ or ?page=.

## `extensions` (type: `array`):

Keep only URLs ending in these, for example pdf or html. A URL with no extension is dropped when this is set.

## `lastmodAfter` (type: `string`):

Keep URLs the sitemap says were changed on or after this date. A URL with no date is kept, because unknown is not the same as old.

## `lastmodBefore` (type: `string`):

Keep URLs the sitemap says were changed on or before this date. A URL with no date is kept.

## `minPriority` (type: `integer`):

Sitemaps may carry a priority from 0 to 1. This keeps only URLs at or above it. A URL with no priority is kept.

## `changefreqIn` (type: `array`):

Keep only URLs the sitemap marks with these frequencies, such as daily or weekly.

## `keywords` (type: `array`):

Keep only URLs containing these words. Matched against the whole address.

## `keywordMatch` (type: `string`):

any = contains at least one. all = contains every one.

## `excludeKeywords` (type: `array`):

Drop URLs containing any of these words.

## `monitoringMode` (type: `boolean`):

Reads the same sites again and returns only the URLs that were not there last time. The first run returns everything and remembers it. Nothing new means nothing returned and nothing charged, so watching a competitor for new pages costs nothing on quiet days.

## `resetMonitoringState` (type: `boolean`):

Clears the memory so the next monitoring run treats every URL as new.

## Actor input object example

```json
{
  "sites": [
    "stripe.com",
    "vercel.com"
  ],
  "sitesText": "",
  "sitemapUrls": [],
  "useCommonPaths": true,
  "maxUrls": 1000,
  "maxUrlsPerSite": 0,
  "maxSitemaps": 50,
  "maxSitemapDepth": 3,
  "timeoutSecs": 20,
  "pathIncludes": [],
  "pathExcludes": [],
  "extensions": [],
  "lastmodAfter": "",
  "lastmodBefore": "",
  "minPriority": 0,
  "changefreqIn": [],
  "keywords": [],
  "keywordMatch": "any",
  "excludeKeywords": [],
  "monitoringMode": false,
  "resetMonitoringState": false
}
```

# Actor output Schema

## `results` (type: `string`):

Every URL this run read out of the sitemaps, with lastmod and where it was found.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "stripe.com",
        "vercel.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("neverempty/sitemap-finder-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sites": [
        "stripe.com",
        "vercel.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("neverempty/sitemap-finder-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "stripe.com",
    "vercel.com"
  ]
}' |
apify call neverempty/sitemap-finder-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,neverempty/sitemap-finder-monitor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0yvqsalvJbv2W8rIs/builds/cKiNyQgK17XFZDh51/openapi.json