# Sitemap URL Scraper: List URLs & Check Changes (`pradio/sitemap-url-scraper`) Actor

List every URL a site's sitemaps declare, with lastmod, changefreq and priority, or add a health check of each sitemap: fetch status, URL count, drift since your last run, a dead-link sample and alerts. Sites that never answer come back as free status rows.

- **URL**: https://apify.com/pradio/sitemap-url-scraper.md
- **Developed by:** [Pradio Actors](https://apify.com/pradio) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 sitemap health checks

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Scraper: List URLs & Check Changes

### What does Sitemap URL Scraper do?

Give it a site or a sitemap URL and you get every address the site lists there, up to 1,000 a run by default. It costs you $0.40 per 1,000 URLs listed. By default you get one row per URL the sitemap lists, like this one:

```json
{
  "url": "https://nces.ed.gov/programs/coe/ataglance",
  "status": 200,
  "changefreq": "monthly",
  "parent_sitemap": "https://nces.ed.gov/programs/coe/sitemap.xml"
}
```

Bare domains and index files are resolved for you. A domain's robots.txt is checked for Sitemap lines, and an index is followed into its children. One site is enough to start:

```json
{
  "startUrls": ["https://cmu.edu/sitemap.xml"]
}
```

A longer list works the same way. This input also adds a health row for each sitemap file, from two sites:

```json
{
  "startUrls": ["https://apify.com", "https://www.sitemaps.org/sitemap.xml"],
  "output": "urls_and_audit",
  "maxUrls": 500
}
```

### Features

- **A health check of every sitemap a site declares**: HTTP status, URL count, newest lastmod and a dead-link sample. An alert flags an error answer, a shrink or a stale file.
- **What changed since your last run**: with the health check on, from your second run on you see which URLs were added or removed, and how long the list has gone unchanged.
- **A row for every site you give it**: each declared URL lands with its lastmod, changefreq and the file that declared it. If a site can't be read, you still get a row saying why.
- **Blocked sites named, not skipped**: if a site blocks the read, you get a row saying so, and the run moves on.

### Why this one

The most-used alternative on this platform lists the URLs and stops there. Both were run on cmu.edu's sitemap, which lists 307 addresses:

| | The most-used alternative | This Actor |
|---|---|---|
| URLs listed | 307 in 51 seconds | 307 in 36 seconds, with the health check on |
| Health check of each sitemap | Not offered | Yes, with an alert |
| What changed since your last run | Not offered | Yes |
| Price per 1,000 URLs | $0.50 | $0.40, plus $0.003 a sitemap when the health check is on |

Every row names the input it came from, so you can trace an unreadable file back to what you pasted.

### How much does it cost?

You pay $0.003 for each sitemap audited (the health rows) and $0.0004 for each URL listed. Apify also charges $0.00005 for each Actor start. The prefilled run lists addresses only and costs you about 16 cents. The pricing page carries every price on the ladder.

### How do I use it?

1. Paste one or more starting points: a bare domain, a site URL, a sitemap URL, or the address of a text file listing sitemap URLs.
2. Under **What to get**, keep **URL list only**, or add the health check of each sitemap, or pick the health check alone.
3. Run it, then download your dataset as JSON, CSV or Excel. You can also wire it into your own workflow through the platform's integrations (the API, webhooks, Zapier, Make). A run lists up to 1,000 addresses by default; raise **Cost cap: URLs listed** for more.

### Input

| Setting | Type | What it does |
|---|---|---|
| Start URLs (`startUrls`) | array | Sitemap URLs, site URLs or domains to read. A site root is resolved through its robots.txt, a sitemap index is followed into its children. |
| Sitemap list file URL (`sitemapListUrl`) | string | Address of a text or CSV file listing sitemap URLs or site roots, one per line. |
| What to get (`output`) | string | What to get: `urls_only` (the default), `urls_and_audit` (adds a health row per file), or `audit_only`. |
| Include URL patterns (`urlFilterInclude`) | array | Glob patterns a listed address must match to be listed. |
| Exclude URL patterns (`urlFilterExclude`) | array | Glob patterns a listed address must not match to be listed. |
| Dead-link check (`linkCheck`) | string | Dead-link check: `sample` (the default), `all` listed pages, or `off`. |
| Cost cap: URLs listed (`maxUrls`) | integer | Cap on listed-address rows. Default 1,000. |
| Cost cap: URLs per site (`maxUrlsPerSite`) | integer | The most URL rows one site may return, so one large site cannot use the whole URL cap. |
| Cost cap: sitemap health rows (`maxItems`) | integer | The most sitemaps checked per run when you add the health check. Default 100. |
| Proxy (`proxyConfiguration`) | object | Optional proxy routing for the fetches and probes. Off by default; a refusal is reported on its row. |

These stay off the form. Set them over the API or in the JSON editor:

| Field | Default | What it does |
|---|---|---|
| `deadLinkSample` | 50 | Pages a sample check reads per run, at most 500. |
| `staleAfterDays` | 30 | Days without a change before a stale-sitemap alert. |
| `maxItemsPerStartUrl` | none | Sitemap files one start URL may audit, at most 200. |
| `maxCrawlingDepth` | 2 | Sitemap levels to follow; 1 reads only the files you name. |
| `maxRequestRetries` | 2 | Retries after a network error. A short stated pause (a 429) is waited for once. |
| `proxyRotation` | `PER_REQUEST` | `PER_REQUEST`: a new proxy address per request. `UNTIL_FAILURE`: one address until it fails. |
| `includeUrls`, `urlsOnly` | none | Older switches, kept for saved tasks; **What to get** replaces them. |
| `checkPageStatus`, `checkAllUrls` | none | Older switches, kept for saved tasks; **Dead-link check** replaces them. |

### Output

Each listed-address row describes one address; with the health check on, each health row describes one sitemap file. The fields most buyers use are below. Every field and what it holds is on the Output tab of this page.

**On every row**

| Field | What it holds |
|---|---|
| `url` | The address the row is about: the sitemap file on a health row, the declared page on an address row. |
| `status` | The page's HTTP status, or a word saying why there is none, such as `not_checked` or `blocked`. Every word is on the Output tab. |
| `row_type` | Tells you which rows you pay for: only `ROW` rows are billed. |
| `entry_kind` | `sitemap` on a file row, `url` on a declared-address row. |
| `parent_sitemap` | The sitemap file that declared this address (the index on a child file). |
| `source_url` | The input entry the row came from, exactly as you submitted it. |
| `lastmod` | Newest lastmod a file declares on a health row; the address's own lastmod on a URL row. |
| `changefreq` | The declared `<changefreq>` on a URL row. Empty when absent or not one of the protocol's seven words; the health row counts those. |
| `priority` | The declared `<priority>` on a URL row, as a number from 0 to 1. Empty when absent or malformed; the health row counts the malformed ones. |

**On a health row**

| Field | What it holds |
|---|---|
| `sitemap_kind` | What the file answered with: `urlset`, `sitemapindex`, `none`, or `not_xml`. |
| `declared_url_count` | How many URLs the file lists. |
| `drift_kind` | How the list moved since your last run: `unchanged`, `added`, `removed` and the like, or `first_run`. |
| `added_since_last_run` | URLs new since your last run (first 500). |
| `removed_since_last_run` | URLs gone since your last run (first 500). |
| `sampled_dead_urls` | Checked URLs that came back dead. |
| `alert` | True when the sitemap itself has a problem. |
| `alert_reasons` | Which problems set `alert`. |

A site the run couldn't read gets a free row, and `reason` says why: refused, not found, or a cap reached.

A few fields stay empty until a site has the matching problem or content: `challenge_page_detected`, `soft_404_signal`, `robots_advertised_but_missing`, `sampled_dead_urls`, `not_found_kind` (filled only when a sitemap can't be found), `video_count`, `video_urls`, `video_titles`, `image_titles` (when the sitemap titles its images), and `invalid_priority_examples` and `invalid_changefreq_examples` (when a file carries a malformed value; the counts beside them read 0 otherwise).

### Limits and fair use

| Bound | Value |
|---|---|
| Sitemap files audited per run | `maxItems`, default 100 |
| Listed addresses | `maxUrls`, default 1,000 |
| Sitemap files read, hard ceiling | 200 per run; one entry's files bounded by `maxItemsPerStartUrl` |
| Dead-link probes | A sample: `deadLinkSample`, default 50. Every listed page: up to the URL cap. |
| Sitemap body | Read up to 50 MB |
| One fetch | Gives up after 20 seconds |

By default the dead-link check covers up to 50 pages a run, so most listed URLs stay `not_checked`: 998 of the 1,109 rows in a large test run (90%). Set **Dead-link check** to every listed page to check them all.

### FAQ

**Does it fetch the pages a sitemap lists?**

Only for the dead-link check. By default it checks up to 50 listed pages a run for their HTTP status and reads the first 64 KB of each. Set **Dead-link check** to off and no page is fetched.

**Do I need a proxy?**

You don't. A sitemap is a file the site publishes for crawlers, and a normal read needs none. A host that refuses the read, or answers with an empty or challenge page, gets a `blocked` row saying what it answered.

**Why is lastmod missing on some records?**

A sitemap declares it per address and many files declare none at all. `lastmod` stays empty where the sitemap publishes no date; `lastmod_url_count` on the file's row says how many it carried.

**What if a site blocks the read?**

You get a row naming the block, and the run moves on to the next site, leaving that site alone for the rest of the run. A site that asks the run to wait a minute or less is waited for once.

**What does a status row cost you?**

Nothing. A row for a site it could not read, or for an entry it skipped, is free.

### Watch a sitemap URL over time

With the health check on, each sitemap's URL list is compared with the list from your last run. Your run after the first reports what was added, what was removed and how many days the set has gone unchanged. You get an alert on a problem with the sitemap itself, such as an error or empty answer (a site refusing the read, such as a 403, counts), a shrink of a fifth or more, a stale file (unchanged for 30 days, or dated more than 30 days ago), a fifth of its checked links dead, or dates stamped in bulk. `alert_reasons` names each one. A site that times out or drops the connection gets a row saying so rather than an alert. Schedule it with the health check on and every run tells you what changed.

### Your responsibility

You are responsible for having the right to read the sitemaps you give it and for how you use the rows. A site's robots.txt and terms tell you what is off limits.

### Personal data

The rows carry sitemap metadata: file addresses, listed addresses, timestamps. Read the platform's data protection terms for how a run's data is stored and retained.

### Release notes

- 2026-10-06: very large sitemap indexes finish within the default memory; a dead-link sample is shared across a site's files; a robots.txt that cannot be read is reported as blocked or unreachable, never as "no sitemap"; an old newest date raises the stale alert on the first run. `priority` is now a number, so it sorts as one, and the health row counts malformed priority and changefreq values with five examples. The health-row cap is shared across your start URLs, a list file that is missing is named as missing, and a domain that does not exist says so.
- 2026-10-06: a site that answers "too many requests" and asks for a pause of a minute or less is now waited for once, then asked again, instead of being dropped.
- 2026-10-06: one **What to get** choice replaces the two URL switches, and one **Dead-link check** choice replaces three check fields. A run now returns the URL list by default, every URL up to **Cost cap: URLs listed** (1,000), and adds the health check when you pick it. Saved tasks and API calls that use the old settings keep working.
- 2026-10-06: dead-link checks no longer wait between requests: on cmu.edu's sitemap a run now takes 36 seconds where it took 97. A refusal still stops the checks, and a site that refuses through a proxy is not asked again in that run.

### Not affiliated

This Actor is an unofficial, independent tool. It reads files a site publishes for crawlers and is not affiliated with, sponsored, or endorsed by any site whose sitemap it reads.

### Feedback

Something not working, or a case the rows miss? Open an issue on the Issues tab and it will be looked at. If the Actor saved you time, a review on the Store helps other buyers find it.

# Actor input Schema

## `startUrls` (type: `array`):

Your sitemap URLs or site roots, one per line. The filled-in examples show the shape: replace them with your own. A site root is found through its robots.txt, else /sitemap.xml.

## `sitemapListUrl` (type: `string`):

Optional: the URL of a plain-text or CSV file with one sitemap URL or site root per line. A robots.txt works too: each Sitemap: line becomes an entry.

## `output` (type: `string`):

URL list only: one row per URL the sitemaps list, billed at $0.40 per 1,000 URLs. Add the health check for a row per sitemap file with its status, freshness, challenge-page and soft-404 signals and what changed since your last run, the rows the Sitemap health check event bills. The default writes both: the health rows are what the sitemap signals are read from, so a run that wants them must ask for them.

## `urlFilterInclude` (type: `array`):

Optional: patterns a URL must match to be listed, such as https://shop.example.com/products/\*. A pattern without \* or ? matches anywhere in the URL. Empty lists every URL.

## `urlFilterExclude` (type: `array`):

Optional: patterns a URL must not match to be listed, such as *://*/*search*. Applied after the include patterns.

## `linkCheck` (type: `string`):

Free: a check adds the page's HTTP status to its row and is not billed. A sample checks up to 50 listed pages per run, every listed page checks them all, off skips it.

## `deadLinkSample` (type: `integer`):

How many listed pages one run checks when linkCheck is sample, spread across the sitemap files. 0 checks none. Default 50, at most 500.

## `staleAfterDays` (type: `integer`):

Raise an alert when a sitemap the site's robots.txt still lists has gone this many days without a change, compared with earlier runs. Default 30.

## `maxUrls` (type: `integer`):

The most URL rows one run lists and bills, across all sites. One sitemap can declare 50,000 URLs.

## `maxUrlsPerSite` (type: `integer`):

Optional: the most URL rows one site may return, as a number of rows (for example 200), so one large site cannot use the whole URL cap. Empty means no per-site limit.

## `maxItems` (type: `integer`):

Used only when the "What to get" field, in its own "🎁 What to get" section above, includes the health check: the most sitemap files one run checks and bills, one health row each. It does not limit URL rows.

## `maxItemsPerStartUrl` (type: `integer`):

The most sitemap files one start URL may audit (200 at most). Empty shares the run's files evenly across the start URLs.

## `maxCrawlingDepth` (type: `number`):

How many sitemap levels to read: 1 reads only the files named, 2 also follows a sitemap index into its children. 0 means no limit.

## `maxRequestRetries` (type: `number`):

Retries after a network error such as a dropped connection or a timeout. A 429 that states a pause of 60 seconds or less is waited for once; any other HTTP answer is final.

## `includeUrls` (type: `boolean`):

Kept for saved tasks and API calls made before What to get existed. On: URL rows with the health check; off: health check only. Sent, it wins over What to get.

## `urlsOnly` (type: `boolean`):

Kept for saved tasks and API calls made before What to get existed. On: URL rows only, no health check. Sent, it wins over What to get.

## `checkPageStatus` (type: `boolean`):

Kept for saved tasks and API calls made before Dead-link check existed. Off turns the check off. Sent, it wins over Dead-link check.

## `checkAllUrls` (type: `boolean`):

Kept for saved tasks and API calls made before Dead-link check existed. On checks every listed URL. Sent, it wins over Dead-link check.

## `proxyConfiguration` (type: `object`):

Optional: route the sitemap fetches and URL checks through a proxy. A refused request is reported on the row and never retried through another address.

## `proxyRotation` (type: `string`):

Only used when a proxy is set. PER_REQUEST (the default): a new address for every request. UNTIL_FAILURE: one address until it fails. A site that refuses is not asked again in that run, either way.

## Actor input object example

```json
{
  "startUrls": [
    "https://cmu.edu/sitemap.xml",
    "https://www.sitemaps.org/sitemap.xml",
    "https://docs.python.org/sitemap.xml"
  ],
  "output": "urls_and_audit",
  "linkCheck": "sample",
  "deadLinkSample": 50,
  "staleAfterDays": 30,
  "maxUrls": 1000,
  "maxItems": 100,
  "maxCrawlingDepth": 2,
  "maxRequestRetries": 2,
  "proxyRotation": "PER_REQUEST"
}
```

# Actor output Schema

## `rows` (type: `string`):

Link to this run's rows in the default dataset: one row per sitemap file, plus one row per declared URL when What to get lists URLs, and any uncharged status rows.

## `summary` (type: `string`):

Link to RUN_SUMMARY in the default key-value store: the run's counts, rowsFetched, rowsPushed, rowsCharged, rowsUncharged, duplicatesDropped and stoppedEarly.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://cmu.edu/sitemap.xml",
        "https://www.sitemaps.org/sitemap.xml",
        "https://docs.python.org/sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("pradio/sitemap-url-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [
        "https://cmu.edu/sitemap.xml",
        "https://www.sitemaps.org/sitemap.xml",
        "https://docs.python.org/sitemap.xml",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("pradio/sitemap-url-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://cmu.edu/sitemap.xml",
    "https://www.sitemaps.org/sitemap.xml",
    "https://docs.python.org/sitemap.xml"
  ]
}' |
apify call pradio/sitemap-url-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pradio/sitemap-url-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4wCef1NNxs66IJFcq/builds/M9haNHXSBd83VzngK/openapi.json