# Sitemap Extractor: Get All URLs from XML Sitemaps (`phaux/sitemap-url-extractor`) Actor

List every page URL a website publishes in its XML sitemaps (auto-found via robots.txt; indexes and .gz too). Use for a site's full URL list or recently changed pages. Input: startUrls (site or sitemap), optional regex, lastmodSince. Output per URL: url, lastmod, changefreq, priority, sitemap.

- **URL**: https://apify.com/phaux/sitemap-url-extractor.md
- **Developed by:** [Phaux](https://apify.com/phaux) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor: Get All URLs from XML Sitemaps

A **sitemap extractor** for any website: get the full list of **sitemap URLs** as a clean dataset for SEO audits, site migrations, change monitoring or as the start list for a crawl.

Give it a website or a sitemap URL and get back **every page URL the site publishes in its XML sitemaps**, with `lastmod`, `changefreq`, `priority`, hreflang alternates and image counts. Sitemaps are discovered automatically from robots.txt, sitemap indexes are followed, and `.xml.gz`, plain-text, RSS and Atom sitemaps are read.

### Quick start for AI agents and API users

- **What it does:** List every page URL a website publishes in its XML sitemaps (auto-found via robots.txt; indexes and .gz too).
- **When to use it:** For a site's full URL list or recently changed pages.
- **Example input** (the form's prefill; it finishes in under a minute):

```json
{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrlsPerSite": 1000
}
```

- **Example output:** one real dataset row from run `gTFrGDOf1X4rtFem0` (2026-10-23), with long fields shortened and empty fields left out. Fields not shown are listed in the output table below.

```json
{
  "url": "https://crawlee.dev/blog",
  "changefreq": "weekly",
  "priority": 0.5,
  "imageCount": 0,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "sitemapType": "urlset",
  "site": "https://crawlee.dev",
  "scrapedAt": "2026-10-23T13:10:53Z"
}
```

- **Cost (pay per event, Free-plan prices):** result $0.0005. Failed items are not charged.
- **From an AI agent:** with the Apify MCP server, call `call-actor` with actor `phaux/sitemap-url-extractor`, or load it as its own tool with `https://mcp.apify.com?tools=phaux/sitemap-url-extractor`. Read results with `get-dataset-items`.

### What does Sitemap URL Extractor do?

- **Auto-discovery:** reads the `Sitemap:` lines in robots.txt, then falls back to `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml` and `/sitemap.xml.gz`.
- **Follows sitemap indexes** recursively (`maxDepth`, up to 10 levels).
- **Reads every common format:** `.xml`, `.xml.gz`, plain-text sitemaps, RSS 2.0 and Atom feeds, streamed up to the 50 MB protocol limit per file.
- **Filters:** include/exclude regex, "modified since", same-host only, and URL caps per site and in total.
- **De-duplicates** across all sitemaps and sites in the run, and returns **one dataset item per unique URL**.

### Why use it?

- **Fast:** only sitemap files are downloaded, never the pages themselves. 8,000 URLs typically take a few seconds.
- **Safe defaults:** 1,000 URLs per site unless you raise the cap, and a time budget that finishes the run cleanly before its timeout, keeping everything saved so far.
- **Pay only per URL saved:** sitemap files, duplicates, filtered URLs and failed sitemaps are free.

Use cases: SEO audits and content inventories, finding new or updated pages (`lastmodSince`), seeding crawlers with URL lists, site migrations, and monitoring competitors' published pages.

### How to use it

1. Add websites (`https://example.com`) or direct sitemap URLs to `startUrls`. With an empty input, the Actor runs a small example (crawlee.dev).
2. Optionally set `includeUrlPatterns` (for example `"/blog/"`), `lastmodSince` (for example `"30 days"`), and caps.
3. Set `maxUrlsPerSite` to 0 to get every URL of a large site.
4. Run the Actor and open the **URLs** dataset view, or download JSON, CSV or Excel.

### Input

```json
{
  "startUrls": ["https://crawlee.dev", "https://www.bbc.com/sitemaps/https-index-com-news.xml"],
  "includeUrlPatterns": ["/blog/"],
  "excludeUrlPatterns": ["/tag/"],
  "lastmodSince": "30 days",
  "sameHostOnly": false,
  "maxUrlsPerSite": 1000,
  "maxItems": 0,
  "timeBudgetSecs": 0
}
```

| Field | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array | `["https://crawlee.dev"]` | A website (https://example.com: sitemaps are discovered from its robots.txt, then /sitemap.xml) or a direct sitemap URL (.xml, .xml.gz, .txt, RSS/Atom). Leave empty to run the small example (crawlee.dev). |
| `includeUrlPatterns` | array | — | Optional. Keep only URLs matching at least one regular expression, e.g. '/blog/' or '\\.html$'. Case-insensitive. |
| `excludeUrlPatterns` | array | — | Optional. Drop URLs matching any of these regular expressions, e.g. '/tag/' or '\\?'. |
| `lastmodSince` | string | — | Optional. Keep only URLs whose <lastmod> is on/after this date (2026-10-01) or within a period ('7 days'). |
| `keepUrlsWithoutLastmod` | boolean | `true` | Only used with 'Modified since'. |
| `sameHostOnly` | boolean | `false` | Drop URLs on other hosts than the start URL (www. is ignored). |
| `includeAlternates` | boolean | `true` | Add the xhtml:link hreflang alternates listed for each URL. |
| `maxUrlsPerSite` | integer | `1000` | Default 1,000 keeps runs quick and cheap. Set 0 for no limit (large sites can have hundreds of thousands of URLs). |
| `maxItems` | integer | `0` | 0 = no limit. You are only charged for URLs saved. |
| `maxSitemapsPerSite` | integer | `500` | Safety cap on sitemap files read per site. |
| `maxDepth` | integer | `5` | How deep to follow nested sitemap indexes. |
| `maxConcurrency` | integer | `3` | Sites processed at once. Requests to one host are always sequential, at least 1 s apart (or robots.txt Crawl-delay). |
| `timeBudgetSecs` | integer | `0` | Stop reading new sitemaps after this many seconds and finish cleanly, keeping every URL saved so far. 0 = automatic: finish ~90 s before the run timeout so a run never times out. |

### Output example

One dataset item per unique URL:

```json
{
  "url": "https://www.bbc.com/news/articles/crwyd02yz22vo",
  "lastmod": "2026-10-19T07:46:54Z",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "alternates": [],
  "sitemapUrl": "https://www.bbc.com/sitemaps/https-sitemap-com-news-1.xml",
  "sitemapType": "urlset",
  "site": "https://www.bbc.com",
  "scrapedAt": "2026-10-20T13:22:30Z"
}
```

### Output fields

| Field | Type | Description |
|---|---|---|
| `url` | string | Page URL listed in the sitemap. |
| `lastmod` | string or null | <lastmod> (ISO 8601, UTC when parseable). |
| `changefreq` | string or null | <changefreq>. |
| `priority` | number or null | <priority>. |
| `imageCount` | integer | Number of image:image entries for the URL. |
| `alternates` | array | hreflang alternates: \[{hreflang, href}]. |
| `sitemapUrl` | string | Sitemap file the URL came from. |
| `sitemapType` | string | urlset, text, rss or atom. |
| `site` | string | Origin of the start URL. |
| `scrapedAt` | string | Run time (ISO 8601, UTC). |

The run also saves `SITEMAP_ERRORS` (sitemaps or sites that couldn't be read, with a code: `robots_disallowed`, `http_error`, `rate_limited`, `not_a_sitemap`, `too_large`, `no_sitemap_found`, `network_error`, `blocked_host`, `invalid_url`, `max_depth`, `time_budget`) and `RUN_STATS` (per-site counts, how each site's sitemap was discovered, and `stoppedByTimeBudget`) in the key-value store.

### How much does it cost?

Pay per event: you pay per URL saved, with no start fee. Apify platform usage is included.

| Apify plan | Price per URL | Per 1,000 URLs |
|---|---|---|
| Free | $0.0005 | $0.50 |
| Bronze | $0.0003 | $0.30 |
| Silver | $0.0002 | $0.20 |
| Gold, Platinum, Diamond | $0.0001 | $0.10 |

Example: the default 1,000 URLs per site cost $0.50 on the Free plan. Use `maxItems` to cap the cost of any run.

### Use with AI agents, MCP and the API

- **MCP:** add `https://mcp.apify.com?tools=phaux/sitemap-url-extractor` to any MCP client and ask, for example, "list the blog URLs in crawlee.dev's sitemap".
- **API:** `POST https://api.apify.com/v2/acts/phaux~sitemap-url-extractor/run-sync-get-dataset-items` with `{"startUrls": ["https://example.com"]}` returns the URLs directly.
- Input and output fields are documented in the input and dataset schemas.

### Data source and compliance

- The Actor reads only **sitemaps the website publishes for crawlers**, and the site's `/robots.txt`. It never fetches the pages themselves, never logs in and needs no API key.
- **Each sitemap URL is checked against that host's robots.txt** (RFC 9309) for the user agent `PhauxSitemapBot`. A disallowed sitemap is skipped and reported, and `Crawl-delay` is honoured.
- It identifies itself honestly (`Mozilla/5.0 (compatible; PhauxSitemapBot/0.1; +https://apify.com/phaux/sitemap-url-extractor)`), uses **no proxies and no IP rotation**, and doesn't try to get around bot protection. After an HTTP 429 it stops contacting that host for the rest of the run.
- Polite pacing: requests to one host are sequential and at least 1 second apart. Only public `http(s)` hosts are fetched, and every redirect hop is checked.
- No personal data: sitemaps list page URLs only.

### Limits

- Only URLs that the site lists in its sitemaps are returned. Pages missing from the sitemaps aren't discovered (this Actor doesn't crawl links).
- 50 MB per sitemap file, `maxSitemapsPerSite` files per site (default 500, up to 5,000), and `maxDepth` index levels (default 5, up to 10).
- Very large sites can list hundreds of thousands of URLs; raise `maxUrlsPerSite` and the run timeout accordingly.

### FAQ

- **"no_sitemap_found"?** The site doesn't advertise a sitemap in robots.txt and has none at the usual paths. Pass the sitemap URL directly if you know it.
- **Only part of a site?** Use `includeUrlPatterns`, for example `"/products/"`.
- **Run stopped early?** `RUN_STATS.stoppedByTimeBudget` is true when the time budget was reached; everything found until then is kept. Raise the run timeout or `timeBudgetSecs`.
- **Something isn't working?** Open an issue on the Issues tab with the run ID.

# Actor input Schema

## `startUrls` (type: `array`):

A website (https://example.com: sitemaps are discovered from its robots.txt, then /sitemap.xml) or a direct sitemap URL (.xml, .xml.gz, .txt, RSS/Atom). Leave empty to run the small example (crawlee.dev). Default: \["https://crawlee.dev"].

## `includeUrlPatterns` (type: `array`):

Optional. Keep only URLs matching at least one regular expression, e.g. '/blog/' or '\\.html$'. Case-insensitive.

## `excludeUrlPatterns` (type: `array`):

Optional. Drop URLs matching any of these regular expressions, e.g. '/tag/' or '\\?'.

## `lastmodSince` (type: `string`):

Optional. Keep only URLs whose <lastmod> is on/after this date (2026-10-01) or within a period ('7 days'). Example: "2026-10-01".

## `keepUrlsWithoutLastmod` (type: `boolean`):

Only used with 'Modified since'. Default: true.

## `sameHostOnly` (type: `boolean`):

Drop URLs on other hosts than the start URL (www. is ignored). Default: false.

## `includeAlternates` (type: `boolean`):

Add the xhtml:link hreflang alternates listed for each URL. Default: true.

## `maxUrlsPerSite` (type: `integer`):

Default 1,000 keeps runs quick and cheap. Set 0 for no limit (large sites can have hundreds of thousands of URLs).

## `maxItems` (type: `integer`):

0 = no limit. You are only charged for URLs saved. Default: 0.

## `maxSitemapsPerSite` (type: `integer`):

Safety cap on sitemap files read per site. Default: 500.

## `maxDepth` (type: `integer`):

How deep to follow nested sitemap indexes. Default: 5.

## `maxConcurrency` (type: `integer`):

Sites processed at once. Requests to one host are always sequential, at least 1 s apart (or robots.txt Crawl-delay). Default: 3.

## `timeBudgetSecs` (type: `integer`):

Stop reading new sitemaps after this many seconds and finish cleanly, keeping every URL saved so far. 0 = automatic: finish ~90 s before the run timeout so a run never times out. Default: 0.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "keepUrlsWithoutLastmod": true,
  "sameHostOnly": false,
  "includeAlternates": true,
  "maxUrlsPerSite": 1000,
  "maxItems": 0,
  "maxSitemapsPerSite": 500,
  "maxDepth": 5,
  "maxConcurrency": 3,
  "timeBudgetSecs": 0
}
```

# Actor output Schema

## `urls` (type: `string`):

All URLs saved by the run (default dataset, overview view).

## `sitemapErrors` (type: `string`):

Sitemaps or sites that could not be read, with the reason.

## `runStats` (type: `string`):

RUN_STATS record.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ],
    "maxUrlsPerSite": 1000
};

// Run the Actor and wait for it to finish
const run = await client.actor("phaux/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev"],
    "maxUrlsPerSite": 1000,
}

# Run the Actor and wait for it to finish
run = client.actor("phaux/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrlsPerSite": 1000
}' |
apify call phaux/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,phaux/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Cuj5xoQOGEdmE7ZbG/builds/W5QLirZJJv57LkVME/openapi.json