# Sitemap Extractor: All URLs from XML Sitemaps & Indexes (`brasspage/sitemap-extractor`) Actor

Sitemap extractor and sitemap URL scraper: finds a website's XML sitemaps through robots.txt, follows sitemap indexes and .xml.gz files, and lists every URL with lastmod, changefreq and priority. Filter by text or modified date. No browser, no proxy. $0.20 per 1,000 URLs.

- **URL**: https://apify.com/brasspage/sitemap-extractor.md
- **Developed by:** [Brasspage](https://apify.com/brasspage) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Sitemap Extractor: All URLs from XML Sitemaps & Indexes

Give it a website or a sitemap URL. The Actor finds the site's sitemaps, follows sitemap indexes, and lists every URL with its `lastmod`, `changefreq` and `priority`.

- **Finds sitemaps for you:** reads the `Sitemap:` lines in robots.txt, then tries `/sitemap.xml` and `/sitemap_index.xml`.
- **Follows sitemap indexes:** nested indexes, `.xml.gz` files, namespaced XML, and plain-text sitemaps (one URL per line).
- **Filters:** keep URLs that contain `/blog/`, drop URLs that contain `/tag/`, or keep only pages modified since a date.
- **Clean output:** each URL once, even when two sitemaps list it.

**$0.20 per 1,000 URLs.** No browser, no proxy, no API key. It reads only robots.txt and sitemap files, which sites publish for crawlers. It never opens the pages themselves. The Apify free plan gets up to 10,000 URLs per run.

### How to use

1. Open the Actor in the Apify Console and go to the **Input** tab.
2. Type a website, like `https://crawlee.dev`, into **Websites or sitemap URLs**. One per line, up to 100.
3. Click **Start**.
4. When the run ends, open the **Output** tab. Download the URLs as CSV, Excel or JSON, or read them through the API.

The prefilled input reads the first 100 URLs from crawlee.dev. Clear **Max URLs** to get them all.

### What can you do with it?

- **Feed a crawler:** get the full URL list of a site first, then pass it to a scraper as start URLs.
- **SEO audits:** count pages per section, find stale pages by `lastmod`, and check what a site tells search engines.
- **Watch a site for new pages:** schedule a daily run with **Modified since** set, or compare today's list with yesterday's.
- **Content and competitor research:** list every blog post or product page on a site in seconds.
- **AI agents and RAG:** call the Actor as a tool (Apify API or Apify MCP server) to find the pages to read.

### Input example

```json
{
    "startUrls": ["https://crawlee.dev", "https://example.com/sitemap.xml"],
    "includeUrlsContaining": ["/blog/"],
    "excludeUrlsContaining": ["/tags/"],
    "modifiedSince": "2026-01-01",
    "sameSiteOnly": true,
    "maxUrls": 5000
}
```

### Output example

One row per URL:

```json
{
    "url": "https://crawlee.dev/blog",
    "lastmod": null,
    "changefreq": "weekly",
    "priority": 0.5,
    "sitemap": "https://crawlee.dev/sitemap.xml",
    "startUrl": "https://crawlee.dev/"
}
```

`lastmod`, `changefreq` and `priority` are `null` when the sitemap does not give them. The `OUTPUT` record holds a summary: sites read, sitemaps read, URLs found and written, duplicates skipped, and the errors for each site (for example a 404 sitemap or a robots.txt rule).

### Input fields

| Field                   | What it does                                                                                     |
| ----------------------- | ------------------------------------------------------------------------------------------------ |
| `startUrls`             | Websites or sitemap URLs. A URL whose path has "sitemap" or ends in `.xml` is read as a sitemap. |
| `includeUrlsContaining` | Keep a URL only if it contains one of these texts. Plain text, not a regex.                      |
| `excludeUrlsContaining` | Drop a URL if it contains one of these texts.                                                    |
| `modifiedSince`         | Keep only URLs with a `lastmod` on or after this date. URLs with no `lastmod` are dropped.       |
| `sameSiteOnly`          | Drop URLs on other domains (default on). Subdomains count as the same site.                      |
| `maxUrls`               | Stop after this many URLs in total.                                                              |
| `maxSitemapsPerSite`    | Stop reading a site after this many sitemap files (default 500).                                 |

### Pricing

Pay per result: **$0.20 per 1,000 URLs** written to the dataset, plus a tiny start fee ($0.00005). Duplicates and filtered-out URLs are free. The run stops at your maximum total charge, so a big site never costs more than you set.

The prefilled input writes at most 100 URLs, so a one-click run costs about $0.02, plus the start fee.

### Polite and safe

- One request at a time per site, at most one a second, and slower if robots.txt asks for a `Crawl-delay` (up to 30 seconds; a site asking for more is skipped).
- Follows robots.txt `Disallow` rules for sitemap files.
- Sitemaps on other sites, and private or local network addresses, are never read.
- Sitemap files are capped at 52 MB (the sitemap limit is 50 MB), also when gzipped.

### FAQ

**Do I need to know the sitemap URL?** No. Give it the website and it looks in robots.txt, then tries `/sitemap.xml` and `/sitemap_index.xml`. You can also give a sitemap URL directly.

**Does it open the pages in the sitemap?** No. It reads only robots.txt and sitemap files, so it is fast and light on the site.

**Am I charged for duplicates?** No. Each URL is written and charged once per run, even when two sitemaps list it. URLs your filters drop are free too.

**What if a site has no sitemap?** The run writes no URLs for it, and the `OUTPUT` summary lists the error for that site, like a 404 sitemap or a robots.txt rule.

### Limits

- Up to 100 websites or sitemap URLs per run.
- Up to 500 sitemap files per site by default (**Max sitemap files per site**, up to 5,000).
- The Apify free plan gets up to 10,000 URLs per run.
- URLs longer than 2,048 characters are skipped.
- It reads sitemaps only. A page that is not in any sitemap is not found. To find those, use a crawler.
- Image, video and news sitemap extras (like `image:loc`) are not in the output yet.
- RSS and Atom feeds used as sitemaps are not read.

### Related Actors

- [Bulk Lighthouse & PageSpeed Audit](https://apify.com/brasspage/lighthouse-bulk-audit): audit the URLs you found for Core Web Vitals.
- [Remove Duplicates & Merge Datasets](https://apify.com/brasspage/dataset-dedup-merge): with "only new items", get just the pages added since the last run.

### Issues and support

Found a bug or need a feature? Open an issue in this Actor's **Issues** tab. We check it every day and reply there.

# Changelog

This Actor's version history is a separate document: https://apify.com/brasspage/sitemap-extractor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

One per line, up to 100. A website (https://example.com) is searched for sitemaps in robots.txt, then /sitemap.xml and /sitemap_index.xml. A sitemap URL (https://example.com/sitemap.xml or .xml.gz) is read as is. Sitemap indexes are followed.

## `includeUrlsContaining` (type: `array`):

Optional. Keep a URL only if it contains one of these texts, for example /blog/ or /products/. Plain text, not a regex.

## `excludeUrlsContaining` (type: `array`):

Optional. Drop a URL if it contains one of these texts, for example /tag/ or ?page=.

## `modifiedSince` (type: `string`):

Optional. Keep only URLs whose lastmod is on or after this date (YYYY-MM-DD). URLs with no lastmod are dropped when this is set.

## `sameSiteOnly` (type: `boolean`):

Drop URLs on other domains. Subdomains of the site count as the same site. Sitemaps on other sites are never read.

## `maxUrls` (type: `integer`):

Stop after this many URLs in total. Leave empty for no limit (the Apify free plan gets up to 10,000 URLs per run).

## `maxSitemapsPerSite` (type: `integer`):

Stop reading a site after this many sitemap files (indexes count too). Each file is fetched at most once a second, so 500 files take about 8 minutes.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "includeUrlsContaining": [
    "/blog/"
  ],
  "excludeUrlsContaining": [
    "/tag/",
    "?page="
  ],
  "modifiedSince": "2026-01-01",
  "sameSiteOnly": true,
  "maxUrls": 100,
  "maxSitemapsPerSite": 500
}
```

# Actor output Schema

## `results` (type: `string`):

One row per URL: url, lastmod, changefreq, priority, the sitemap it came from, and the start URL.

## `summary` (type: `string`):

Sites read, sitemaps read, URLs found and written, duplicates skipped, and errors per site.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ],
    "maxUrls": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("brasspage/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev"],
    "maxUrls": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("brasspage/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrls": 100
}' |
apify call brasspage/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,brasspage/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/MI8daXJWNQ6wfYVN1/builds/zwbVAXJzGffED5Rgi/openapi.json