# Sitemap URL Extractor: sitemap.xml, index and gzip (`obligate_castle/robust-sitemap-extractor`) Actor

Extract every page URL from the sitemap.xml of any website, in bulk. Give the Actor a domain and it finds the sitemaps through robots.txt, follows sitemap indexes recursively, reads .xml.gz files, removes duplicates and returns one clean row per URL with lastmod, changefreq, priority and the...

- **URL**: https://apify.com/obligate\_castle/robust-sitemap-extractor.md
- **Developed by:** [Yoann TUQUET](https://apify.com/obligate_castle) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor – sitemap.xml, sitemap index & gzip via robots.txt

Extract every page URL from the **sitemap.xml** of any website, in bulk. Give the Actor a domain and it finds the sitemaps through **robots.txt**, follows **sitemap indexes** recursively, reads **.xml.gz** files, removes duplicates and returns one clean row per URL with `lastmod`, `changefreq`, `priority` and the sitemap it came from.

The data comes straight from the public sitemap protocol (robots.txt and sitemap files that sites publish for crawlers). No HTML scraping, no login, no captcha, no personal data.

### Why use it

- ✅ **Finds the sitemap for you**: reads the `Sitemap:` lines of robots.txt, then tries `/sitemap.xml`, `/sitemap_index.xml` and `/sitemap.xml.gz`. You can also paste a direct sitemap URL.
- ✅ **Handles the awkward cases**: nested sitemap indexes, gzip files (even when served without the right headers), CDATA and XML entities, plain-text sitemaps, and a byte limit against oversized files.
- ✅ **Clean output**: URLs are de-duplicated across all sitemaps and all sites in the run, and every row tells you which sitemap file listed it.
- ✅ **Filters built in**: keep or drop URLs by regular expression, or keep only pages modified since a date, so you do not pay for URLs you would throw away.
- ✅ **Polite by design**: respects robots.txt and sends at most 2 requests per second to any domain.
- ✅ **Pay only for results**: one charge per unique URL returned, nothing for sites without a sitemap or invalid inputs.

### Use cases

1. **SEO audits and migrations**: list every indexable URL of a site before a migration, then compare it with what you crawl or what is indexed in Google Search Console.
2. **Change monitoring**: schedule a weekly run with **Modified since** to catch new and updated pages on competitor, supplier or documentation sites.
3. **Crawl seeding**: feed a clean, de-duplicated URL list to your crawler (Crawlee, Scrapy, Playwright) instead of following links blindly.
4. **Content and AI pipelines**: build the list of documentation or blog pages to ingest into a search index or a RAG knowledge base, filtered by path.
5. **Market research**: count and compare the size and structure of catalogues (`/product/`, `/category/`) across many websites.

### How to use it

1. Click **Try for free** and paste domains, site URLs or direct sitemap URLs into **Websites or sitemap URLs**.
2. Optionally set **Max URLs per site**, a regex filter or **Modified since**.
3. Click **Start**. Export the results as **JSON, CSV, Excel or HTML**, or fetch them through the Apify API.

Tip: create a **Schedule** and use **Modified since** to receive only the recent changes each time.

### What is extracted

| Field | Details |
|---|---|
| `url` | The page URL as declared in the sitemap |
| `lastmod` | Last modification date declared by the site, or `null` |
| `changefreq`, `priority` | Crawl hints declared by the site, or `null` |
| `sitemap` | The sitemap file that listed the URL |
| `site` | Hostname of the website |
| `foundVia` | `robots.txt`, `well-known-path` or `direct` |
| `extractedAt` | Extraction timestamp |

A `SUMMARY` record in the run's key-value store lists, for each input, the number of sitemap files read, the number of URLs found and a note when something stopped the extraction (no sitemap, blocked by robots.txt, limit reached).

### Input

```json
{
  "items": ["www.sitemaps.org", "https://docs.apify.com", "https://example.com/sitemap_index.xml"],
  "maxUrlsPerSite": 1000,
  "excludePattern": "/blog/",
  "modifiedSince": "2026-01-01"
}
```

| Field | Description | Default |
|---|---|---|
| `items` | Domains, site URLs or direct sitemap URLs | required |
| `maxUrlsPerSite` | Stop a site after this many unique URLs (caps the cost per site) | `10000` |
| `includePattern` | Regex: keep only matching URLs | none |
| `excludePattern` | Regex: drop matching URLs | none |
| `modifiedSince` | Keep URLs with `lastmod` on or after this date; URLs without `lastmod` are kept | none |
| `maxSitemapsPerSite` | Safety limit on sitemap files fetched per site | `500` |
| `timeoutSecs` | Network timeout per download | `20` |

### Output

One dataset item per unique URL (real example from a test run):

```json
{
  "url": "https://www.sitemaps.org/",
  "lastmod": "2016-11-21",
  "changefreq": null,
  "priority": null,
  "sitemap": "https://www.sitemaps.org/sitemap.xml",
  "site": "www.sitemaps.org",
  "foundVia": "robots.txt",
  "extractedAt": "2026-09-18T16:34:56.345Z"
}
```

### Pricing

Pay per event: **$0.001 per URL**, which is **$1 per 1,000 URLs**. Sites without a sitemap, invalid inputs and duplicate URLs are not charged. You can cap your spending with the maximum charge setting of each run, and with **Max URLs per site**.

### Integrations

- **API**: start runs and download results from any language with the Apify API or clients.
- **Schedules and webhooks**: run periodically and trigger a webhook when a run finishes.
- **Make, Zapier, n8n**: send new URLs to Google Sheets, Slack or your ticketing tool.
- **AI agents**: callable as a tool through the Apify MCP server.

### FAQ

**Does it respect robots.txt?**
Yes. Sitemaps declared in robots.txt are read as the site intends. Guessed paths (such as `/sitemap.xml`) and sitemap URLs you provide are only fetched if robots.txt allows them, and if robots.txt cannot be fetched, guessing is skipped. Requests are limited to 2 per second per domain.

**What if a site has no sitemap?**
The input is not charged and the `SUMMARY` record explains why (no sitemap found, blocked, unreachable). This Actor reads sitemaps only; it does not crawl pages to discover links.

**Does it read sitemap indexes and .gz files?**
Yes. Indexes are followed up to 5 levels deep, and gzip is detected from the file content, not from the extension.

**Can I get images, videos or news metadata?**
Not at the moment: the output covers the page URL and its `lastmod`, `changefreq` and `priority`.

**How do I only get new pages?**
Run it on a schedule with **Modified since** set to the date of your last run. Only URLs declared with a more recent `lastmod`, or with no `lastmod`, are returned.

### Limitations

- Only URLs declared in sitemaps are returned; pages missing from the sitemap are not discovered.
- `lastmod`, `changefreq` and `priority` are whatever the site publishes and are not always accurate or present.
- Sitemap files above about 60 MB uncompressed are skipped, and each site is capped by **Max URLs per site** and **Max sitemap files per site**.
- Sites that block automated requests (HTTP 403 or 429) cannot be read; the Actor does not try to bypass protections.

# Actor input Schema

## `items` (type: `array`):

Domains (example.com), site URLs (https://example.com) or direct sitemap URLs (https://example.com/sitemap.xml.gz). For a domain, sitemaps are discovered from robots.txt, then from well-known paths. Sitemap indexes and .gz files are followed automatically.

## `maxUrlsPerSite` (type: `integer`):

Stop extracting a site after this many unique URLs. Each URL is one billed result, so this also caps your cost per site.

## `includePattern` (type: `string`):

Optional regular expression. Only URLs matching it are kept, for example /products/ or .html$.

## `excludePattern` (type: `string`):

Optional regular expression. URLs matching it are dropped, for example /tag/|/page/\d+.

## `modifiedSince` (type: `string`):

Optional date (for example 2026-01-31). Keeps URLs whose lastmod is on or after this date. URLs with no lastmod are kept, since their age is unknown.

## `maxSitemapsPerSite` (type: `integer`):

Safety limit on the number of sitemap files fetched for one site (large indexes can list thousands).

## `timeoutSecs` (type: `integer`):

Network timeout for each robots.txt or sitemap download.

## Actor input object example

```json
{
  "items": [
    "www.sitemaps.org"
  ],
  "maxUrlsPerSite": 1000,
  "maxSitemapsPerSite": 500,
  "timeoutSecs": 20
}
```

# Actor output Schema

## `overview` (type: `string`):

Main fields for each URL.

## `results` (type: `string`):

Complete result items.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        "www.sitemaps.org"
    ],
    "maxUrlsPerSite": 1000
};

// Run the Actor and wait for it to finish
const run = await client.actor("obligate_castle/robust-sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "items": ["www.sitemaps.org"],
    "maxUrlsPerSite": 1000,
}

# Run the Actor and wait for it to finish
run = client.actor("obligate_castle/robust-sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    "www.sitemaps.org"
  ],
  "maxUrlsPerSite": 1000
}' |
apify call obligate_castle/robust-sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,obligate_castle/robust-sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5gd5SCe8bnwGX3zTz/builds/5r6FrrobGcXPNQp7O/openapi.json