# XML Sitemap URL Extractor (`devilscrapes/xml-sitemap-url-extractor`) Actor

Turn any site's sitemap.xml, sitemap index, or robots.txt Sitemap directive into a clean URL dataset with loc, lastmod, changefreq and priority per row. Recursively expands sitemap indexes (bounded depth), follows redirects, and decompresses .xml.gz automatically.

- **URL**: https://apify.com/devilscrapes/xml-sitemap-url-extractor.md
- **Developed by:** [DevilScrapes](https://apify.com/devilscrapes) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

<div align="center">
  <img src=".actor/icon.svg" width="160" alt="Devil Scrapes mark" />

## XML Sitemap URL Extractor

**💰 $2.20 / 1 000 results**  ·  pay only for results  ·  no credit card to try

*We do the dirty work so your dataset stays clean.* 😈

Turn any site's sitemap.xml, sitemap index, or robots.txt Sitemap directive into a clean URL dataset with loc, lastmod, changefreq and priority per row. Recursively expands sitemap indexes (bounded depth), follows redirects, and decompresses .xml.gz automatically.

</div>

***

### 🎯 What this scrapes

Give it a list of site URLs and it discovers every sitemap the site publishes — first by trying `/sitemap.xml` directly, then by parsing `Sitemap:` lines out of `robots.txt` (a 200 response on either is not treated as proof of a real sitemap; the Actor checks the Content-Type and looks like real sitemap XML before it commits to parsing it). Sitemap **index** files (`<sitemapindex>`) are expanded recursively up to a configurable depth, gzip-compressed sitemaps (`.xml.gz`) are decompressed transparently, and redirects are followed. Every leaf `<url>` entry becomes one dataset row tagged with the sitemap file and source site it came from.

### 🔥 What we handle for you

- 🛡️ **Browser fingerprint rotation** — `curl-cffi` impersonates real Chrome / Firefox / Safari TLS handshakes so the target sees a browser, not Python.
- 🌐 **Residential proxy rotation** via Apify Proxy — fresh session and exit IP on every block.
- 🔁 **Retries with exponential backoff** on `408 / 429 / 5xx` — up to 5 attempts per page, `Retry-After` honoured.
- 🧱 **Rate-limit-aware pacing** — when the target pushes back, we slow down instead of getting banned.
- 🧊 **Clean, typed dataset rows** — Pydantic-validated, ISO-8601 timestamps, stable IDs, JSON / CSV / Excel export straight from the Apify Console.
- 💰 **Pay-Per-Event pricing** — you only pay for results that hit your dataset. No data, no charge.

### 💡 Use cases

- **SEO content audits** — pull a site's full published-URL inventory in one pass, no crawling required.
- **Site migration checks** — diff sitemap output before/after a replatform to confirm nothing silently dropped.
- **Competitive research** — see how many pages a competitor publishes and how often they update (`lastmod`, `changefreq`).
- **Freshness monitoring** — schedule runs and diff `lastmod` values to catch stale or newly-published pages.

### ⚙️ How to use it

1. Click **Try for free** at the top of the page.
2. Fill in the input form — most fields have sensible defaults.
3. Click **Start**. Output streams into the run's dataset.
4. Export from **Storage → Dataset** as JSON, CSV, or Excel — or fetch via the API.

### 📥 Input

| Field | Type | Required | Default | Notes |
|---|---|:--:|---|---|
| `siteUrls` | `array` | **yes** | \['https://www.cloudflare.com', 'https://apify.com'] | Root or homepage URLs to scan, one per line. For each, the Actor tries `/sitemap.xml` directly and parses `Sitemap:`… |
| `maxUrlsPerSite` | `integer` | no | 5000 | Stop emitting rows for one site after this many URLs. Also lets the Actor stop fetching further sitemap files for that… |
| `maxSitemapDepth` | `integer` | no | 3 | How many levels of nested `<sitemapindex>` to follow. A depth-1 index pointing at leaf sitemaps is depth 1; an index of… |
| `maxTotalUrls` | `integer` | no | 20000 | Hard ceiling on rows emitted across every site in this run, regardless of `maxUrlsPerSite`. Bandwidth is our biggest… |
| `proxyConfiguration` | `object` | no | {'useApifyProxy': True} | Sitemaps are public, unauthenticated files, so this is mainly IP hygiene and rate-limit avoidance across many sites in… |

#### Example input

```json
{
  "siteUrls": [
    "https://www.cloudflare.com",
    "https://apify.com"
  ],
  "maxUrlsPerSite": 10,
  "maxSitemapDepth": 3,
  "maxTotalUrls": 20,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

### 📤 Output

Every row is one dataset item.

| Field | Type | Notes |
|---|---|---|
| `loc` | `string` | The URL listed in the sitemap `<loc>` element. |
| `lastmod` | `['string', 'null']` | `<lastmod>` value, verbatim (ISO-8601 date or datetime per the sitemap protocol). |
| `changefreq` | `['string', 'null']` | `<changefreq>` value (e.g. `daily`, `weekly`) when the sitemap publishes one. |
| `priority` | `['number', 'null']` | `<priority>` value (0.0–1.0) when the sitemap publishes one. |
| `sitemap_url` | `string` | The specific sitemap file this row was read from (the leaf file, after index expansion). |
| `source_site` | `string` | The original input site URL this sitemap was discovered for. |

#### Example output

```json
{
  "loc": "https://apify.com/about",
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "sitemap_url": "https://apify.com/sitemap/pages.xml",
  "source_site": "https://apify.com"
}
```

### 💰 Pricing

Pay-Per-Event — you pay only when these events fire:

| Event | USD | What it is |
|---|---:|---|
| `actor-start` | $0.2 | One-off warm-up charge per run |
| `result` | $0.002 | Per dataset item |

Example: 1 000 results at the rates above ≈ **$2.20**. No subscription, no minimum, no card to start — Apify gives every new account $5 of free credit.

### 🚧 Limitations

- We read what a site's sitemap files declare — we don't crawl or discover pages a site doesn't list in a sitemap.
- The standard sitemap protocol caps a single leaf file at 50,000 URLs; large sites split across many files via a sitemap index, which we follow up to `maxSitemapDepth`.
- `maxUrlsPerSite` and `maxTotalUrls` are hard caps by design (bandwidth control) — raise them if you need a fuller export, within the input limits.

### ❓ FAQ

**What if a site has no sitemap.xml or robots.txt Sitemap entry?**

That site contributes zero rows and the run still SUCCEEDS — the status message reports how many of the input sites actually yielded a sitemap. This is a completed search that found nothing, not a failure.

**Does it handle sitemap index files?**

Yes — `<sitemapindex>` files are expanded recursively up to `maxSitemapDepth`. A visited-URL set stops circular or self-referencing indexes from looping.

**Does it handle gzip-compressed sitemaps?**

Yes, transparently — both `.xml.gz` URLs and sitemaps served with `Content-Encoding: gzip` (or just gzip magic bytes regardless of headers) are decompressed before parsing.

**What happens if one site is unreachable or its sitemap is malformed?**

That one site or sitemap file is skipped with a logged warning; the rest of the run continues. One bad target never fails the whole run.

**Why did I get an HTML page instead of sitemap data for a site?**

Some sites return a 200 HTML page (a redirect landing page, a CMS 404-as-200) at the sitemap path. The Actor checks the Content-Type and the actual root XML tag before parsing — an HTML shell is logged and skipped, not force-parsed as XML.

### 💬 Your feedback

Spotted a bug, hit a weird edge case, or need a new field? Open an
issue on the Actor's **Issues** tab on Apify Console — we ship
fixes weekly and we read every report.

***

<div align="center">

Built by **[Devil Scrapes](https://apify.com/DevilScrapes)** 😈 — a small fleet of
opinionated public-data Actors. Honest pricing, real engineering, zero fine print.

</div>

# Actor input Schema

## `siteUrls` (type: `array`):

Root or homepage URLs to scan, one per line. For each, the Actor tries `/sitemap.xml` directly and parses `Sitemap:` lines from `/robots.txt`.

## `maxUrlsPerSite` (type: `integer`):

Stop emitting rows for one site after this many URLs. Also lets the Actor stop fetching further sitemap files for that site once reached.

## `maxSitemapDepth` (type: `integer`):

How many levels of nested `<sitemapindex>` to follow. A depth-1 index pointing at leaf sitemaps is depth 1; an index of indexes is depth 2. Enforced with a visited-URL set, so a self-referencing or circular index cannot loop forever.

## `maxTotalUrls` (type: `integer`):

Hard ceiling on rows emitted across every site in this run, regardless of `maxUrlsPerSite`. Bandwidth is our biggest cost driver, so this is the last line of defense against an unbounded crawl.

## `proxyConfiguration` (type: `object`):

Sitemaps are public, unauthenticated files, so this is mainly IP hygiene and rate-limit avoidance across many sites in one run, not anti-bot evasion.

## Actor input object example

```json
{
  "siteUrls": [
    "https://www.cloudflare.com",
    "https://apify.com"
  ],
  "maxUrlsPerSite": 5000,
  "maxSitemapDepth": 3,
  "maxTotalUrls": 20000,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `datasetItems` (type: `string`):

All dataset items as JSON.

## `datasetItemsCsv` (type: `string`):

Same data exported to CSV.

## `datasetView` (type: `string`):

Open the run dataset in the Console.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteUrls": [
        "https://www.cloudflare.com",
        "https://apify.com"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("devilscrapes/xml-sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "siteUrls": [
        "https://www.cloudflare.com",
        "https://apify.com",
    ],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("devilscrapes/xml-sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteUrls": [
    "https://www.cloudflare.com",
    "https://apify.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call devilscrapes/xml-sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,devilscrapes/xml-sitemap-url-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bzjK7dDPKmC0QAAWu/builds/ZDd02r5y5KEZKHlTd/openapi.json