# Sitemap URL Extractor – Get All URLs from XML Sitemaps (`chris_elder/sitemap-url-extractor`) Actor

Extract every URL from a website's sitemaps. Auto-discovers sitemaps via robots.txt, follows nested indexes, handles .xml.gz, text, RSS/Atom and malformed XML. Returns lastmod, hreflang and optional HTTP status. $0.50 per 1,000 URLs; failed sitemaps are free.

- **URL**: https://apify.com/chris_elder/sitemap-url-extractor.md
- **Developed by:** [Chris Elder](https://apify.com/chris_elder) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor – get every URL from any website's sitemap

**Sitemap URL Extractor** returns every page URL a website publishes in its XML sitemaps – in seconds, with no crawling, no browser and no proxies. Give it a homepage and it finds the sitemaps itself (via `robots.txt` and common locations), or paste sitemap URLs directly.

✅ **Pay per result: $0.50 per 1,000 URLs** · 🆓 **Failed sitemaps, duplicates and filtered-out URLs are never charged** · 🔑 No API key · ⚡ 100,000 URLs in about a minute

[▶ Try it with one click](#how-to-use-it) · [📄 Real example output](#output-example-from-a-real-public-run) · [💲 Pricing](#how-much-does-it-cost-to-extract-sitemap-urls) · [🔌 API & integrations](#integrations-api-ai-agents-make-zapier) · [❓ FAQ](#faq) · [🛟 Support](#support-reliability-and-feature-requests)

### What does Sitemap URL Extractor do?

It follows nested **sitemap indexes**, unpacks **`.xml.gz`**, reads **plain-text, RSS and Atom sitemaps**, survives **malformed XML**, and returns one clean, de-duplicated list with `lastmod`, `changefreq`, `priority`, **hreflang alternates**, **image** counts and **Google News** data. An optional **HTTP status check** finds broken and redirecting URLs listed in the sitemap.

- 🔎 **Auto-discovery** from `robots.txt` `Sitemap:` lines, then `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemap.txt`
- 🌳 **Nested indexes** (configurable depth): WordPress/Yoast, Shopify, Wix, Squarespace, Next.js, Docusaurus, news sites
- 🗜️ **Every format**: gzip, text sitemaps, RSS 2.0 / Atom, BOMs, CDATA, entities, broken XML (fallback parser)
- 🧹 **Filters**: regex include/exclude, same-host only, `lastmod` on/after a date ("what changed since last week?")
- 🩺 **Optional status check** (HEAD, falls back to GET): 404s, 5xx and redirect chains
- 🛡️ **Polite by default**: one request at a time per host, 300 ms delay, honest User-Agent, respects `robots.txt` for guessed paths and status checks
- 📊 **Run summary** per website: how the sitemap was found, files read, files failed (with the exact error), duplicates and filtered counts

### Use cases

- **SEO audits** – check index coverage, find orphan, broken or redirecting URLs, compare sitemap vs. crawl
- **Crawler seeding** – feed an exact URL list to Website Content Crawler, Cheerio/Playwright scrapers or a RAG ingestion pipeline instead of crawling blindly
- **Content monitoring** – schedule a daily run with `lastmodAfter` and get only new or updated pages from a competitor or publisher
- **Site inventories & migrations** – export every URL to a spreadsheet before a redesign or domain move
- **AI agents** – a cheap, deterministic "list all pages of this site" tool

### How to use it

1. Click **Try for free** – the input is prefilled (`https://crawlee.dev`, max 100 URLs), so **Start** works straight away and costs about $0.05.
2. Replace it with your websites (`https://example.com`) or sitemap URLs (`https://example.com/sitemap_index.xml`).
3. Optional: include/exclude patterns (e.g. include `/blog/`, exclude `/tag/`), a `lastmodAfter` date, or **Check HTTP status**.
4. Click **Start**. Download JSON, CSV, Excel, XML or HTML, or read results via API.

#### Input example

```json
{
  "startUrls": [{ "url": "https://crawlee.dev" }],
  "maxUrls": 100
}
```

More options: `includeUrlPatterns`, `excludeUrlPatterns`, `lastmodAfter` (YYYY-MM-DD), `sameHostOnly`, `includeAlternates`, `includeImages`, `checkStatus`, `maxStatusChecks`, `maxSitemaps`, `maxDepth`. Every field is explained in the **Input** tab.

#### Output example (from a real public run)

This is the first item from the example run above (crawlee.dev, 100 URLs, 9 Oct 2026). **See the full public dataset:** [JSON](https://api.apify.com/v2/datasets/jnsOnueMEZwU5gzMR/items?format=json\&clean=1) · [CSV](https://api.apify.com/v2/datasets/jnsOnueMEZwU5gzMR/items?format=csv\&clean=1)

```json
{
  "url": "https://crawlee.dev/blog",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "sitemapDepth": 0,
  "inputUrl": "https://crawlee.dev",
  "host": "crawlee.dev",
  "alternates": [],
  "imageCount": 0
}
```

With **Check HTTP status** on, each item also gets `statusCode`, `finalUrl` and `redirected`. News sitemaps add `newsTitle` and `newsPublicationDate`. A `RUN_SUMMARY` record in the key-value store explains, per input, what was found, skipped or failed.

| Field | Description |
|---|---|
| `url` | Page URL from the sitemap (de-duplicated across all sitemaps) |
| `lastmod`, `changefreq`, `priority` | Values from the sitemap (`null` if the site doesn't publish them) |
| `sitemapUrl`, `sitemapDepth`, `inputUrl` | Where the URL came from |
| `alternates` | hreflang language versions |
| `imageCount`, `images` | Image sitemap data (`images` only with *Include image URLs*) |
| `statusCode`, `finalUrl`, `redirected` | Only with *Check HTTP status* |

### How much does it cost to extract sitemap URLs?

Pay-per-event pricing: you pay only for URLs you get. No platform usage charges on top.

| Event | Price | Notes |
|---|---|---|
| **URL extracted** | **$0.0005** ($0.50 per 1,000) | Silver plan $0.0004, Gold and above $0.0003 |
| URL status checked (optional) | $0.0005 | Only when *Check HTTP status* is on |
| Run start | $0.0005 per GB of memory | Default 512 MB = one start event = $0.0005 |

**Worked examples (Free/Starter prices, default memory):**

| Job | Calculation | Total |
|---|---|---|
| One-click example (100 URLs) | 100 × $0.0005 + $0.0005 | **$0.05** |
| Blog section of a site (2,000 URLs) | 2,000 × $0.0005 + $0.0005 | **$1.00** |
| Mid-size site + broken-link check (10,000 URLs, 1,000 status checks) | $5.00 + $0.50 + $0.0005 | **$5.50** |
| Large site (100,000 URLs) | 100,000 × $0.0005 + $0.0005 | **$50.00** (Gold: $30.00) |

**Never charged:** duplicates, URLs removed by your filters, sitemaps that fail to download or parse. Cap spending with **Max URLs** or Apify's **Max cost per run** – the Actor stops cleanly at the limit. Apify's Free plan includes $5 of monthly credit.

### Integrations (API, AI agents, Make, Zapier)

**API** – run it and get results in one call:

```bash
curl -X POST "https://api.apify.com/v2/acts/chris_elder~sitemap-url-extractor/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://crawlee.dev"}],"maxUrls":100}'
```

```python
from apify_client import ApifyClient
client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("chris_elder/sitemap-url-extractor").call(run_input={"startUrls": [{"url": "https://crawlee.dev"}], "maxUrls": 100})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"])
```

**AI agents (MCP)** – add `https://mcp.apify.com/?tools=chris_elder/sitemap-url-extractor` to Claude, Cursor, VS Code or any MCP client. The agent can then call "list all URLs of this site" as a tool.

**Make, Zapier, n8n** – use the Apify app's **Run Actor** action, pick `chris_elder/sitemap-url-extractor`, then map dataset items into Google Sheets, Slack, Airtable or your CRM. **Schedules and webhooks** in Apify Console let you run it daily and push results anywhere.

### Limits and good to know

- It reads sitemaps only – it never crawls HTML pages. If a site publishes no sitemap, the run finishes with a clear warning (and costs only the start fee).
- `lastmod` is self-reported by the site and is often missing or inaccurate.
- Very large sites: raise **Max sitemap files per website** and **Max URLs**. 100,000 URLs typically take about a minute.
- Sitemaps behind logins or bot walls can't be read.

### Is it legal?

Sitemaps are files that website owners publish so that automated tools can discover their URLs. This Actor reads only those files and `robots.txt`. It collects no personal data, doesn't log in and doesn't bypass any protection. You are responsible for how you use the URLs and for each website's terms. See [Is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/)

### FAQ

**Does it work with WordPress, Shopify, Wix and Squarespace?** Yes. They publish standard sitemap indexes, which are followed automatically.

**Does it render JavaScript?** It doesn't need to: sitemaps are plain XML or text.

**Why did I get fewer URLs than the site has pages?** The site's sitemap lists fewer pages, or your *Max URLs* or filters stopped the run. Check `RUN_SUMMARY`.

**Can I get only new pages?** Yes. Set *Only URLs modified after* and schedule the Actor.

**How do I check for broken links in my sitemap?** Turn on *Check HTTP status* and filter results for `statusCode >= 400`.

### Support, reliability and feature requests

- **Failed items are free.** You pay only for URLs delivered.
- **Tested on real sites before every release** (gov.uk, crawlee.dev, docs.apify.com, BBC and WordPress news sitemaps).
- **Issues answered within 2 business days.** Open an issue on the **Issues** tab with your input URL; parsing bugs are fixed first.
- **Feature requests welcome** – open an issue titled "Feature request: …".
- Built by Chris Elder with AI-assisted development; every release is reviewed and tested before it ships.

### Need the data done for you?

If you'd rather not set up runs yourself, the UK company behind these tools (GPT Shop Ltd) offers a fixed-price done-for-you scrape. Tell us the website and the fields you need, and you get a clean CSV or Google Sheet within 48 hours of the confirmed brief, up to 5,000 rows, for £149 or $179. If we can't get the data, you're refunded in full. [See the done-for-you scraper service](https://sme-free-tools.pages.dev/done-for-you-scraper/?utm_source=apify\&utm_medium=readme\&utm_campaign=sitemap-url-extractor)

### More tools by Chris Elder

- [PDF to Markdown & Text Converter (with OCR)](https://apify.com/chris_elder/pdf-to-markdown) – LLM- and RAG-ready Markdown from any PDF
- [Bulk Lighthouse & PageSpeed Audit](https://apify.com/chris_elder/lighthouse-batch-auditor) – Core Web Vitals and SEO scores for many URLs (pairs well with this Actor's sitemap output)
- [Bulk WHOIS Domain Lookup](https://apify.com/chris_elder/bulk-whois-domain-lookup) – registrar, expiry, availability and DNS for many domains via official RDAP
- [Image to Text OCR](https://apify.com/chris_elder/image-to-text-ocr) – bulk text extraction from images in 100+ languages
- All tools: [apify.com/chris_elder](https://apify.com/chris_elder)

# Actor input Schema

## `startUrls` (type: `array`):

Website homepages (sitemaps are auto-discovered from robots.txt and common locations) or direct sitemap URLs (.xml, .xml.gz, .txt, RSS/Atom).

## `maxUrls` (type: `integer`):

Stop after this many unique URLs (across all inputs). You pay per URL returned, so this caps your cost.

## `includeUrlPatterns` (type: `array`):

Only keep URLs matching at least one of these regular expressions (case-insensitive), e.g. /blog/ or .pdf$

## `excludeUrlPatterns` (type: `array`):

Drop URLs matching any of these regular expressions, e.g. /tag/ or ?page=

## `lastmodAfter` (type: `string`):

Keep only URLs whose <lastmod> is on/after this date (YYYY-MM-DD). URLs without lastmod are dropped when this is set. Great for 'what changed since last week' monitoring.

## `sameHostOnly` (type: `boolean`):

Drop URLs whose host differs from the input website (www. is ignored).

## `includeAlternates` (type: `boolean`):

Output \<xhtml:link rel="alternate" hreflang> language versions for each URL.

## `includeImages` (type: `boolean`):

Output the list of <image:loc> URLs per page (the image count is always included).

## `checkStatus` (type: `boolean`):

Send one polite HEAD request per URL (falls back to GET) and add statusCode, finalUrl and redirected. Finds broken (4xx/5xx) and redirecting URLs listed in sitemaps. Charged as an extra event per URL checked. Respects robots.txt.

## `maxStatusChecks` (type: `integer`):

Upper limit of URLs to status-check per run.

## `maxSitemaps` (type: `integer`):

Safety limit on how many sitemap files are downloaded per input.

## `maxDepth` (type: `integer`):

How deep to follow nested sitemap indexes.

## `requestDelayMs` (type: `integer`):

Politeness delay. The default keeps load on target websites negligible.

## `respectRobotsTxt` (type: `boolean`):

Skip guessed sitemap locations and status checks that robots.txt disallows. Sitemaps declared in robots.txt are always fetched.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxUrls": 100,
  "includeUrlPatterns": [],
  "excludeUrlPatterns": [],
  "sameHostOnly": false,
  "includeAlternates": true,
  "includeImages": false,
  "checkStatus": false,
  "maxStatusChecks": 1000,
  "maxSitemaps": 500,
  "maxDepth": 5,
  "requestDelayMs": 300,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        }
    ],
    "maxUrls": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("chris_elder/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev" }],
    "maxUrls": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("chris_elder/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxUrls": 100
}' |
apify call chris_elder/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,chris_elder/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/vG8BJyilUpwaCmnhV/builds/EswgamzF1pU8skgyl/openapi.json