# Sitemap Extractor: All URLs from XML Sitemaps (`ataism/sitemap-extractor`) Actor

Get every URL from a site's XML sitemaps and indexes, with lastmod, changefreq and priority. robots.txt discovery, .gz, filters. $0.40 per 1,000 URLs.

- **URL**: https://apify.com/ataism/sitemap-extractor.md
- **Developed by:** [Kaizoku Data](https://apify.com/ataism) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.40 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor: All URLs from XML Sitemaps

Get **every URL a website lists in its XML sitemaps**: one dataset row per URL with `lastmod`, `changefreq`, `priority`, the sitemap it came from and counts of image, video, news and hreflang entries. Give a site URL and the Actor finds the sitemaps itself, or give a sitemap URL. No browser, no proxy, no key.

**Price: $0.40 per 1,000 URLs ($0.0004 per URL row)**, plus Apify's default Actor-start event. Sitemap files, robots.txt and error rows are free.

- **Finds the sitemaps for you:** reads the `Sitemap:` lines of robots.txt, then tries `/sitemap.xml` and `/sitemap_index.xml`.
- **Follows sitemap indexes** to any depth up to 5 (default 3), reads `.xml.gz` files and plain-text sitemaps.
- **Filters before you pay:** keep only URLs containing a text or `*` pattern, skip others, keep only URLs modified on or after a date. Filtered URLs and duplicates are not charged.
- **Streams large sitemaps:** a 50,000-URL file is read as it downloads and rows are written in batches; 50,000 URLs from apify.com's sitemap index took about 9 s.
- **Quick start:** press *Start*. The prefilled input reads 20 URLs from `https://crawlee.dev/` (Apify's open-source crawling library; its robots.txt allows all crawlers). That is 2 requests, about 3 seconds and $0.008.

> **Polite by design.** robots.txt is obeyed for the sitemap requests themselves, `Crawl-delay` is honoured in full, at most one request per second per host, an honest User-Agent, and blocks (401, 403, 429, challenge pages) are reported as free rows, never worked around.

### What it does

1. For each start URL: a **site URL** (`https://crawlee.dev/`, `https://example.org/blog/`) is looked up at the origin. The Actor reads robots.txt; its `Sitemap:` lines are the sitemaps. Without such a line it tries `/sitemap.xml`, then `/sitemap_index.xml`. A **sitemap URL** (path ending in `.xml`, `.xml.gz`, `.gz` or `.txt`, or containing `sitemap`) is read directly.
2. Each sitemap is downloaded and parsed as it streams in. A `<urlset>` gives URL rows; a `<sitemapindex>` gives child sitemaps, read depth first in file order down to **Max index depth**.
3. Filters, then de-duplication, decide which URLs become rows. Rows are written every 1,000 URLs or 30 seconds, and the run stops cleanly at **Max URLs**, at your max charge, or at a safety cap.

Every request, robots.txt included, is rate-limited per host (one per second, or slower if robots.txt asks for it). Sitemaps on other hosts (for example a CDN listed in robots.txt) are allowed; the robots.txt of that host is read and obeyed too. Social-media and Google hosts (YouTube, Facebook, Instagram, X, TikTok, LinkedIn, Reddit, Google domains and similar) are **never contacted**: such start URLs, and redirects to them, give a free row with `errorCode: "platform"`.

### Input

| Field | What it does |
|---|---|
| **Site or sitemap URLs** (`startUrls`) | Up to 100. A site URL is discovered through robots.txt, then `/sitemap.xml` and `/sitemap_index.xml`. A sitemap URL is read directly. A bare host (`crawlee.dev`) gets `https://`. |
| **Max URLs** (`maxUrls`) | URL rows for the whole run. Default 50,000, hard cap 200,000 (100,000 when the run has less than 512 MB of memory). The run stops cleanly when it is reached (`stoppedBy: "max-urls"`). |
| **Only URLs containing** (`includePattern`) | Optional text, case-insensitive; `*` means "anything". `/blog/` keeps blog URLs; `/products/*/reviews` keeps product review pages. No regular expressions. |
| **Skip URLs containing** (`excludePattern`) | Optional, same rules, for example `/tag/` or `?page=`. Wins over the include pattern. |
| **Modified on or after** (`lastmodAfter`) | Optional date, `YYYY-MM-DD` or a full date-time with zone. Keeps URLs whose `lastmod` is on or after it. **URLs without a `lastmod` are dropped while this is set.** Child sitemaps whose index entry has an older `lastmod` are not downloaded at all. |
| **Max index depth** (`maxDepth`) | Levels of sitemap indexes followed. 0 reads only the sitemaps you give or robots.txt lists; default 3; maximum 5. Deeper child sitemaps are counted in one free `depth-cap` row. |
| **Remove duplicates** (`dedupe`) | Default on: a URL in several sitemaps is output and charged once (the first sitemap wins). Off: one row per listing. |

#### Example inputs

**1. All URLs of a site (sitemaps found automatically):**

```json
{ "startUrls": ["https://crawlee.dev/"] }
```

**2. Only blog posts changed this year, from a known sitemap index:**

```json
{
  "startUrls": ["https://www.example.org/sitemap_index.xml"],
  "includePattern": "/blog/",
  "excludePattern": "/blog/tag/",
  "lastmodAfter": "2026-01-01"
}
```

**3. Several sites, first 1,000 URLs each run, top-level sitemaps only:**

```json
{
  "startUrls": ["https://crawlee.dev/", "https://www.example.org/", "https://shop.example.com/sitemap.xml.gz"],
  "maxUrls": 1000,
  "maxDepth": 0
}
```

### Output

One row per URL:

```json
{
  "url": "https://crawlee.dev/blog",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "imageCount": 0,
  "videoCount": 0,
  "newsCount": 0,
  "alternateCount": 0,
  "sourceSitemap": "https://crawlee.dev/sitemap.xml",
  "site": "https://crawlee.dev",
  "status": "ok"
}
```

- `lastmod` and `changefreq` are given exactly as the sitemap writes them (`lastmod` is a W3C date or date-time such as `2026-10-01T08:00:00+00:00`); `null` when absent. `priority` is a number from 0 to 1, or `null`.
- `imageCount`, `videoCount`, `newsCount` and `alternateCount` count the Google image, video and news extension entries and the `xhtml:link rel="alternate" hreflang` entries of that URL (0 when there are none). Their contents (image URLs, titles) are not output.
- `url` is resolved to an absolute URL; entries with an invalid or non-http(s) `<loc>`, or longer than 2,048 characters, are skipped and counted in `RUN_SUMMARY.invalidEntries`.

Problems are **free rows** with `status: "error"`:

```json
{
  "url": "https://www.example.org/sitemap-old.xml",
  "status": "error",
  "errorCode": "http",
  "message": "The server answered HTTP 404.",
  "site": "https://www.example.org",
  "sourceSitemap": "https://www.example.org/sitemap_index.xml",
  "httpStatus": 404
}
```

#### Error codes

| `errorCode` | Meaning |
|---|---|
| `platform` | A social-media, video or Google host (also when reached by a redirect). Never contacted. |
| `invalid-url`, `ssrf` | Not an http(s) URL, or a private, loopback or internal address. |
| `dns`, `network`, `timeout`, `redirects` | Host not found, connection failed, no data for 30 s or a file took over 5 minutes, more than 5 redirects. |
| `http` | Any other non-2xx answer (`httpStatus` says which, for example 404). |
| `refused` | 401, 403, 429, 503, 999 or a bot challenge. Not retried, not worked around. After 2 refusals the host's other sitemaps are skipped (`host-skipped`, one summary row with `count`). |
| `robots-blocked` | robots.txt disallows the sitemap's path for this Actor. Not requested. |
| `robots-unavailable` | robots.txt answered 5xx or 429, or could not be reached, twice. No sitemap of that site is fetched without its rules (RFC 9309). A 4xx robots.txt means "no rules". |
| `slow-host` | robots.txt asks for a `Crawl-delay` over 30 s, or honouring the delays would take more than 10 minutes of waiting in this run. Skipped; **the delay is never shortened**. |
| `no-sitemap` | A site URL with no `Sitemap:` line in robots.txt and no sitemap at `/sitemap.xml` or `/sitemap_index.xml`. |
| `not-sitemap` | The file is an HTML page, an RSS or Atom feed (not supported), another XML document or empty. |
| `malformed-xml` | The XML is broken or ends early (a tag left open), or a tag, comment or CDATA section is too long. URLs read before the problem are kept (and charged); the message says how many. |
| `gzip-error`, `too-large` | A damaged `.gz` file; a file over 60 MB downloaded or 60 MB once decompressed (gzip bombs stop here). URLs read before are kept. |
| `entry-cap` | A file with more than 50,000 entries (the protocol limit); the rest of it was not read. |
| `depth-cap`, `sitemap-cap` | Summary rows (with `count`): child sitemaps deeper than Max index depth, or beyond 500 sitemap files per run. |
| `internal` | An unexpected error in this Actor, not a problem with your site. |

#### RUN_SUMMARY (key-value store record)

URL rows and charged events, error rows, sitemap files read and failed, indexes read, entries read, duplicates, entries filtered by pattern or date, invalid entries, child sitemaps skipped (by date, depth, cap or refusing host), bytes downloaded (and the part that led to no URL row), seconds spent waiting for `Crawl-delay`, errors by code, and `stoppedBy`: `finished`, `max-urls`, `max-charge`, `unpaid-work-cap`, `time-limit`, `migrating`, `aborted` or `error`.

### Pricing

Pay per event: **$0.0004 per URL row** (event `url-extracted`), plus Apify's standard Actor-start event. No other fee from this Actor.

| URL rows | Cost |
|---|---|
| 20 (prefilled example) | $0.008 |
| 1,000 | $0.40 |
| 10,000 | $4.00 |
| 50,000 (default Max URLs) | $20.00 |
| 200,000 (hard cap; needs 512 MB) | $80.00 |

**Free:** robots.txt and sitemap files, error rows, URLs removed by your filters, duplicates (with Remove duplicates on) and invalid entries. Set **Max URLs**, or **Max total charge** in the run options, to cap spending: the run checks the budget before each sitemap and before each write, and never writes a row it cannot charge. Apify subscription discounts apply on top.

To keep free work bounded, a run stops (`stoppedBy: "unpaid-work-cap"`) after 50 failed sitemaps or start URLs, 100 MB downloaded for sitemaps that gave no URL row, or 1,000,000 sitemap entries that were filtered out, duplicates or invalid. Everything found up to then is saved.

### Use it via API, n8n or Make

Get an API token in Apify Console under *Settings*, *API & Integrations*, and keep it in an environment variable (`APIFY_TOKEN` below). Never paste it into shared code.

**cURL** (starts a run, waits and returns the rows; the synchronous endpoint waits up to 5 minutes, so use it for small runs and the clients below for big ones):

```bash
curl -X POST "https://api.apify.com/v2/acts/ataism~sitemap-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev/"], "maxUrls": 100}'
```

**Python** (`pip install "apify-client>=3"`):

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("ataism/sitemap-extractor").call(run_input={
    "startUrls": ["https://crawlee.dev/"],
    "includePattern": "/docs/",
})
for row in client.dataset(run.default_dataset_id).iterate_items():
    if row["status"] == "ok":
        print(row["url"], row["lastmod"])
```

(With apify-client versions before 3.0, `call()` returns a dict: use `run["defaultDatasetId"]`.)

**JavaScript** (`npm install apify-client`):

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('ataism/sitemap-extractor').call({
    startUrls: ['https://crawlee.dev/'],
    lastmodAfter: '2026-01-01',
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const row of items.filter((r) => r.status === 'ok')) console.log(row.url, row.lastmod);
```

**n8n:**

1. Add a Schedule trigger (for example weekly) or any trigger that gives you a site URL.
2. Add the **Apify** node, operation **Run an Actor and get dataset**, Actor `ataism/sitemap-extractor`, input `{"startUrls": ["{{ $json.siteUrl }}"], "lastmodAfter": "{{ $now.minus({days: 7}).toFormat('yyyy-MM-dd') }}"}`.
3. Send the new or changed URLs to a sheet, a crawler or Slack; filter on `status` = `ok`.

**Make:**

1. Add the **Apify** module **Run an Actor**, choose `ataism/sitemap-extractor`, put your site URL in the input JSON and set *Run synchronously* to *Yes*.
2. Add the **Apify** module **Get Dataset Items** and map the run's default dataset ID (one bundle per URL).
3. Send the rows to Google Sheets, Airtable, a database or a webhook.

### Privacy and terms

- The Actor reads robots.txt and the sitemap files of the sites you enter, nothing else: it never opens the pages the sitemaps list. It outputs the URLs and dates the sitemaps contain. A sitemap that puts personal data into its URLs (for example user profile URLs) puts it into your dataset too.
- Nothing is sent to any AI or third-party service. No logins, no proxies, no CAPTCHA solving.
- Your dataset belongs to your Apify account and follows your retention settings.
- **You are responsible for having the right to read and use the sites you enter** (their terms, your agreement with the owner). Sitemaps are published for crawlers, and the Actor obeys robots.txt, but it does not know your agreement with the site.
- User-Agent: `sitemap-extractor-actor/0.1 (https://apify.com/ataism/sitemap-extractor)`. In robots.txt, `User-agent: sitemap-extractor-actor` targets this Actor (exact match; otherwise the `*` group applies).

### Safety

Only `http` and `https` to **public** addresses: private, loopback, link-local and internal addresses and other schemes are blocked, also after every redirect (at most 5) and at connection time. Each sitemap is capped at 60 MB downloaded and 60 MB decompressed, 50,000 entries and 5 minutes; robots.txt at 512 KB. The XML parser is a streaming scanner with bounded buffers: it builds no tree (deep nesting costs nothing), never expands DTD entities (no "billion laughs"), and caps every tag, value and comment.

### Limits and honest caveats

- Limits: 100 start URLs, 200,000 URL rows (100,000 below 512 MB of memory), 500 sitemap files, 5 index levels and 60 minutes per run; 50,000 entries per sitemap file.
- **Default memory is 256 MB**, enough for the default 50,000 URLs. Below 512 MB a run is capped at 100,000 URLs (with a warning in the log); for up to 200,000 URLs in one run, start it with **512 MB** (the de-duplication list and buffers grow with the URL count).
- Only what the sitemaps say: a page missing from the sitemaps is not found (this is not a crawler), and `lastmod` is only as accurate as the site makes it.
- RSS and Atom feeds used as sitemaps are not supported (`not-sitemap`). Image, video and news extensions are counted, not extracted.
- With **Modified on or after** set, URLs without `lastmod` are dropped, and child sitemaps whose index says they are older are skipped without downloading them. A site with wrong index dates can make you miss URLs; leave the date empty if in doubt.
- Rows are written in batches (every 1,000 URLs or 30 s). If the platform migrates the run, buffered rows are written and progress is saved; the resumed run reads the dataset and never writes or charges a URL twice.
- Up to one request per second per host, slower when robots.txt asks for a `Crawl-delay` (honoured in full up to 30 s; at most 10 minutes of such waiting per run). A site with many small sitemap files and a long delay therefore takes long, or ends in free `slow-host` rows.

### FAQ

**Do I pay for sitemap files or robots.txt?** No. Only URL rows (`status: "ok"`) are charged.

**The site has no sitemap. What happens?** You get one free `no-sitemap` row for it. Nothing is charged.

**Why is a sitemap `robots-blocked`?** The site's robots.txt disallows that path for crawlers in general or for this Actor. The Actor respects that; ask the site owner or use a sitemap they allow.

**Can I get only new pages each week?** Yes: schedule a run with **Modified on or after** set to last week's date (see the n8n example), or compare runs on `url`.

**Does the default run work without setup?** Yes. It reads 20 URLs from `https://crawlee.dev/` (2 requests, about 3 seconds, $0.008).

### Open-source components

The Actor uses the **Apify SDK** (`apify` 3.7.2, Apache-2.0). The sitemap scanner, robots.txt parser, filters and fetching are plain code in this Actor, with no other dependencies.

# Actor input Schema

## `startUrls` (type: `array`):

One per line, up to 100. A site URL (https://crawlee.dev/) is looked up in robots.txt (Sitemap: lines), then at /sitemap.xml and /sitemap_index.xml. A sitemap URL (path ending in .xml, .xml.gz or .txt, or containing "sitemap") is read directly. Social-media and Google hosts are refused.

## `maxUrls` (type: `integer`):

Upper limit of URL rows for the whole run (each row is charged). Default 50,000, hard cap 200,000 (100,000 when the run has less than 512 MB of memory). The run stops cleanly when it is reached.

## `includePattern` (type: `string`):

Optional. Keep only URLs that contain this text, case-insensitive. Use \* for "anything": /blog/ keeps blog URLs, /products/\*/reviews keeps product review pages. Plain text and \* only (no regular expressions). Filtered URLs are not charged.

## `excludePattern` (type: `string`):

Optional. Drop URLs that contain this text (same rules as above), for example /tag/ or ?page=. Wins over the include pattern.

## `lastmodAfter` (type: `string`):

Optional date (YYYY-MM-DD, or a full date-time with zone). Keeps only URLs whose <lastmod> is on or after it; URLs without a lastmod are dropped while this is set. Child sitemaps whose index lastmod is older are not downloaded.

## `maxDepth` (type: `integer`):

How many levels of sitemap indexes are followed. 0 reads only the sitemaps you give or that robots.txt lists; the default 3 covers almost every site; maximum 5. Deeper child sitemaps are reported in one free depth-cap row.

## `dedupe` (type: `boolean`):

On (default): a URL listed in several sitemaps is output and charged once (the first sitemap wins). Off: one row per listing.

## `mode` (type: `string`):

Internal: "selfTest" runs the maintainer health check and saves no items.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev/"
  ],
  "maxUrls": 20,
  "maxDepth": 3,
  "dedupe": true,
  "mode": "run"
}
```

# Actor output Schema

## `results` (type: `string`):

One item per URL found in the sitemaps (url, lastmod, changefreq, priority, sourceSitemap, site), plus free error rows (JSON, CSV, Excel or XML).

## `runSummary` (type: `string`):

URL rows and charges, sitemap files read, filtered and duplicate entries, bytes downloaded, errors by code and why the run stopped.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev/"
    ],
    "maxUrls": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("ataism/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev/"],
    "maxUrls": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("ataism/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev/"
  ],
  "maxUrls": 20
}' |
apify call ataism/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ataism/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/mPMOdkfFqKzZCcd2n/builds/vMZTkPkIxa4RooZMy/openapi.json