# Sitemap URL Extractor — every URL, $0.0002/URL (`plainapi/sitemap-xml-url-list-extractor`) Actor

You pay only for URLs actually returned. Nothing found, nothing charged. List every URL a site publishes, from robots.txt and its sitemaps, with last-modified dates. Expands sitemap indexes, honours robots.txt, never crawls or guesses.

- **URL**: https://apify.com/plainapi/sitemap-xml-url-list-extractor.md
- **Developed by:** [Shun Furu](https://apify.com/plainapi) (community)
- **Categories:** SEO tools, AI, MCP servers
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap URL Extractor — every page a site declares

List every URL a site publishes, taken from **robots.txt and its sitemaps**, with the
last-modified date each page declares.

```
Input:   ["https://apify.com"]
Output:  one row per URL — address, last modified, change frequency, which sitemap declared it
```

### Why read a sitemap rather than crawl

`robots.txt` and `sitemap.xml` exist so that a machine can ask a site what it publishes. Reading
them is the most explicitly invited request on the web — and it is also faster, complete, and
carries information a crawler cannot infer:

- **`lastmod`** — when the page last changed, so you can re-fetch only what moved.
- **Completeness** — pages no link points at are still declared.
- **Speed** — one request per sitemap instead of one per page.

This Actor **never crawls, follows links, or guesses at addresses**. It reports the declaration and
nothing else, and it skips anything `robots.txt` disallows even though only the declaration is
being read.

### What it handles

- **Sitemap indexes.** Large sites split their sitemap into an index of further sitemaps; those are
  expanded automatically.
- **Sitemaps declared in robots.txt**, which is where they are supposed to be announced — and the
  usual locations when a site announces nothing.
- **Language alternates.** `hreflang` versions of the same page, on request.
- **Pages that are not a sitemap.** A site that answers `200` with an error page produces no rows
  rather than nonsense.

### Input

| Field | Type | Default | What it does |
|---|---|---|---|
| `sites` | array of strings | one example | Site URLs, or sitemap URLs directly |
| `pathContains` | string | — | Keep only URLs containing any of these words, e.g. `/blog, /docs` |
| `maxUrls` | integer | `0` (no limit) | Cap URLs per site |
| `followIndexes` | boolean | `true` | Expand sitemap indexes |
| `includeAlternates` | boolean | `false` | Include `hreflang` alternates |

#### Examples

Every documentation page on a site:

```json
{ "sites": ["https://docs.apify.com"], "pathContains": "/docs" }
```

Several sites at once, capped:

```json
{ "sites": ["https://blog.rust-lang.org", "https://github.blog"], "maxUrls": 500 }
```

### Output

| Field | Type | Description |
|---|---|---|
| `site` | string | Origin the URL belongs to |
| `url` | string | The declared URL |
| `sitemap` | string | Which sitemap declared it — useful on sites with many |
| `lastModified` | string | null | As declared |
| `changeFrequency` | string | As declared |
| `priority` | string | As declared |
| `alternates` | array | `hreflang` versions, when requested |
| `fetchedAt` | string | When the row was produced |

The `SUMMARY` record adds a breakdown per site: how many sitemaps were read, how many were
indexes, whether robots.txt declared them, and a count of URLs by first path segment — the shape
of the site at a glance.

### Pricing

| | Price |
|---|---|
| Starting a run | **$0.00001** — one US cent per 1,000 runs |
| Each URL returned | **$0.0002** — 20 US cents per 1,000 URLs |

A site with no sitemap produces no rows, and errors go to the log and `SUMMARY` rather than the
dataset — so a site that cannot be read costs nothing.

### Works well with

Pair it with a text extractor: this decides **which pages exist and which ones changed**, that one
reads them. Filter by `lastModified` and you re-ingest only what moved since your last run.

### Notes on data and compliance

- Only `robots.txt` and the sitemaps it points to are fetched.
- Paths disallowed by `robots.txt` are skipped, including sitemaps themselves.
- No page content is downloaded, so no personal data is touched.
- The Actor stops after 200 sitemaps per site rather than following an index loop indefinitely.

### Limitations — stated up front

- **A site with no sitemap returns nothing.** That is the honest answer; this will not fall back to
  crawling.
- Gzipped sitemaps (`.xml.gz`) are requested but not decompressed yet.
- `lastmod` is whatever the site declares, which is not always accurate — some CMSs stamp every
  page with the deploy time.
- Very large sites are capped at 200 sitemaps per run; use `pathContains` to narrow instead.

# Actor input Schema

## `sites` (type: `array`):

Site URLs. robots.txt is read to find the sitemaps a site declares; a sitemap URL can also be given directly.

## `pathContains` (type: `string`):

Keep only URLs containing any of these words, e.g. "/blog, /docs".

## `maxUrls` (type: `integer`):

0 means no limit.

## `followIndexes` (type: `boolean`):

Large sites split their sitemap into an index of further sitemaps. Leave on to expand them.

## `includeAlternates` (type: `boolean`):

Include hreflang alternates a sitemap declares for each page.

## Actor input object example

```json
{
  "sites": [
    "https://apify.com",
    "https://blog.rust-lang.org"
  ],
  "maxUrls": 0,
  "followIndexes": true,
  "includeAlternates": false
}
```

# Actor output Schema

## `urls` (type: `string`):

Every URL declared, with its last-modified date.

## `urlsCsv` (type: `string`):

The same rows as CSV.

## `summary` (type: `string`):

Sitemaps read per site, and the URL count by section.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "https://apify.com",
        "https://blog.rust-lang.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("plainapi/sitemap-xml-url-list-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sites": [
        "https://apify.com",
        "https://blog.rust-lang.org",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("plainapi/sitemap-xml-url-list-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "https://apify.com",
    "https://blog.rust-lang.org"
  ]
}' |
apify call plainapi/sitemap-xml-url-list-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,plainapi/sitemap-xml-url-list-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/QubeOjikGMGm01BQ1/builds/8b7tpJ0djhZXOMVYh/openapi.json