# Sitemap URL Extractor — All Page URLs from a Website (`power_on/sitemap-url-list`) Actor

Get every page URL from a website's sitemaps. Finds sitemaps in robots.txt and common paths, follows sitemap-index files, unpacks gzip, and reports the status of each one. Filter by URL pattern or lastmod date. One row per URL, ready to feed a crawler.

- **URL**: https://apify.com/power\_on/sitemap-url-list.md
- **Developed by:** [Power On Labs](https://apify.com/power_on) (community)
- **Categories:** Developer tools, Automation, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap URL Extractor — every page URL from a website's sitemaps

Give it a domain, a homepage, or a direct link to a `sitemap.xml`, and get back
the full list of the site's page URLs — one dataset row per URL, with the
`lastmod`, `changefreq` and `priority` the sitemap declares.

It is built to finish the job on sites where a naive sitemap reader gives up:

- **Finds the sitemap for you.** Reads `robots.txt`, then tries the common
  locations (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, …).
- **Follows sitemap-index files**, to any depth. A site that splits its sitemap
  into 40 partitions behind two levels of index still comes back whole.
- **Unpacks gzip.** `.xml.gz` sitemaps, and sitemaps served with a gzip body,
  are decompressed automatically.
- **Never fails the whole run for one bad file.** Every sitemap gets a row in
  the report saying whether it was read, and why not if it wasn't.
- **No browser.** It is HTTP plus XML parsing, so it is fast and the run is
  cheap.

***

### Output

With **one row per URL** (the default), each row is:

```json
{
  "url": "https://example.com/blog/how-sitemaps-work",
  "lastmod": "2026-08-14",
  "changefreq": "monthly",
  "priority": "0.7",
  "sourceSitemap": "https://example.com/sitemap-posts.xml",
  "website": "https://example.com"
}
```

That is exactly the shape a crawler or a screenshot tool wants as input, so the
dataset drops straight into the next step — for example
[Website Screenshot & PDF Generator](https://apify.com/power_on/screenshot-url-pdf)
or any crawler that takes a URL list.

Turn **"One row per URL" off** to get instead one row per website, with the URLs
as an array and a per-sitemap breakdown.

#### The sitemap report

Unless you turn it off, the dataset ends with one row per sitemap file found:

| website | sitemapUrl | discoveredVia | status | urlCount | note |
|---|---|---|---|---|---|
| https://example.com | /sitemap.xml | robots.txt | ok (index) | 12 | 12 new child sitemap(s) |
| https://example.com | /sitemap-posts.xml | index: /sitemap.xml | ok | 1 843 | |
| https://example.com | /sitemap-authors.xml | index: /sitemap.xml | error | 0 | HTTP 404 |

This is the row that tells you *why* a count is lower than you expected — a
sitemap the index points to that 404s, a file that isn't XML, a timeout.

### Options

| Option | What it does |
|---|---|
| **Websites or sitemap URLs** | One per line. A bare domain, a homepage, or a direct sitemap link. |
| **Max URLs per website** | Stop after N URLs for one site (0 = up to the run limit). |
| **Include only URLs matching** / **Exclude URLs matching** | JavaScript regular expressions applied to each URL. |
| **Only URLs changed on or after** | A date. Keeps only URLs whose `<lastmod>` is on or after it — the rest, and URLs with no `<lastmod>`, are dropped. |
| **One row per URL** | On: a flat URL list. Off: one row per site with an array and a per-sitemap breakdown. |
| **Max sitemap files per website** | Safety limit on how many sitemap files to fetch for one site. |
| **Proxy** | Optional, for sites that block datacenter traffic or serve a region-specific sitemap. |

### Pricing

Pay per URL returned. A website with no readable sitemap costs nothing.

**A run returns at most 25,000 URLs**, so a single run can never surprise you.
When a run stops at that limit — or at the maximum cost you allow for one run —
it finishes normally and adds a report row saying so, with the sitemap files
still left to read: point a new run straight at those, or narrow the result with
**Include only URLs matching**.

### Common uses

- **Get all URLs from a website** to seed a crawl, an archive, or a migration
  checklist.
- **Convert a `sitemap.xml` to a plain URL list** (JSON or CSV via the dataset
  export).
- **Find the pages that changed** since your last crawl, using the `lastmod`
  filter, and re-process only those.
- **Feed a screenshot or PDF Actor** one row per page.

### Limits

- It reads XML sitemaps (`<urlset>` and `<sitemapindex>`). It does not parse
  HTML sitemap pages or discover URLs by crawling links — give it the sitemap.
- `lastmod` is reported as the sitemap writes it; it is not independently
  verified against the page.
- Sites that publish no sitemap return an empty result with a report row saying
  so. Some large sites (for example `python.org`) genuinely have none at a
  standard location.

### Issues

Found a sitemap it can't read? Open an issue on the Actor's **Issues** tab with
the URL and it will be looked at.

# Actor input Schema

## `startUrls` (type: `array`):

One per line. Give a bare domain (example.com), a homepage, or a direct link to a sitemap.xml / sitemap\_index.xml / sitemap.xml.gz. For a domain, the Actor reads robots.txt and tries the common sitemap paths. One set of results is produced per entry.

## `maxUrls` (type: `integer`):

Stop after this many URLs for a single website. Set it to 0 to go up to the run limit. Every run returns at most 25,000 URLs in total; when it stops there, a report row says so and lists the sitemap files still to read, so you can run those directly. The default keeps a first run small and predictable — raise it once you know the site.

## `includeRegex` (type: `string`):

A JavaScript regular expression. When set, only URLs that match are kept, e.g. ^https://example.com/blog/ for one section.

## `excludeRegex` (type: `string`):

A JavaScript regular expression. URLs that match are dropped, e.g. .(pdf|jpg|png)$ or /tag/ .

## `lastmodAfter` (type: `string`):

A date (YYYY-MM-DD). Keeps only URLs whose <lastmod> is on or after it. URLs without a <lastmod> are dropped when this is set.

## `flatten` (type: `boolean`):

On (default): the dataset has one row per URL, ready to pipe into a crawler or the Website Screenshot Actor. Off: one row per website, with the URLs as an array and a per-sitemap breakdown.

## `includeSitemapReport` (type: `boolean`):

Add a final row per website listing every sitemap found, where it was discovered, its status and how many URLs it contributed. This is what lets you see why a count is lower than expected.

## `maxSitemaps` (type: `integer`):

Safety limit on how many sitemap files (including those linked from a sitemap index) to fetch for one website.

## `timeoutSecs` (type: `integer`):

How long to wait for a single sitemap or robots.txt file before giving up on it and recording the reason.

## `extraHeaders` (type: `object`):

Extra HTTP headers sent with every request, as a JSON object. Use it for an Authorization header or a custom User-Agent.

## `proxyConfiguration` (type: `object`):

Optional proxy, for sites that block datacenter traffic or serve a region-specific sitemap. Not needed for most sites.

## Actor input object example

```json
{
  "startUrls": [
    "https://apify.com"
  ],
  "maxUrls": 1000,
  "flatten": true,
  "includeSitemapReport": true,
  "maxSitemaps": 200,
  "timeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("power_on/sitemap-url-list").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("power_on/sitemap-url-list").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://apify.com"
  ]
}' |
apify call power_on/sitemap-url-list --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,power_on/sitemap-url-list"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3DrafOP4gJCqbzDKc/builds/3IVtU9YqmjhlVxbFC/openapi.json