# Sitemap URL Extractor: the full URL list of any site (`bepolabs/sitemap-urls`) Actor

Enter a hostname, get one row per URL with lastmod, priority and changefreq. Reads gzip too. Pay 0.005 USD per 1,000 URLs, nothing for duplicates.

- **URL**: https://apify.com/bepolabs/sitemap-urls.md
- **Developed by:** [BEPO Labs](https://apify.com/bepolabs) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.50 / 1,000 listed urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### 🗺️ What is Sitemap URL Extractor?

This extractor takes a domain, finds every sitemap, opens each nested one, unzips archives and parses text. You get one clean set of URLs the owner published; only those documents are fetched, never the content itself, so a huge job costs little.

- **SEO audits**: get the full list of indexable URLs to check status codes, titles, canonicals or internal links
- **Crawl planning**: hand the exact URL list to a scraper instead of walking the whole site
- **Content monitoring**: schedule a daily run with `modifiedSince` and see which pages a competitor added or changed
- **Site migrations**: export every old URL before a relaunch and build the redirect map from it
- **AI and RAG pipelines**: pick the docs or blog URLs of a site and pass them to a Markdown converter for your LLM

![Sitemap URL Extractor: Every URL of a website from its sitemaps, in one deduplicated list with lastmod, changefreq and priority.](https://api.apify.com/v2/key-value-stores/wdeAWGooHdIhvFgSY/records/sitemap-urls-hero.png)

#### ✨ Why use this Actor?

- ✅ **A domain is enough**: the Actor reads the `Sitemap:` lines of robots.txt, else tries `/sitemap.xml` and `/sitemap_index.xml`
- ✅ **Every sitemap format**: sitemap indexes to any depth, gzip files, plain-text sitemaps, namespaced tags, CDATA and entities
- ✅ **No duplicates**: a URL in several sitemaps is saved once, keeps its latest lastmod and records how often it appeared
- ✅ **Filters built in**: include or exclude URLs by regular expression, keep only URLs changed since a date, same host only
- ✅ **Polite by design**: one request per sitemap file, its own user agent, and sitemaps the site's robots.txt disallows are not fetched
- ✅ **Very cheap**: 0.005 USD per 1,000 URLs; duplicates, filtered URLs and failed sitemaps cost nothing

### 📦 What data can you get with Sitemap URL Extractor?

| | | |
|---|---|---|
| 🔗 URL | 📅 Last modified (lastmod, ISO) | 🔁 Change frequency |
| ⭐ Priority | 🗺️ Sitemap it was found in | 🌐 Host |
| 👯 Duplicate count | 📋 Run summary: every sitemap read, its type and URL count | ⚠️ Failed sitemaps with the reason |

> Sites publish lastmod, changefreq and priority as they like: when a site leaves one out, the field is null. The Actor never guesses a date.

### 🚀 How to use Sitemap URL Extractor

Sitemap URL Extractor works in a few clicks, no coding needed:

1. [Create a free Apify account](https://console.apify.com/sign-up) (the free plan includes 5 USD of monthly credit).
2. Open [Sitemap URL Extractor](https://apify.com/bepolabs/sitemap-urls) and click **Try for free**.
3. Enter one or more domains, sitemap URLs or robots.txt URLs. Optionally set a URL pattern, a date in `modifiedSince` or a URL limit.
4. Click **Start** and wait for the run to finish (most runs take seconds).
5. Download your data as JSON, CSV, Excel, XML or HTML, or send it straight to Google Sheets, Make, Zapier or your AI agent.

![How Sitemap URL Extractor works](https://api.apify.com/v2/key-value-stores/wdeAWGooHdIhvFgSY/records/sitemap-urls-workflow.png)

### ⬇️ Input

Add up to 100 domains, pages, sitemap URLs or robots.txt URLs. Everything else is optional.

##### Which start?

A **domain** or any page of the site (`example.com`) starts from the standard crawler rules. A **sitemap URL** (index, `.xml`, `.xml.gz` or `.txt`) is read directly. A **robots.txt URL** reads every sitemap it lists.

##### Only some URLs

`includePattern` and `excludePattern` are case-insensitive regular expressions, for example `/blog/` or `\?|/tag/`. `modifiedSince` (YYYY-MM-DD) keeps only URLs with a lastmod on or after that day.

##### Big sites

The run stops at `maxUrls` unique URLs (10,000 unless changed, up to 1,000,000) or after `maxSitemaps` sitemap files (500 by default). The run summary says when a limit was hit.

| Field | Type | What it does | Default |
|---|---|---|---|
| **Domains, sitemaps or robots.txt** (`startUrls`) | array | A domain (example.com) or any page of the site: its robots.txt Sitemap lines are read, else /sitemap.xml and /sitemap_index.xml are tried. | `["apify.com"]` |
| **Max URLs** (`maxUrls`) | integer | Stop after this many unique URLs. | `10000` |
| **Only URLs matching** (`includePattern`) | string | Regular expression, case-insensitive, e.g. |  |
| **Skip URLs matching** (`excludePattern`) | string | Regular expression, case-insensitive, e.g. |  |
| **Modified since** (`modifiedSince`) | string | Only URLs whose lastmod is on or after this date (YYYY-MM-DD). |  |
| **Same site only** (`sameHostOnly`) | boolean | Drop URLs on other hosts than the start or its sitemap (www. | `true` |
| **Max sitemaps** (`maxSitemaps`) | integer | Stop after reading this many sitemap files (indexes included). | `500` |
| **Retries** (`maxRetries`) | integer | Retries per sitemap on timeouts, 429 and 5xx. | `2` |

![Sitemap URL Extractor input form](https://api.apify.com/v2/key-value-stores/wdeAWGooHdIhvFgSY/records/sitemap-urls-input.png)

#### Input example

```json
{
    "startUrls": [
        "https://developer.mozilla.org/sitemaps/en-us/sitemap.xml.gz"
    ],
    "modifiedSince": "2026-01-01",
    "maxUrls": 300
}
```

### ⬆️ Output

Every unique URL becomes one row. The run summary in the key-value store (`OUTPUT`) lists every sitemap that was read, with its type (index, urlset or text), the URLs it held and the error if it failed, plus how many URLs were filtered out.

![Sitemap URL Extractor output table](https://api.apify.com/v2/key-value-stores/wdeAWGooHdIhvFgSY/records/sitemap-urls-output.png)

#### Output example

One row of the dataset (local run on MDN's gzip sitemap, 2026-10-16):

```json
{
    "url": "https://developer.mozilla.org/en-US/docs/Games/Anatomy",
    "lastmod": "2026-08-21T00:00:00.000Z",
    "changefreq": null,
    "priority": null,
    "sitemap": "https://developer.mozilla.org/sitemaps/en-us/sitemap.xml.gz",
    "host": "developer.mozilla.org",
    "duplicates": 0
}
```

#### Run summary (OUTPUT)

The key-value store record `OUTPUT` of a real run on `nodejs.org/robots.txt` with `includePattern` `/learn/` (local run, 2026-10-16): two sitemaps found through robots.txt, 89 URLs kept, 1,657 filtered out.

```json
{
    "ok": true,
    "rows": 89,
    "sitemapsRead": 2,
    "sitemapsFailed": 0,
    "filtered": 1657,
    "truncated": false,
    "withLastmod": 89,
    "problems": [],
    "sitemaps": [
        {
            "sitemap": "https://nodejs.org/sitemap.xml",
            "from": "https://nodejs.org/robots.txt",
            "type": "urlset",
            "urls": 0,
            "error": null
        },
        {
            "sitemap": "https://nodejs.org/learn/sitemap.xml",
            "from": "https://nodejs.org/robots.txt",
            "type": "urlset",
            "urls": 89,
            "error": null
        }
    ]
}
```

View the results as a table or JSON in the Output tab and download them as JSON, CSV, Excel, XML, HTML or RSS, all fields or only those you pick.

### 💰 How much does Sitemap URL Extractor cost?

Sitemap URL Extractor is priced **per event**: **0.005 USD per 1,000 URLs** (0.000005 USD per URL), with no monthly rental. Each unique URL saved to the dataset is billed; duplicates, filtered URLs and sitemaps that fail cost nothing.

| You get | You pay |
|---|---|
| 100 URLs | 0.0005 USD |
| 1,000 URLs | 0.005 USD |
| 10,000 URLs | 0.05 USD |
| 100,000 URLs | 0.5 USD |

💡 Apify's free plan gives you 5 USD of credit every month, enough for about **1,000,000 URLs** at no cost. Set a spending limit per run and the Actor stops there.

### ⚖️ Sitemap URL Extractor vs. the alternatives

| | Sitemap URL Extractor | Online sitemap viewers | Full-site crawlers |
|---|---|---|---|
| **Finds the sitemaps from a domain** | ✅ robots.txt + common paths | often needs the sitemap URL | ✅ |
| **Sitemap indexes and .xml.gz** | ✅ | sometimes | ✅ |
| **lastmod, changefreq, priority** | ✅ | ✅ | rarely |
| **Changed-since and pattern filters** | ✅ | ❌ | some |
| **Speed on a 100,000-page site** | ✅ reads only the sitemaps | page limits | visits every page |
| **API, schedules and MCP for AI agents** | ✅ | ❌ | ✅ |

### 🔌 Integrations, API and MCP

#### Connect Sitemap URL Extractor to your AI agent (MCP)

Give Claude, ChatGPT, Cursor or any MCP client direct access to Sitemap URL Extractor through the [Apify MCP server](https://mcp.apify.com). Add this URL to its settings:

```text
https://mcp.apify.com/?actors=bepolabs/sitemap-urls
```

For Claude Desktop, add it under `mcpServers` in `claude_desktop_config.json`:

```json
{
    "mcpServers": {
        "sitemap-urls": {
            "command": "npx",
            "args": [
                "-y",
                "@apify/actors-mcp-server",
                "--actors",
                "bepolabs/sitemap-urls"
            ],
            "env": {
                "APIFY_TOKEN": "<your Apify API token>"
            }
        }
    }
}
```

Then ask: *"List every blog URL in the sitemap of example.com that changed since the first of this month."*

#### Run it from your code

Python (`pip install apify-client`):

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("bepolabs/sitemap-urls").call(run_input={"startUrls":["https://developer.mozilla.org/sitemaps/en-us/sitemap.xml.gz"],"modifiedSince":"2026-01-01","maxUrls":300})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

JavaScript (`npm install apify-client`):

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('bepolabs/sitemap-urls').call({"startUrls":["https://developer.mozilla.org/sitemaps/en-us/sitemap.xml.gz"],"modifiedSince":"2026-01-01","maxUrls":300});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

#### No-code integrations

Send results to **Google Sheets, Slack, Airtable, HubSpot, Make, Zapier, n8n** and more from the Integrations tab, start runs on a **schedule**, or trigger your own endpoint with a **webhook** when a run finishes. The [Apify API](https://docs.apify.com/api/v2) gives you full control over runs and datasets.

### ❓ FAQ

##### The site has no sitemap. What do I get?

An empty list, which costs nothing, and a run summary in `OUTPUT` naming each sitemap tried (robots.txt lines, `/sitemap.xml`, `/sitemap_index.xml`) and why it failed. The run fails only when no sitemap could be read at all.

##### Why are some lastmod values null?

The site did not publish a lastmod for that URL. The Actor never invents a date.

##### How do I get only new or changed pages?

Use `modifiedSince` (YYYY-MM-DD). URLs without a lastmod are left out then. Run it on a schedule to watch a site for changes.

##### Does it visit the pages?

No. It reads only robots.txt and the sitemap files, one request per file. To get the text of the pages, pass the URLs to Website to Markdown.

##### The run stopped at 10,000 URLs.

That is the default `maxUrls`. Increase it (up to 1,000,000), or `maxSitemaps` for sites with many sitemap files; the summary says when a limit was hit.

##### Does it keep URLs on other domains?

By default only URLs on the same host as the start address or its sitemap (www. ignored). Switch off `sameHostOnly` to keep all of them.

##### Is it legal to use Sitemap URL Extractor?

Sitemaps and robots.txt are published on purpose so that crawlers can find a website's pages; reading them is exactly what they are for. The Actor fetches only these documents, one call each, with its own user agent, and skips any that the rules disallow. You choose the sites and are responsible for how the list is used.

##### What if something is missing or does not work?

Open an issue on the **Issues** tab; we read each one and reply. Feature requests are welcome too.

### 💬 Your feedback

Tell us what Sitemap URL Extractor should add on the **Issues** tab or leave a review; your notes decide what we work on next.

### 🔗 Related Actors

| Actor | What it does |
|---|---|
| 🔗 [Broken Link Checker](https://apify.com/bepolabs/link-checker) | Every link and image on your pages checked: HTTP status, each redirect hop, final URL and response time. |
| 🩺 [SEO Audit](https://apify.com/bepolabs/seo-audit) | Any page to a 0 to 100 SEO score and grade, with the fixes that matter most first: on-page, links and Core Web Vitals. |
| 🕰️ [Wayback Machine Scraper](https://apify.com/bepolabs/wayback-snapshots) | The archive history of any URL or domain: first and last seen, every capture, and the page text of old versions. |
| ⚡ [Bulk PageSpeed Insights](https://apify.com/bepolabs/pagespeed) | Lighthouse scores, lab metrics and real-user Core Web Vitals for any list of URLs, on mobile, desktop or both. |
| 📝 [Website to Markdown](https://apify.com/bepolabs/web-to-markdown) | Any web page or docs site to clean Markdown for LLMs, RAG and AI agents, with headings, links and ready-made chunks. |
| 📸 [Website Screenshot](https://apify.com/bepolabs/website-screenshot) | Full-page PNG, JPEG or PDF of any URL, on desktop, tablet and phone, with cookie banners closed. |
| 🌐 [Domain RDAP and DNS](https://apify.com/bepolabs/domain-rdap) | Domains to registrar, creation and expiry dates, age, status, nameservers, DNSSEC and DNS records from official RDAP, in bulk. |
| 📈 [Google Trends Scraper](https://apify.com/bepolabs/google-trends) | Interest over time for up to five keywords on one 0 to 100 scale, any country or worldwide, plus top and rising related queries. |

# Changelog

This Actor's version history is a separate document: https://apify.com/bepolabs/sitemap-urls/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

A domain (example.com) or any page of the site: its robots.txt Sitemap lines are read, else /sitemap.xml and /sitemap_index.xml are tried. Or a sitemap URL directly (index, .xml.gz or plain text), or a robots.txt URL. Up to 100.

## `maxUrls` (type: `integer`):

Stop after this many unique URLs.

## `includePattern` (type: `string`):

Regular expression, case-insensitive, e.g. /blog/

## `excludePattern` (type: `string`):

Regular expression, case-insensitive, e.g. ?|/tag/

## `modifiedSince` (type: `string`):

Only URLs whose lastmod is on or after this date (YYYY-MM-DD). URLs without lastmod are left out when set.

## `sameHostOnly` (type: `boolean`):

Drop URLs on other hosts than the start or its sitemap (www. ignored).

## `maxSitemaps` (type: `integer`):

Stop after reading this many sitemap files (indexes included).

## `maxRetries` (type: `integer`):

Retries per sitemap on timeouts, 429 and 5xx.

## Actor input object example

```json
{
  "startUrls": [
    "apify.com"
  ],
  "maxUrls": 10000,
  "sameHostOnly": true,
  "maxSitemaps": 500,
  "maxRetries": 2
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset items, one row per result, in the overview table view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("bepolabs/sitemap-urls").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("bepolabs/sitemap-urls").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "apify.com"
  ]
}' |
apify call bepolabs/sitemap-urls --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,bepolabs/sitemap-urls"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/CT3pFhNNLvVm4ehNF/builds/6v2MCZTSUJxKM78yC/openapi.json