# Sitemap URL Extractor — Nested Indexes, Gzip & lastmod (`relevate/sitemap-urls`) Actor

Turn any website into a clean list of its URLs. Finds the sitemap from robots.txt, follows nested sitemap indexes, unpacks .xml.gz, reads RSS/Atom feeds, and can return only pages changed since a date — the cheap way to re-crawl just what is new.

- **URL**: https://apify.com/relevate/sitemap-urls.md
- **Developed by:** [Relevate](https://apify.com/relevate) (community)
- **Categories:** Agents, SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.10 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap URL Extractor — Nested Indexes, Gzip & Incremental `lastmod`

Give it a domain. Get back every URL the site publishes, with `lastmod`, `changefreq` and `priority`, and the sitemap each URL came from.

You do not need to know where the sitemap lives. The Actor reads `robots.txt` first, and if the site declares nothing it probes the usual locations. Then it follows the whole tree: sitemap indexes pointing at other indexes, `.xml.gz` archives, plain-text sitemaps, and RSS or Atom feeds when that is all a site has.

### What it handles that simple extractors do not

| | |
|---|---|
| **Nested sitemap indexes** | Follows index → index → `urlset` to a configurable depth, instead of stopping at the first file. |
| **Gzip** | Unpacks `.xml.gz` by inspecting the bytes, so it works even when the server sends the wrong `Content-Type`. |
| **RSS / Atom** | Falls back to feeds, the only "sitemap" many blogs and news sites publish. |
| **Plain-text sitemaps** | A file of one URL per line is valid per the sitemap protocol, and is read as such. |
| **Duplicates** | Real sitemaps repeat URLs. They are deduplicated, and the number skipped is **reported**, not hidden. |
| **Incremental extraction** | `lastmodAfter` returns only pages changed since a date. |
| **Honest limits** | If a cap or a depth limit stops the run early, it says so in the log and in the `SUMMARY` record. |

### Incremental crawling — the reason to use this

Re-crawling a 50,000-page site to find the 40 pages that changed is a waste. Set `lastmodAfter` to your last run's date and you get just the new and updated URLs, which you can feed straight into a content crawler:

```json
{
  "startUrls": ["example.com"],
  "lastmodAfter": "2026-07-01"
}
```

URLs with no `lastmod` are dropped when this filter is set — if a site publishes no dates, there is nothing to compare, and silently returning everything would defeat the point.

### Input

```json
{
  "startUrls": ["apify.com"],
  "lastmodAfter": "2026-07-01",
  "includePattern": "/blog/",
  "excludePattern": "/tag/|/author/",
  "maxUrls": 10000,
  "maxDepth": 3,
  "outputSitemapList": false,
  "minExpectedResults": 0
}
```

- **`startUrls`** — a bare domain (`example.com`), a full sitemap URL, a sitemap index, an `.xml.gz` file, or a feed. Mix them freely in one run.
- **`includePattern` / `excludePattern`** — case-insensitive regular expressions over the full URL. An invalid expression fails the run immediately with the reason, rather than silently matching nothing.
- **`outputSitemapList`** — return the site's sitemap *structure* instead of its URLs. Useful on a large site before you extract from it.
- **`minExpectedResults`** — fail the run if fewer records come back than expected. See *Scheduling* below.

### Output

One record per URL:

```json
{
  "url": "https://apify.com/store/apify/website-content-crawler",
  "lastmod": "2026-07-14T00:00:00.000Z",
  "lastmodRaw": "2026-07-14",
  "changefreq": "weekly",
  "priority": 0.8,
  "site": "https://apify.com",
  "sourceSitemap": "https://apify.com/sitemap/actors1.xml",
  "sitemapKind": "urlset",
  "depth": 1
}
```

With `outputSitemapList: true` you get one record per sitemap instead — its `kind`, how it was discovered, and how many child sitemaps and URLs it holds.

Every run also writes a `SUMMARY` record to the key-value store: sitemaps read, sitemaps failed, records pushed, URLs filtered, duplicates skipped, whether the cap was hit, and the full sitemap map.

### Typical uses

- **Feed a content crawler** a precise URL list instead of letting it discover links, so you crawl what you meant to and nothing else.
- **Incremental re-crawls** — only pages changed since your last run.
- **SEO audits** — count indexable pages, find sections missing from the sitemap, spot duplicate entries and missing `lastmod`.
- **Site migration checks** — extract the URL list before and after a move and diff the two.
- **Agent tooling** — a fast, cheap way for an AI agent to see what a website actually contains before deciding what to read.

### Scheduling and health checks

Set `minExpectedResults` to roughly what you expect. If a sitemap moves, gets renamed or breaks, the run **fails loudly** instead of quietly returning nothing — which is what you want from a monitor. Pair it with a Schedule and Apify's failure notifications.

### Limits

- Sitemap files above 60 MB uncompressed are skipped with an explicit error, which is well beyond the 50 MB the sitemap protocol allows.
- `maxDepth` defaults to 3 levels of nesting; deeper indexes are reported as not followed rather than dropped silently.
- Only `robots.txt` and the sitemaps themselves are fetched. The Actor never renders pages, so it needs no browser, no proxy and no captcha solving.

### Pricing

Pay per event: a small charge to start the run, then a per-URL charge. You pay for URLs returned, not for compute time.

# Actor input Schema

## `startUrls` (type: `array`):

Give a plain domain such as example.com and the Actor finds the sitemap itself — first from robots.txt, then by probing the usual paths. Or pass a full URL to a sitemap, a sitemap index, an .xml.gz file or an RSS/Atom feed.

## `lastmodAfter` (type: `string`):

Keep only URLs whose <lastmod> is newer than this date, e.g. 2026-07-01. This turns a full sitemap into an incremental feed of what changed — the cheap way to re-crawl only new pages. URLs with no <lastmod> are dropped when this is set.

## `includePattern` (type: `string`):

Case-insensitive regular expression. Only matching URLs are kept — for example /blog/ or \\.pdf$

## `excludePattern` (type: `string`):

Case-insensitive regular expression. Matching URLs are dropped — for example /tag/|/author/

## `maxUrls` (type: `integer`):

Safety cap on how many URLs are returned in one run.

## `maxDepth` (type: `integer`):

How deep to follow sitemap indexes that point at other sitemap indexes. 3 covers essentially every real site.

## `outputSitemapList` (type: `boolean`):

Output one row per sitemap found — its type, how it was discovered, how many child sitemaps and URLs it holds — instead of the URLs themselves. Use it to understand a large site's sitemap structure before extracting from it.

## `minExpectedResults` (type: `integer`):

Fail the run if fewer than this many records are produced. Use it in scheduled pipelines to get alerted when a sitemap moves or changes shape. 0 disables the check.

## Actor input object example

```json
{
  "startUrls": [
    "apify.com"
  ],
  "maxUrls": 10000,
  "maxDepth": 3,
  "outputSitemapList": false,
  "minExpectedResults": 0
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("relevate/sitemap-urls").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("relevate/sitemap-urls").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "apify.com"
  ]
}' |
apify call relevate/sitemap-urls --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,relevate/sitemap-urls"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/2sTRytdEBagVzTaRu/builds/3GIJkzVAl5luDzeEX/openapi.json