# Sitemap URL Extractor — All Page URLs, Lastmod & Hreflang (`actor-forge/sitemap-extractor`) Actor

Get every page URL of a website from its XML sitemaps: robots.txt discovery, sitemap indexes, .gz, text and RSS sitemaps. Each URL comes with lastmod, changefreq, priority, images, videos, news and hreflang alternates. Filters, dedupe, change detection. $0.40 per 1,000 URLs.

- **URL**: https://apify.com/actor-forge/sitemap-extractor.md
- **Developed by:** [ActorForge](https://apify.com/actor-forge) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / 1,000 url results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor — All Page URLs, Lastmod & Hreflang

Get every page URL of a website from its XML sitemaps: robots.txt discovery, sitemap indexes, .gz, text and RSS sitemaps. Each URL comes with lastmod, changefreq, priority, images, videos, news and hreflang alternates. Filters, dedupe, change detection. $0.40 per 1,000 URLs.

**Get all URLs of a website from its XML sitemaps, with all the metadata.** This sitemap extractor (sitemap URL extractor / XML sitemap parser / sitemap scraper) takes a domain, a home page or a sitemap URL and returns **one row per page URL**. It finds the sitemaps for you (the **Sitemap: lines in robots.txt**, then /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and other common paths), follows **sitemap indexes** to any depth with a cycle guard, and reads **gzip (.xml.gz)**, **text (sitemap.txt)** and **RSS/Atom feeds used as sitemaps**. Each URL comes with **lastmod (ISO 8601), changefreq and priority**, plus the extensions other extractors drop: **images** (image:image loc, title, caption), **videos** (title, thumbnail, content and player URLs, duration), **Google News** (title, publication, date) and **hreflang alternates** (xhtml:link language versions). Filter by **URL pattern** and by **lastmod date** ("modified in the last 7 days"; old child sitemaps are not even downloaded), remove duplicates across sitemaps, optionally **check each URL's HTTP status** (200/301/404), and turn on **change detection** to get only new and updated URLs on scheduled runs, with removed URLs listed in the summary. The run summary shows the **sitemap tree with URL counts per sitemap, a lastmod histogram by month** and any errors. Huge sitemaps (50 MB / 50,000+ URLs per file) are streamed, not loaded whole, and malformed XML is read leniently. Use it for SEO audits, site migrations, content inventories, feeding crawlers and RAG pipelines, monitoring competitors' new pages, and hreflang checks. **$0.40 per 1,000 URLs**, with no compute bill on top (Apify's own sitemap extractor is $0.50 per 1,000).

### What data does it return?

Every run outputs clean, schema-validated rows like these:

| url | lastmod | changefreq | priority | sitemapUrl | images | videos | alternates |
| --- | --- | --- | --- | --- | --- | --- | --- |
| https://www.ikea.com/ee/et/p/zebrasaev-laelamp-valge-plast-10522481/ | 2026-10-18T22:35:41.152Z | weekly | 1 | https://www.ikea.com/sitemaps/prod-et-EE_1.xml | 2 | 0 | 0 |
| https://crawlee.dev/python/blog |  | weekly | 0.5 | https://crawlee.dev/python/sitemap.xml | 0 | 0 | 0 |
| https://wordpress.org/embed-image/ | 2022-10-04T14:46:18.000Z |  |  | https://wordpress.org/image-sitemap-1.xml | 1 | 0 | 0 |
| https://wordpress.org/6-3-hero-animation-v2-mp4-compressed/ | 2023-08-10T02:07:05.000Z |  |  | https://wordpress.org/video-sitemap-1.xml | 0 | 1 | 0 |

### How much does it cost?

Transparent pay-per-event pricing — you only pay for what you get, and the
actor **always respects your run budget**: set a maximum total charge and it
stops cleanly at the cap instead of overrunning.

| Event | Price | Charged | Worked example |
| --- | --- | --- | --- |
| `url-result` | $0.0004 | per page URL delivered (one dataset row) | A 5,000-page site = $2.00; 50,000 URLs = $20; filtered-out URLs and duplicates are free |
| `url-checked` | $0.001 | per URL whose live HTTP status is checked (only with 'Check each URL's HTTP status' on) | 1,000 URLs with status checks = $0.40 + $1.00 = $1.40 |
| `apify-actor-start` | $0.00005 | per run per GB of memory (platform start fee) | One run at 512 MB = $0.000025 |

### Input options

| Field | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `urls` | array | yes | — | Domains (example.com), home pages, or direct sitemap URLs (sitemap.xml, sitemap_index.xml, .xml.gz, sitemap.txt, RSS/Atom feeds). For a domain or page, the site's sitemaps are found automatically from robots.txt and the usual paths. A page URL stands for its whole site: use the URL filters to narrow it down. |
| `discover` | boolean | no | `true` | For domains and page URLs, read the Sitemap: lines in robots.txt, and when there are none try /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt and other common paths. Turn off to read only the exact URLs you give. |
| `includeGlobs` | array | no | `[]` | Keep only page URLs matching at least one of these patterns. `*` matches anything except `/`, `**` matches anything, e.g. `https://example.com/blog/**` or `**/products/*`. Leave empty to keep all. |
| `excludeGlobs` | array | no | `[]` | Drop page URLs matching any of these patterns, e.g. `**/tag/**` or `**?page=*`. Filtered URLs are free. |
| `lastmodAfter` | string | no | — | Only URLs whose \<lastmod> is after this point: a date like 2026-10-01, or a window like "7 days", "24 hours" or "3 months". URLs without a lastmod are left out when this is set. Child sitemaps an index marks as older are not even downloaded. |
| `lastmodBefore` | string | no | — | Only URLs whose \<lastmod> is before this point (same formats as above). URLs without a lastmod are left out when this is set. |
| `maxUrls` | integer | no | `100000` | Stop after delivering this many URLs in total. |
| `maxSitemaps` | integer | no | `500` | Read at most this many sitemap files (indexes included). Big sites split their URLs over hundreds of files. |
| `includeImages` | boolean | no | `true` | Return each URL's image:image (loc, title, caption), video:video (title, thumbnail, content and player URLs, duration, date) and news:news (title, publication, date) entries. Same price either way. |
| `includeAlternates` | boolean | no | `true` | Return each URL's xhtml:link hreflang alternates (language versions). Same price either way. |
| `dedupe` | boolean | no | `true` | Deliver each URL once, even when it is listed in several sitemaps. Duplicates are free. |
| `checkStatus` | boolean | no | `false` | Also request every delivered URL (HEAD, or GET when HEAD is refused) and return its status code: 200, 301, 404… Redirects are reported, not followed. Charged separately per URL checked, and slower: about 2 requests at a time per site. |
| `detectChanges` | boolean | no | `false` | Save this run's URL list and, next time, mark each URL new, updated (lastmod changed) or unchanged, and list URLs that disappeared in the run summary. Runs with the same URLs and filters (or the same state key) are compared. |
| `onlyChanges` | boolean | no | `false` | With change detection on: deliver only new and updated URLs. Unchanged URLs are skipped and free, so a scheduled run only pays for what changed. |
| `stateKey` | string | no | — | Optional name for the saved URL list (letters, digits, - and \_), e.g. "my-shop-weekly". Leave empty to derive it from the URLs and filters. |

### Limits & honest notes

- Reads only what the site publishes in its sitemaps: pages missing from them are not found (there is no link crawling). Sites with no sitemap at all are reported in the summary, free.
- Robots policy: sitemaps listed in the site's robots.txt Sitemap: lines are always read (the owner published them for crawlers). Any other sitemap URL, whether you give it, it is guessed at a common path or linked from an index, is skipped when robots.txt disallows it for our user agent or `*`. If robots.txt itself can't be fetched (HTTP 5xx or a network error, after one retry), RFC 9309 says to treat the site as off-limits; there is then no Sitemap: list to go by, so on that site only the sitemap URLs you give explicitly are read (no discovery, guessed paths or index children), and the reason is listed in the run's errors. Page URLs are never fetched unless you turn on status checks.
- A domain or page URL stands for its whole site (all of example.com's sitemaps); use 'Only URLs matching' to keep one section.
- Polite by design: at most 2 requests at a time per host, and 429/5xx answers are retried up to 3 times (honouring Retry-After). Status checks therefore run at roughly 10-20 URLs per second per site.
- Sitemap files up to 100 MB downloaded (300 MB uncompressed) are streamed; indexes are followed up to 10 levels deep; at most 'Max sitemap files' files per run (default 500). Truncated .gz files yield the URLs before the cut.
- lastmod values are normalised to ISO 8601 UTC; time-zone-less values are read as UTC, and placeholder or impossible dates (1970, 0000-00-00) become null.
- URLs on internal network addresses (localhost, private IPs, cloud metadata) are never requested and never charged.
- Respects your run budget: set a maximum total charge and the run stops cleanly at the cap, delivering exactly the URLs that were charged. A run restarted by the platform skips URLs it already delivered.

### Changelog

- **2026-10-19** — Initial release: robots.txt and common-path sitemap discovery, recursive sitemap indexes with cycle guard, gzip by magic bytes, text and RSS/Atom sitemaps, streaming lenient parser, lastmod/changefreq/priority, image/video/news extensions, hreflang alternates, URL and lastmod filters, dedupe, optional HTTP status checks, change detection against the previous run, sitemap tree and lastmod histogram summary.

***

*Built and maintained by **ActorForge**. Reliability first: every actor in the
portfolio is monitored by an automated nightly self-test, and breakages are
typically fixed the same day.*

# Actor input Schema

## `urls` (type: `array`):

Domains (example.com), home pages, or direct sitemap URLs (sitemap.xml, sitemap_index.xml, .xml.gz, sitemap.txt, RSS/Atom feeds). For a domain or page, the site's sitemaps are found automatically from robots.txt and the usual paths. A page URL stands for its whole site: use the URL filters to narrow it down.

## `discover` (type: `boolean`):

For domains and page URLs, read the Sitemap: lines in robots.txt, and when there are none try /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /sitemap.txt and other common paths. Turn off to read only the exact URLs you give.

## `includeGlobs` (type: `array`):

Keep only page URLs matching at least one of these patterns. `*` matches anything except `/`, `**` matches anything, e.g. `https://example.com/blog/**` or `**/products/*`. Leave empty to keep all.

## `excludeGlobs` (type: `array`):

Drop page URLs matching any of these patterns, e.g. `**/tag/**` or `**?page=*`. Filtered URLs are free.

## `lastmodAfter` (type: `string`):

Only URLs whose <lastmod> is after this point: a date like 2026-10-01, or a window like "7 days", "24 hours" or "3 months". URLs without a lastmod are left out when this is set. Child sitemaps an index marks as older are not even downloaded.

## `lastmodBefore` (type: `string`):

Only URLs whose <lastmod> is before this point (same formats as above). URLs without a lastmod are left out when this is set.

## `maxUrls` (type: `integer`):

Stop after delivering this many URLs in total.

## `maxSitemaps` (type: `integer`):

Read at most this many sitemap files (indexes included). Big sites split their URLs over hundreds of files.

## `includeImages` (type: `boolean`):

Return each URL's image:image (loc, title, caption), video:video (title, thumbnail, content and player URLs, duration, date) and news:news (title, publication, date) entries. Same price either way.

## `includeAlternates` (type: `boolean`):

Return each URL's xhtml:link hreflang alternates (language versions). Same price either way.

## `dedupe` (type: `boolean`):

Deliver each URL once, even when it is listed in several sitemaps. Duplicates are free.

## `checkStatus` (type: `boolean`):

Also request every delivered URL (HEAD, or GET when HEAD is refused) and return its status code: 200, 301, 404… Redirects are reported, not followed. Charged separately per URL checked, and slower: about 2 requests at a time per site.

## `detectChanges` (type: `boolean`):

Save this run's URL list and, next time, mark each URL new, updated (lastmod changed) or unchanged, and list URLs that disappeared in the run summary. Runs with the same URLs and filters (or the same state key) are compared.

## `onlyChanges` (type: `boolean`):

With change detection on: deliver only new and updated URLs. Unchanged URLs are skipped and free, so a scheduled run only pays for what changed.

## `stateKey` (type: `string`):

Optional name for the saved URL list (letters, digits, - and \_), e.g. "my-shop-weekly". Leave empty to derive it from the URLs and filters.

## Actor input object example

```json
{
  "urls": [
    "https://crawlee.dev"
  ],
  "discover": true,
  "includeGlobs": [],
  "excludeGlobs": [],
  "maxUrls": 100000,
  "maxSitemaps": 500,
  "includeImages": true,
  "includeAlternates": true,
  "dedupe": true,
  "checkStatus": false,
  "detectChanges": false,
  "onlyChanges": false
}
```

# Actor output Schema

## `results` (type: `string`):

One row per page URL with lastmod, changefreq, priority, images, videos, news, hreflang alternates and (optionally) HTTP status and change flag.

## `summary` (type: `string`):

Sitemap tree (index → child sitemaps) with URL counts per sitemap, lastmod histogram by month, filtered and duplicate counts, change-detection results and errors (never charged).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://crawlee.dev"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("actor-forge/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://crawlee.dev"] }

# Run the Actor and wait for it to finish
run = client.actor("actor-forge/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://crawlee.dev"
  ]
}' |
apify call actor-forge/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,actor-forge/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0JwugKN866IUiDaJF/builds/RNw4euMYapoiHfMEr/openapi.json