# Sitemap URL Extractor (`lost.fany/sitemap-url-extractor`) Actor

Extract every URL from a website's sitemap.xml, sitemap index and robots.txt in seconds — with lastmod, changefreq, priority and hreflang. No crawling, no browser: fast, cheap, reliable.

- **URL**: https://apify.com/lost.fany/sitemap-url-extractor.md
- **Developed by:** [Natalia Ivanova](https://apify.com/lost.fany) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap URL Extractor

Get **every URL of a website in seconds** — straight from its `sitemap.xml`, sitemap index files and `robots.txt`. No crawling, no browser, no guessing: the Actor reads what the site itself publishes for search engines and returns a clean, deduplicated list with `lastmod`, `changefreq`, `priority`, hreflang alternates and image counts.

Typical run: a 20,000-page site in under a minute for a few cents.

### What you can do with it

- **SEO audits and content inventories** — list all indexable pages, spot stale sections by `lastmod`, compare against your crawl.
- **Site migrations** — export the old URL list, build redirects, verify nothing is lost.
- **Change monitoring** — run it daily with `modifiedAfter` to get only pages updated since yesterday.
- **Feeding other scrapers and AI agents** — produce the exact list of pages to visit instead of discovering links; call it from Claude, Cursor or any MCP client through the Apify MCP server.
- **Competitor research** — count pages per section (`/blog/`, `/product/`) with `includePattern`.
- **International SEO** — check hreflang coverage from the sitemap's `xhtml:link` alternates.

### How it works

1. You give it one or more start URLs: a homepage, a `sitemap.xml`, a sitemap index or a `sitemap.txt`.
2. For a homepage it reads `/robots.txt` (`Sitemap:` lines) and falls back to common paths (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, …).
3. It opens sitemap index files recursively, decompresses `.gz`, parses XML, text and RSS/Atom sitemaps, applies your filters and writes each URL to the dataset.

It never fetches the pages themselves, so it is fast, cheap and gentle to the target site.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array | — | Homepage, sitemap or sitemap index URLs. |
| `maxUrls` | integer | 50000 | Stop after this many unique URLs. |
| `includePattern` | regex | — | Keep only matching URLs, e.g. `^https://example.com/blog/`. |
| `excludePattern` | regex | — | Drop matching URLs, e.g. `\?page=\|/tag/`. |
| `modifiedAfter` | date | — | Keep only URLs with `lastmod` on or after `YYYY-MM-DD` (URLs without `lastmod` are kept). |
| `discoverFromRobots` | boolean | true | Discover sitemaps via `robots.txt` and common paths. |
| `followSitemapIndexes` | boolean | true | Open child sitemaps listed in index files. |
| `includeMetadata` | boolean | true | Add `lastmod`, `changefreq`, `priority`, `alternates`, `images`. |
| `maxSitemaps` | integer | 500 | Safety cap on sitemap files downloaded. |
| `requestTimeoutSecs` | integer | 30 | Per-request timeout. |
| `proxyConfiguration` | object | off | Apify Proxy, only if the site blocks datacenter requests. |

Example:

```json
{
  "startUrls": [{ "url": "https://example.com" }],
  "includePattern": "^https://example.com/blog/",
  "modifiedAfter": "2026-01-01",
  "maxUrls": 10000
}
```

### Output

One dataset item per URL:

```json
{
  "url": "https://example.com/blog/hello-world",
  "lastmod": "2026-09-01",
  "changefreq": "weekly",
  "priority": 0.8,
  "alternates": [{ "hreflang": "pt", "href": "https://example.com/pt/blog/hello-world" }],
  "images": 2,
  "sitemapUrl": "https://example.com/sitemap-posts.xml",
  "host": "example.com",
  "depth": 1
}
```

Download it as JSON, CSV, Excel or XML from the run, or read it through the API. A run summary (sitemaps fetched, URLs found, discovery method, errors) is stored in the `OUTPUT` record of the key-value store.

### Pricing

Pay per event: you are charged per extracted URL, nothing per page visited and nothing for failed sitemaps. Set **Max total charge** on the run to cap spending; the Actor stops cleanly when the cap is reached.

### Use it from code and from AI agents

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("<ACTOR_ID>").call(run_input={"startUrls": [{"url": "https://example.com"}]})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item.get("lastmod"))
```

Through the **Apify MCP server** any agent can call this Actor as a tool: ask it to "list all pages of example.com modified this month" and it will run the Actor and read the dataset.

### Limits and notes

- Only URLs listed in sitemaps are returned. If a site has no sitemap the Actor reports `sitemap not found` in `OUTPUT.errors`.
- Sitemap indexes are followed up to five levels deep; `maxSitemaps` protects you from runaway sites.
- Relative `<loc>` values are resolved against the sitemap URL.
- Duplicates are removed across all sitemaps of a run.

### FAQ

**Does it crawl the website?** No. It reads only sitemap files, which is why it is fast and costs a fraction of a crawl.

**The site has millions of URLs — will it work?** Yes; raise `maxUrls` and `maxSitemaps`, and cap spending with Max total charge.

**Can I get only new pages?** Use `modifiedAfter`. Pages without `lastmod` are kept so you never miss content silently.

**Is this legal?** Sitemaps are public files intended for machines. The Actor sends a descriptive user agent and downloads only sitemap files.

# Actor input Schema

## `startUrls` (type: `array`):

Homepage, sitemap.xml or sitemap index URLs. For a homepage the Actor discovers sitemaps from robots.txt and common paths.

## `maxUrls` (type: `integer`):

Stop after this many unique page URLs.

## `includePattern` (type: `string`):

Keep only URLs matching this regular expression, e.g. ^https://example.com/blog/

## `excludePattern` (type: `string`):

Drop URLs matching this regular expression, e.g. ?page=|/tag/

## `modifiedAfter` (type: `string`):

Keep only URLs with lastmod on or after this date (YYYY-MM-DD). URLs without lastmod are kept.

## `discoverFromRobots` (type: `boolean`):

When a start URL is not a sitemap, read /robots.txt Sitemap: lines and try /sitemap.xml, /sitemap\_index.xml and similar paths.

## `followSitemapIndexes` (type: `boolean`):

Recursively open child sitemaps listed in sitemap index files.

## `includeMetadata` (type: `boolean`):

Add lastmod, changefreq, priority, hreflang alternates and image count to every URL.

## `maxSitemaps` (type: `integer`):

Safety cap on how many sitemap files to download.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each sitemap or robots.txt download, in seconds.

## `proxyConfiguration` (type: `object`):

Usually not needed: sitemaps are public. Enable Apify Proxy only if a site blocks datacenter requests.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxUrls": 50000,
  "discoverFromRobots": true,
  "followSitemapIndexes": true,
  "includeMetadata": true,
  "maxSitemaps": 500,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `urls` (type: `string`):

One item per URL with lastmod, changefreq, priority, hreflang alternates, image count and the sitemap it came from.

## `summary` (type: `string`):

Sitemaps fetched and failed, URLs found and emitted, discovery method per start URL, errors.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("lost.fany/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://crawlee.dev" }] }

# Run the Actor and wait for it to finish
run = client.actor("lost.fany/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ]
}' |
apify call lost.fany/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,lost.fany/sitemap-url-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3VaRxlh8bnltaFfjU/builds/vnq1BQteRqRrwGpK3/openapi.json