# Sitemap Extractor API: All URLs, Lastmod, Sitemap Index (`actose/sitemap-extractor`) Actor

Every URL of any website from its sitemaps: lastmod, changefreq, priority, hreflang, images, news. Finds sitemaps via robots.txt, follows nested indexes, .gz, text and RSS. Only-new mode for monitoring. No login. MCP-ready for AI agents. Errors and empty results are free. $0.20 per 1,000 URLs.

- **URL**: https://apify.com/actose/sitemap-extractor.md
- **Developed by:** [ABM](https://apify.com/actose) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.17 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor: every URL of a website with lastmod

**$0.20 per 1,000 URLs.** Pay only for the URLs you get. Errors, missing sitemaps, duplicates and filtered URLs are free. No monthly fee, no API key, no login.

Give it a domain (`example.com`), any page of a site, a `robots.txt` or a sitemap URL. The Sitemap Extractor finds the sitemaps, follows nested sitemap indexes and returns **one row per page URL** with:

- `lastmod`, `changefreq`, `priority`
- `alternates` (hreflang language versions), `imageUrls`, `videoTitles`
- `newsTitle` and `newsPublicationDate` (Google News sitemaps), `title` (RSS/Atom feeds)
- `sitemapUrl` (the file where the URL was found), `domain`, `path`, `inputUrl`

What it handles that simple sitemap parsers miss:

- **Discovery**: `Sitemap:` lines of robots.txt, then `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml`, `/sitemap.txt`, with and without `www.`
- **Nested sitemap indexes** (up to 10 levels), **.xml.gz** files (even when the server gzips them twice), **plain-text** sitemaps, **RSS and Atom** feeds used as sitemaps
- **Huge sitemaps**: files are read as a stream, so a 50,000-URL / 50 MB sitemap uses little memory, and the download stops as soon as your limit is reached
- **Clear errors per input** in the run summary: "no sitemap found", "domain does not resolve", "HTTP 404", "anti-bot page", so you know exactly why a site gave nothing

### How much does it cost?

One event: **`url` = $0.0002** (one unique URL delivered).

| URLs delivered | Price |
|---|---|
| 100 | $0.02 |
| 1,000 | $0.20 |
| 50,000 (a full large sitemap) | $10 |
| 10 sites × 5,000 URLs, every week | $10 per week, or ~$1 with `onlyNew` once the first run is done |

Not billed: duplicates across sitemaps, URLs removed by your filters, URLs already delivered in a previous run (`onlyNew`), sites with no sitemap, unreachable or blocked sites. Set **Max URLs** or a maximum cost per run in Apify: the actor stops cleanly at your limit.

### Input

```json
{
  "urls": ["https://www.gov.uk/sitemap.xml", "crawlee.dev"],
  "maxUrls": 100
}
```

| Field | What it does |
|---|---|
| `urls` | Domains, page URLs, robots.txt URLs or sitemap URLs (XML, .gz, index, text, RSS/Atom) |
| `maxUrls` | Total limit (default 10,000; 0 = no limit) |
| `maxUrlsPerInput` | Limit per domain or sitemap |
| `includePatterns` / `excludePatterns` | Regular expressions or plain text matched on the URL (e.g. `/blog/`, `/tag/`) |
| `modifiedAfter` | Keep URLs with `lastmod` on or after a date: `2026-01-01` or `7 days` |
| `keepUrlsWithoutLastmod` | With `modifiedAfter`, also keep URLs without lastmod |
| `deduplicate` | One row per URL even if it is listed in several sitemaps (default on) |
| `onlyNew`, `onlyNewMode`, `stateKey` | For scheduled runs: only URLs not delivered before (`new`), or also those whose lastmod changed (`newOrUpdated`) |
| `useRobotsTxt`, `tryCommonPaths`, `maxDepth`, `maxSitemapsPerInput` | Discovery settings |
| `saveUrlList` | Also save a plain-text list of URLs (record `URL_LIST`) |
| `proxyFallback`, `residentialFallback`, `maxResidentialMb` | What to do if a site refuses the connection |

### Output

One dataset item per URL:

```json
{
  "url": "https://www.gov.uk/government/news/miniature-surveillance-helicopters-help-protect-front-line-troops",
  "title": null,
  "lastmod": "2026-08-14T16:41:45Z",
  "changefreq": null,
  "priority": 0.29,
  "domain": "www.gov.uk",
  "path": "/government/news/miniature-surveillance-helicopters-help-protect-front-line-troops",
  "sitemapUrl": "https://www.gov.uk/sitemaps/sitemap_1.xml",
  "sitemapType": "urlset",
  "inputUrl": "https://www.gov.uk/sitemap.xml",
  "alternates": null,
  "imageUrls": null,
  "newsTitle": null,
  "newsPublicationDate": null,
  "videoTitles": null,
  "scrapedAt": "2026-10-17T09:12:03Z"
}
```

A multilingual page gives `"alternates": [{"hreflang": "fr", "url": "https://example.com/fr/page"}, ...]`.

Dates: `lastmod` is kept as `YYYY-MM-DD` when the sitemap gives only a date, and converted to ISO 8601 UTC when it gives a time.

Also in the key-value store:

- `URL_LIST`: every delivered URL, one per line (up to 200,000), ready for `wget -i`, a crawler or an LLM pipeline
- `SUMMARY`: per input, the sitemaps found, the number of sitemap files read, URLs delivered and the exact errors; plus duplicates, filtered and already-seen counts and network traffic

Download the dataset as JSON, CSV, Excel, XML or HTML from the run page or the API.

### Use cases

#### 1. SEO audit: find stale and orphan pages

**Who**: an SEO consultant auditing a client site before a migration.
**Input**:

```json
{ "urls": ["client-site.com"], "maxUrls": 0, "excludePatterns": ["/tag/", "/author/"] }
```

**Output (extract)**:

```json
[
  { "url": "https://client-site.com/services/audit", "lastmod": "2026-09-30", "priority": 0.8, "sitemapUrl": "https://client-site.com/page-sitemap.xml" },
  { "url": "https://client-site.com/blog/old-guide", "lastmod": "2019-03-02", "priority": 0.3, "sitemapUrl": "https://client-site.com/post-sitemap.xml" }
]
```

**Next**: export to CSV, sort by `lastmod` to list pages not updated for 2+ years, and compare with the crawl or Google Search Console to find pages in the sitemap but not linked (orphans) or linked but missing from the sitemap. 12,000 URLs = $2.40.

#### 2. Competitor content monitoring every morning

**Who**: a content marketing manager watching what 5 competitors publish.
**Input** (scheduled daily):

```json
{
  "urls": ["competitor-a.com", "competitor-b.com", "competitor-c.io", "competitor-d.com", "competitor-e.co.uk"],
  "includePatterns": ["/blog/", "/guides/"],
  "onlyNew": true
}
```

**Output (extract)**:

```json
[
  { "url": "https://competitor-b.com/blog/pricing-page-teardown", "lastmod": "2026-10-16", "inputUrl": "competitor-b.com" }
]
```

**Next**: an n8n or Make scenario sends the new URLs to Slack and to a Google Sheet. The first run returns the whole back catalogue; after that, a typical day returns 5 to 20 URLs, i.e. less than $0.01 per day.

#### 3. Feed a RAG index or a crawler with the right pages only

**Who**: an AI engineer building a support chatbot on a product's documentation.
**Input**:

```json
{ "urls": ["https://docs.apify.com/sitemap.xml"], "includePatterns": ["/platform/"], "modifiedAfter": "30 days", "maxUrls": 2000 }
```

**Output (extract)**:

```json
[
  { "url": "https://docs.apify.com/platform/actors/running", "lastmod": "2026-10-02", "sitemapUrl": "https://docs.apify.com/sitemap_base.xml" }
]
```

**Next**: pass the `URL_LIST` file to a content crawler (or a Markdown converter) and re-embed only the pages changed in the last 30 days instead of re-crawling the whole site. 2,000 URLs = $0.40.

#### 4. E-commerce catalogue tracking

**Who**: a pricing analyst tracking a retailer's product range.
**Input**: `{ "urls": ["shop.example.com"], "includePatterns": ["/products/"], "onlyNew": true, "onlyNewMode": "newOrUpdated" }`
**Next**: new product URLs (new references) and updated ones (price or stock change pages) go to the price scraper; nothing else is re-scraped.

### Use with the API

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("YOUR_USERNAME/sitemap-extractor").call(run_input={
    "urls": ["crawlee.dev"],
    "maxUrls": 500,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item["lastmod"])
```

**JavaScript**

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('YOUR_USERNAME/sitemap-extractor').call({
    urls: ['crawlee.dev'],
    maxUrls: 500,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((i) => `${i.url} ${i.lastmod}`).join('\n'));
```

**One HTTP call (sync, returns the items)**

```bash
curl -X POST "https://api.apify.com/v2/acts/YOUR_USERNAME~sitemap-extractor/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["crawlee.dev"], "maxUrls": 100}'
```

### Integrations: Make, Zapier, n8n, Google Sheets

Example automation (n8n or Make):

1. **Schedule** trigger, every day at 7:00.
2. **Apify: Run actor** `sitemap-extractor` with `{"urls": ["competitor-a.com", "competitor-b.com"], "onlyNew": true}`.
3. **Apify: Get dataset items** of the run.
4. **Google Sheets: Append row** with `url`, `lastmod`, `inputUrl`, then **Slack: Send message** "N new pages on competitor sites".

On Zapier, use the Apify app ("Run Actor" then "Find dataset items"). You can also connect the run to Google Sheets, Airtable, webhooks or email directly in the Apify Console (Integrations tab).

### Use with AI agents (MCP)

The actor is available through the Apify MCP server, so Claude, Cursor, ChatGPT and other AI agents can call it as a tool: "list every blog URL of example.com updated this month" becomes `{"urls": ["example.com"], "includePatterns": ["/blog/"], "modifiedAfter": "30 days"}`. Point your MCP client to `https://mcp.apify.com` and add this actor. Output is flat JSON with self-explanatory field names, and the `URL_LIST` text file is easy to pass to a model.

### FAQ

**Do I need to know where the sitemap is?**
No. Give the domain; the actor reads robots.txt and tries the usual paths. If you already know the sitemap URL, give it directly (it is faster).

**Does it crawl the website?**
No. It reads only the sitemaps (and robots.txt), never the pages themselves. That is why it is fast and cheap. A site without a sitemap returns nothing (and costs nothing); use a crawler for those.

**Are sitemap index files billed?**
No. Only the page URLs delivered are billed, not the sitemap files that list other sitemaps.

**Some URLs have no lastmod. Why?**
Many sites do not publish it. The field is then `null`. With `modifiedAfter`, these URLs are dropped unless you enable `keepUrlsWithoutLastmod`.

**How do I get only new pages every day?**
Enable `onlyNew` and schedule the actor (or a task). The history is stored in your account in the key-value store `sitemap-extractor-state`, one history per input configuration (or per `stateKey`).

**A site returned nothing. What happened?**
Open the `SUMMARY` record: each input lists its errors ("no sitemap found", "HTTP 404", "domain does not resolve", "anti-bot challenge page"). Nothing is charged for it.

**Is it legal?**
Sitemaps are public files that site owners publish precisely so that robots can find their pages. The actor reads only those public files, without login.

### Limits

- Sites that protect their sitemap behind a JavaScript challenge or a captcha are reported as blocked; the actor does not try to get around them.
- Requests go out directly from Apify. If a site refuses them (HTTP 403/429), the actor retries through Apify datacenter proxies, then through residential IPs, capped by `maxResidentialMb` (20 MB by default).
- The plain-text `URL_LIST` holds up to 200,000 URLs; the dataset has no limit.
- `onlyNew` remembers up to 500,000 URLs per history.
- `lastmod`, `priority` and `changefreq` are what the site declares; some CMSs set every `lastmod` to today's date.
- Sitemap files larger than 80 MB compressed or 250 MB decompressed are truncated (the sitemap protocol limit is 50 MB).

# Actor input Schema

## `urls` (type: `array`):

One per line. Accepted: a domain (example.com), any page of the site (https://example.com/blog/), a robots.txt URL, or a sitemap URL (XML, .xml.gz, sitemap index, text, RSS/Atom). For a domain, the sitemaps are discovered from robots.txt, then from /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml.

## `maxUrls` (type: `integer`):

Stop after this many URLs across all inputs. 0 = no limit. You pay only for the URLs delivered.

## `maxUrlsPerInput` (type: `integer`):

Limit for each domain or sitemap of the list, so that one huge site does not use the whole budget. 0 = no limit.

## `includePatterns` (type: `array`):

Keep only URLs that match at least one of these regular expressions (or plain text), e.g. /blog/ or /products/. Case-insensitive. URLs filtered out are not charged.

## `excludePatterns` (type: `array`):

Drop URLs that match any of these regular expressions (or plain text), e.g. /tag/ or ?page=.

## `modifiedAfter` (type: `string`):

Keep only URLs whose lastmod is on or after this date. Absolute (2026-01-01) or relative (7 days, 2 weeks, 3 months).

## `keepUrlsWithoutLastmod` (type: `boolean`):

With a 'Modified after' date: also keep URLs whose sitemap gives no lastmod (otherwise they are dropped).

## `deduplicate` (type: `boolean`):

A URL listed in several sitemaps is returned (and charged) only once.

## `onlyNew` (type: `boolean`):

For scheduled runs: return only the URLs not delivered by a previous run with the same input. The history is kept in the named key-value store 'sitemap-extractor-state'. The first run returns everything.

## `onlyNewMode` (type: `string`):

'new': URLs never seen before. 'newOrUpdated': also URLs whose lastmod changed since the last run.

## `stateKey` (type: `string`):

Name of the history used by 'Only new'. Leave empty to get one history per input configuration. Set the same name in two tasks to share it.

## `useRobotsTxt` (type: `boolean`):

For domains and page URLs: read the Sitemap: lines of robots.txt.

## `tryCommonPaths` (type: `boolean`):

If robots.txt declares no sitemap, try /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml and /sitemap.txt.

## `maxDepth` (type: `integer`):

How many levels of nested sitemap indexes to follow.

## `maxSitemapsPerInput` (type: `integer`):

Safety limit on the number of sitemap files read for one input.

## `saveUrlList` (type: `boolean`):

Also save all delivered URLs, one per line, in the key-value store record URL_LIST (up to 200,000 URLs). Handy for crawlers, LLM pipelines and wget.

## `maxConcurrency` (type: `integer`):

How many inputs are processed at the same time. Within one site, at most 3 sitemap files are read at once, politely.

## `proxyFallback` (type: `boolean`):

Requests go out directly. If a site refuses the connection (403/429), retry through Apify datacenter proxies. No extra cost for you.

## `residentialFallback` (type: `boolean`):

If datacenter IPs are refused too, try residential IPs (capped, see below). Never used to solve captchas or JavaScript challenges.

## `maxResidentialMb` (type: `integer`):

Maximum residential proxy traffic per run.

## Actor input object example

```json
{
  "urls": [
    "https://www.gov.uk/sitemap.xml",
    "crawlee.dev"
  ],
  "maxUrls": 100,
  "maxUrlsPerInput": 0,
  "keepUrlsWithoutLastmod": false,
  "deduplicate": true,
  "onlyNew": false,
  "onlyNewMode": "new",
  "useRobotsTxt": true,
  "tryCommonPaths": true,
  "maxDepth": 5,
  "maxSitemapsPerInput": 1000,
  "saveUrlList": true,
  "maxConcurrency": 5,
  "proxyFallback": true,
  "residentialFallback": true,
  "maxResidentialMb": 20
}
```

# Actor output Schema

## `urls` (type: `string`):

One item per page URL found in the sitemaps

## `urlsFull` (type: `string`):

No description

## `urlList` (type: `string`):

All delivered URLs, one per line (up to 200,000)

## `summary` (type: `string`):

Sitemaps found per input, errors, duplicates and filtered counts, network traffic

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.gov.uk/sitemap.xml",
        "crawlee.dev"
    ],
    "maxUrls": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("actose/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": [
        "https://www.gov.uk/sitemap.xml",
        "crawlee.dev",
    ],
    "maxUrls": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("actose/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.gov.uk/sitemap.xml",
    "crawlee.dev"
  ],
  "maxUrls": 100
}' |
apify call actose/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,actose/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/rUAEjl5VLJmkVHcMw/builds/8KBEwFjDVjj9pREmq/openapi.json