# Sitemap Extractor & Parser - All URLs from sitemap.xml (`pennydata/sitemap-extractor-plus`) Actor

Sitemap URL extractor: follows nested indexes, reads gzip, finds sitemaps via robots.txt, dedupes, filters, returns lastmod and priority. Accepts the same startUrls input as other sitemap extractors; dead sitemaps never fail the run. Pay per URL.

- **URL**: https://apify.com/pennydata/sitemap-extractor-plus.md
- **Developed by:** [Penny Data](https://apify.com/pennydata) (community)
- **Categories:** SEO tools, Developer tools, MCP servers
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.28 / 1,000 sitemap url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor & Parser - All URLs from sitemap.xml, Index, Gzip, robots.txt, lastmod

Extract every page URL from a website's sitemaps, reliably. Give it a site (or a sitemap URL) and get a flat, deduplicated dataset of URLs with `lastmod`, `priority` and `changefreq`. **You are charged only per unique URL extracted ($0.0004 each, $0.40 per 1,000 URLs on the Free plan; paid Apify plans get lower tiers); sitemap files, errors, duplicates and filtered-out URLs are free.** A dead site, a 404 sitemap or a malformed URL never fails the run: it is logged and listed in the `SUMMARY` record.

### Quick start

Input (`startUrls` is the only field you need):

```json
{ "startUrls": ["https://docs.apify.com"], "maxUrls": 20 }
```

Output, one row per unique URL (illustrative shape):

```json
{ "url": "https://docs.apify.com/platform", "lastmod": "2026-09-30", "priority": 0.8, "changefreq": "weekly", "sitemapUrl": "https://docs.apify.com/sitemap.xml", "source": "https://docs.apify.com" }
```

Pricing event: `url-extracted`, $0.0004 per unique URL ($0.40 per 1,000 on the Free plan; paid Apify plans get lower tiers); sitemap files, errors, duplicates and filtered-out URLs are free.

### What it handles

- **Sitemap indexes**, followed recursively (configurable depth).
- **Gzip sitemaps** (`sitemap.xml.gz`), detected by content, even when the server mislabels them.
- **robots.txt discovery**: pass a home page and `Sitemap:` lines are found automatically; falls back to `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap.xml.gz`.
- **Plain-text sitemaps** (one URL per line).
- **Dedupe** across all sitemaps and all start URLs.
- **Filters**: include/exclude regex, `modifiedSince` / `modifiedUntil` (date or `7d`), `onlyChanged` monitoring, `maxUrls` cap (also your spend cap).
- Retries on network errors; 404/403 sitemaps are skipped and reported.

### Input

`startUrls` (sites or sitemap URLs; aliases `sitemapUrls`, `urls`, `sitemaps` are merged with it), `maxUrls` (default 20), `includePattern`, `excludePattern`, `modifiedSince`, `modifiedUntil`, `includeMissingLastmod`, `onlyChanged`, `maxSitemapDepth` (default 5), `checkPageStatus` (adds an HTTP `statusCode` per URL via HEAD, no extra charge).

**Switching from another sitemap actor?** Keep your saved input and change only the Actor ID. See the Switching section below.

**Drop-in for the input of apify/sitemap-extractor:** `startUrls` as `[{"url": ...}]`, `maxCrawlingDepth` (same as `maxSitemapDepth`) and `checkPageStatus` work as-is; `proxyConfiguration`, `proxyRotation` and `maxRequestRetries` are accepted and ignored, and no proxy is required.

### Output per URL

`url`, `lastmod`, `daysSinceModified`, `priority`, `changefreq`, `sitemapUrl` (file it came from), `source` (your start URL). Missing fields are `null`.

### Sitemap changed URLs: recently updated pages and lastmod filter

Use the sitemap lastmod filter to list only recently updated pages:

- `modifiedSince`: a date (`2026-10-01`) or a relative age (`7d`, `2w`, `3m`, `1y`). `modifiedUntil` sets the upper bound.
- URLs with no `lastmod` are skipped by default (counted in `skippedNoLastmod`); set `includeMissingLastmod: true` to keep them.
- Every row has a `status` (`ok`, `error` or `skipped`); unreadable sitemaps appear as free `error` rows with a reason. The `SUMMARY` record holds counts (`rows`, `ok`, `error`, `skipped`) that equal the dataset rows.
- Each row gets `daysSinceModified`. The `RUN_SUMMARY` key-value record holds `{ total, kept, skippedNoLastmod }`.

**Sitemap changed URLs on a schedule (`onlyChanged`)**: the actor stores each URL's last seen `lastmod` in a named key-value store per site (`sitemap-state-<host>`). Every later run outputs only URLs that are new or whose `lastmod` changed, so a daily Apify schedule works as a sitemap change monitor and you pay only for changed URLs. The first run outputs all URLs as the baseline. To reset, delete that named store.

```json
{ "startUrls": ["https://example.com"], "modifiedSince": "7d", "onlyChanged": true, "maxUrls": 1000 }
```

### Use cases

SEO audits, content inventories, crawl seeding, change monitoring (`modifiedSince`, `onlyChanged`), site migration checklists, feeding a crawler or scraper a clean URL list.

### FAQ

#### Does it follow a sitemap index (nested sitemaps)?

Yes. A `<sitemapindex>` is followed recursively down to `maxSitemapDepth` levels (default 5, max 10; `maxCrawlingDepth` is accepted as the same setting). Each sitemap file is read once. If the limit cuts off nested sitemaps, the `SUMMARY` record lists a "max sitemap depth reached" problem.

#### Can it read .xml.gz sitemaps?

Yes. A response that starts with the gzip header is decompressed whatever the URL or `Content-Type` says, so `sitemap.xml.gz` and mislabelled files both work, including gzipped files listed inside a sitemap index.

#### Do I need to know the sitemap URL? How does robots.txt discovery work?

No. Give a site such as `https://example.com`. The Actor reads its `robots.txt` and follows every `Sitemap:` line. If none is found it tries `/sitemap.xml`, and if that yields no URLs, `/sitemap_index.xml` and `/sitemap.xml.gz`. If you give a sitemap URL directly (ending in `.xml`, `.txt`, `.gz` or containing "sitemap"), robots.txt is skipped.

#### How do I get only URLs changed recently (lastmod filter)?

Set `modifiedSince` to a date (`2026-10-01`) or a relative age (`7d`, `2w`, `3m`, `1y`); `modifiedUntil` sets the upper bound. URLs with no `lastmod` are skipped unless `includeMissingLastmod` is `true`. For a schedule, add `onlyChanged` to output only new or changed URLs on each run.

#### What does it cost?

$0.0004 per unique URL ($0.40 per 1,000 on the Free plan; paid Apify plans get lower tiers) extracted. Sitemap files, duplicates, errors and URLs removed by your filters are free; `maxUrls` caps both the output and your spend.

### Honest limits

- Only fetches `robots.txt` and sitemap files that the site publishes for crawlers. It does not crawl pages, log in, or use proxies. Sites that block the request (403) are skipped.
- Sites without a sitemap return nothing; this is not a link crawler.
- Very large sites: raise `maxUrls`; each extracted URL is billed.

### Switching from another sitemap actor

This Actor accepts the input field names of these Store actors, so a saved task or API call needs only the Actor ID changed. Field names were read from each actor's public input schema on 2026-10-21; output rows are this Actor's own shape (`url`, `lastmod`, `priority`, `changefreq`, `sitemapUrl`, `source`), so check any code that reads the dataset.

| You used | Your input field | Here |
|---|---|---|
| apify/sitemap-extractor | `startUrls` (`[{"url": ...}]` or strings) | works as-is |
| apify/sitemap-extractor | `maxCrawlingDepth` | same as `maxSitemapDepth` |
| apify/sitemap-extractor | `checkPageStatus` | works as-is (adds `statusCode`, no extra charge) |
| apify/sitemap-extractor | `proxyConfiguration`, `proxyRotation`, `maxRequestRetries` | accepted, ignored (no proxy needed) |
| onescales/sitemap-url-extractor | `sitemapUrls` | works as-is (merged with `startUrls`) |
| pink_comic/sitemap-url-extractor | `urls`, `maxResults`, `modifiedSince` | `urls` and `modifiedSince` as-is; `maxResults` = `maxUrls`; `autoDiscover` accepted (discovery is always on); `includeImages` is not supported |
| mikolabs/sitemap-url-extractor | `sitemapUrls`, `maxDepth`, `filterUrlPattern` | `maxDepth` = `maxSitemapDepth`; `filterUrlPattern` = `includePattern`; `requestTimeoutSecs` accepted, ignored |

Not affiliated with any of those actors. Pricing here: $0.0004 per unique URL, nothing for sitemap files, errors or duplicates.

### Use from Claude, Cursor or any MCP client

Apify's MCP server can expose this actor as a tool. Use this URL (format from the [Apify MCP docs](https://docs.apify.com/platform/integrations/mcp), `?tools=` takes a comma-separated list of actors):

`https://mcp.apify.com/?tools=pennydata/sitemap-extractor-plus`

Client config (`mcpServers` format; sign in with OAuth when the client asks):

```json
{"mcpServers": {"apify": {
  "url": "https://mcp.apify.com/?tools=pennydata/sitemap-extractor-plus"
}}}
```

Example prompt: "List every URL in the sitemap of docs.apify.com, limited to 50 URLs."

### Related actors

- [Shot PDF](https://apify.com/pennydata/shot-pdf): capture screenshots or PDFs of the URLs you extracted.
- [Website Tech Stack Detector](https://apify.com/pennydata/website-tech-stack-detector): checks the technology behind the site you mapped.
- [RSS/Atom Feed Reader (bulk)](https://apify.com/pennydata/rss-atom-feed-reader-bulk): reads the site's feeds when it has them.

Not affiliated with sitemaps.org or any site you extract from. Written with the assistance of an AI (Claude Sonnet 5.5).

# Actor input Schema

## `startUrls` (type: `array`):

Website home pages (robots.txt is checked for Sitemap: lines, then /sitemap.xml is tried) or direct sitemap URLs (.xml or .xml.gz). Sitemap indexes are followed recursively. Aliases: sitemapUrls, urls, sitemaps (all merged).

## `maxUrls` (type: `integer`):

Stop after this many unique URLs in total (and charge for at most this many). Use a higher value to extract an entire large site.

## `includePattern` (type: `string`):

Only keep URLs matching this JavaScript regular expression, e.g. /blog/. An invalid regex is ignored with a warning.

## `excludePattern` (type: `string`):

Drop URLs matching this JavaScript regular expression, e.g. .(pdf|jpg)$.

## `modifiedSince` (type: `string`):

Keep only URLs whose sitemap lastmod is on or after this. Accepts a date (2026-01-01) or a relative age like 7d, 2w, 3m, 1y.

## `modifiedUntil` (type: `string`):

Keep only URLs whose lastmod is on or before this date (YYYY-MM-DD or relative like 7d).

## `includeMissingLastmod` (type: `boolean`):

When a date filter is set, also keep URLs that have no (or an unreadable) lastmod. Default false: they are skipped and counted in RUN_SUMMARY.skippedNoLastmod.

## `onlyChanged` (type: `boolean`):

Remembers each URL's lastmod in a named key-value store per site (sitemap-state-<host>) and outputs only URLs that are new or whose lastmod changed since the previous run. The first run outputs everything. Schedule it daily to monitor a sitemap for changes.

## `maxSitemapDepth` (type: `integer`):

How many levels of nested sitemap indexes to follow. Alias: maxCrawlingDepth.

## `checkPageStatus` (type: `boolean`):

Send a HEAD request to each extracted URL and add its HTTP status code as statusCode (null if unreachable). Slower; no extra charge.

## `maxCrawlingDepth` (type: `integer`):

Compatibility with apify/sitemap-extractor: same as Max sitemap index depth. Used only when maxSitemapDepth is not set.

## `proxyConfiguration` (type: `object`):

Accepted for compatibility with apify/sitemap-extractor. Not required and currently ignored.

## `proxyRotation` (type: `string`):

Accepted for compatibility with apify/sitemap-extractor; ignored.

## `maxRequestRetries` (type: `integer`):

Accepted for compatibility with apify/sitemap-extractor; ignored (fetches use built-in retries).

## `sitemapUrls` (type: `array`):

Alias of Sites or sitemap URLs (startUrls). Lists from all four fields are merged.

## `urls` (type: `array`):

Alias of Sites or sitemap URLs (startUrls). Lists from all four fields are merged.

## `sitemaps` (type: `array`):

Alias of Sites or sitemap URLs (startUrls). Lists from all four fields are merged.

## `maxResults` (type: `integer`):

Compatibility with pink_comic/sitemap-url-extractor: same as Max URLs. Used only when maxUrls is not set.

## `maxDepth` (type: `integer`):

Compatibility with mikolabs/sitemap-url-extractor: same as Max sitemap index depth. Used only when maxSitemapDepth is not set.

## `filterUrlPattern` (type: `string`):

Compatibility with mikolabs/sitemap-url-extractor: same as Include regex. Used only when includePattern is not set.

## `requestTimeoutSecs` (type: `integer`):

Accepted for compatibility with mikolabs/sitemap-url-extractor; ignored.

## `autoDiscover` (type: `boolean`):

Accepted for compatibility with pink_comic/sitemap-url-extractor; robots.txt discovery is always on for site URLs.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com"
  ],
  "maxUrls": 5,
  "includeMissingLastmod": false,
  "onlyChanged": false,
  "maxSitemapDepth": 5,
  "checkPageStatus": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com"
    ],
    "maxUrls": 5,
    "includePattern": "",
    "excludePattern": "",
    "modifiedSince": "",
    "modifiedUntil": "",
    "maxSitemapDepth": 5,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("pennydata/sitemap-extractor-plus").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com"],
    "maxUrls": 5,
    "includePattern": "",
    "excludePattern": "",
    "modifiedSince": "",
    "modifiedUntil": "",
    "maxSitemapDepth": 5,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("pennydata/sitemap-extractor-plus").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com"
  ],
  "maxUrls": 5,
  "includePattern": "",
  "excludePattern": "",
  "modifiedSince": "",
  "modifiedUntil": "",
  "maxSitemapDepth": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call pennydata/sitemap-extractor-plus --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pennydata/sitemap-extractor-plus"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Coy8cbQTI06jXLXKb/builds/lUvjLIrbzXFeTXYgM/openapi.json