# Sitemap Extractor: All URLs from XML Sitemaps for SEO (`fecklesspleb/sitemap-extractor`) Actor

Extract URLs from a site's XML sitemaps with lastmod, image counts and hreflang. Capped at 1,000 URLs per site by default (raise maxUrlsPerSite, 0 = no limit). Auto-finds sitemaps, filters by regex and date. $0.20 per 1,000 URLs.

- **URL**: https://apify.com/fecklesspleb/sitemap-extractor.md
- **Developed by:** [Scott Williams](https://apify.com/fecklesspleb) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.20 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor: get URLs from any XML sitemap for SEO audits and crawling

Sitemap Extractor finds a website's sitemaps and returns the URLs in them as structured data (capped at 1,000 URLs per site by default; raise `maxUrlsPerSite`, or set it to 0 for no limit), with `lastmod`, `changefreq`, `priority`, image counts, Google News titles and, optionally, hreflang alternates. Paste a domain and it reads the sitemaps listed in robots.txt, falling back to the usual paths (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml` and others). Or paste a sitemap URL directly. It follows sitemap indexes, reads gzipped (`.gz`) and plain-text sitemaps, filters by regex and last-modified date, and can return only URLs that are new since the last run. You pay $0.0002 per URL written to the dataset.

### Who it's for

- **SEO specialists** auditing a site's indexable pages, checking `lastmod` freshness, or comparing sitemap URLs with crawl and Search Console data.
- **Content and competitive researchers** tracking which pages a competitor publishes, and when.
- **Developers and data teams** feeding URL lists into a crawler, a migration check or a RAG pipeline.
- **Agencies** pulling page inventories for many client sites at once.

### What data you get

One dataset item per URL:

| Field | What it holds |
|---|---|
| `url` | The page URL from the sitemap. |
| `site` | The site origin it came from, e.g. `https://www.nasa.gov`. |
| `lastmod` | Last-modified date, normalized to ISO 8601 (null if the sitemap has none). |
| `changefreq`, `priority` | As given in the sitemap (often null). |
| `sitemapUrl` | The sitemap file the URL was listed in. |
| `imageCount` | Number of `<image:image>` entries, if any. |
| `newsTitle` | Google News sitemap title, if any. |
| `alternates` | hreflang alternates, when `includeAlternates` is on. |

Sample item (real output from nasa.gov):

```json
{
  "url": "https://www.nasa.gov/history/glenn-orbits-the-earth/",
  "site": "https://www.nasa.gov",
  "lastmod": "2023-07-25T15:19:07+00:00",
  "changefreq": null,
  "priority": null,
  "sitemapUrl": "https://www.nasa.gov/wp-sitemap-posts-post-1.xml",
  "imageCount": null,
  "newsTitle": null
}
```

### How to use it

Default mode: if you leave **Websites or sitemap URLs** empty, or keep only the sample sitemap `https://www.nasa.gov/wp-sitemap.xml` that the form starts with, the run reads that NASA sitemap and returns at most 40 URLs, charged at the normal per-URL price (so at most $0.008). The log says `Default mode: no start URLs given (or only the sample default), returning up to 40 sample URLs from https://www.nasa.gov/wp-sitemap.xml`. Replace the sample with your own sites for a normal run.

| Input field | Label | Notes |
|---|---|---|
| `startUrls` | Websites or sitemap URLs | Defaults to the sample NASA sitemap (default mode, at most 40 URLs). A domain auto-discovers sitemaps; a sitemap URL is read directly. |
| `includePatterns` | Include URL patterns (regex) | Keep URLs matching any, e.g. `/blog/`. |
| `excludePatterns` | Exclude URL patterns (regex) | Drop URLs matching any. |
| `sitemapPatterns` | Only follow child sitemaps matching (regex) | For big indexes, e.g. `post-sitemap`. Saves requests. |
| `lastmodAfter` | Changed after (date) | `YYYY-MM-DD`. |
| `lastmodWithinDays` | Changed within last N days | Overrides `lastmodAfter`. |
| `keepUndatedUrls` | Keep URLs without a lastmod | Default `false`: with a date filter set, URLs that have no `lastmod` are dropped. Set `true` to keep them. |
| `maxUrlsPerSite` | Max URLs per site | Default 1,000. 0 = no limit. |
| `maxTotalUrls` | Max URLs in total | 0 = no limit. |
| `maxSitemapsPerSite` | Max sitemap files per site | Default 1,000. |
| `includeAlternates` | Include hreflang alternates | Default `false`. |
| `onlyNewUrls` | Only new URLs (monitoring) | Default `false`. Seen URLs are remembered for 365 days. |
| `stateStoreName` | State store name | Default `sitemap-extractor-state`. One name per monitor. |

Example: blog URLs changed in the last 30 days.

```json
{
  "startUrls": [
    "https://www.example.com"
  ],
  "includePatterns": [
    "/blog/"
  ],
  "lastmodWithinDays": 30
}
```

Example: new pages on a competitor's site since the last run (schedule it daily or weekly).

```json
{
  "startUrls": [
    "https://competitor.example.com"
  ],
  "onlyNewUrls": true,
  "stateStoreName": "competitor-new-pages"
}
```

Notes:

- **Date filters and missing `lastmod`:** when `lastmodAfter` or `lastmodWithinDays` is set, URLs without a `lastmod` are dropped by default, because their age can't be checked, and the log and `RUN_SUMMARY` (`undatedDropped`) say how many. Set `keepUndatedUrls` to `true` to keep them (`undatedKept` in `RUN_SUMMARY`). Many sites leave `lastmod` out, so on such a site that returns every URL, up to `maxUrlsPerSite`, and each one is billed.
- Duplicate URLs within a site are returned once.
- The Actor reads sitemaps only. It doesn't visit the listed pages, so it doesn't report HTTP status codes, titles or canonical tags.
- `RUN_SUMMARY` in the default key-value store shows, per site, where the sitemap was found (`robots.txt`, a fallback path, or `given`), sitemap files read, errors, URLs found and URLs pushed, and, when a date filter is set, how many URLs without a `lastmod` were dropped (`undatedDropped`) or kept (`undatedKept`). Each site's `status` is `ok`, `no-sitemap`, `skipped` (robots.txt disallows the sitemaps or can't be read) or `error`. A site whose sitemaps all failed to load is `error`, with the last error, never `ok`.

### Pricing

This Actor uses pay-per-event pricing: **$0.0002 per URL** written to the dataset. That is **$0.20 per 1,000 URLs**. There is no separate start fee in this Actor's pricing (as listed on October 28, 2026; the Pricing tab on the Store page is the source of truth).

You are charged only for URLs that end up in your dataset. URLs removed by your filters or date limits, duplicates within a site, and URLs skipped in monitoring mode are not charged.

| Example | Urls | Cost |
|---|---:|---:|
| Small site | 500 | $0.10 |
| Default cap of one site (1,000 URLs) | 1,000 | $0.20 |
| One site with the cap raised to 10,000 | 10,000 | $2.00 |
| Large site, no cap (illustration) | 250,000 | $50.00 |

With the default `maxUrlsPerSite` of 1,000, one site costs at most $0.20 per run. Raise it, or set 0 for no limit, to read bigger sitemaps.

Worked example: 1,000 URLs × $0.0002 = **$0.20**.

To cap spend, set **Maximum cost per run** in the run options (or `maxTotalChargeUsd` via the API). The Actor stops cleanly when the next url would go over the limit and records `stoppedByCostLimit: true` in the run summary.

### How it compares

Prices are from the public Apify Store listings on October 28, 2026.

| Option | Pricing | What it is good at |
|---|---|---|
| This Actor | $0.0002 per URL | Sitemap discovery via robots.txt and common paths, indexes, `.gz` and text sitemaps, lastmod/regex filters, image/news/hreflang data, only-new-URLs mode. |
| [apify/sitemap-extractor](https://apify.com/apify/sitemap-extractor) | See listing | Apify's own sitemap Actor. Check its listing for current features. |
| Desktop SEO crawlers | License | Crawl the pages themselves and compare against the sitemap. Better for full technical audits. |
| Free online sitemap URL extractors | Free | Fine for a single small sitemap. |

### Integrations and API

The actor ID is `fecklesspleb/sitemap-extractor` (in API URLs: `fecklesspleb~sitemap-extractor`).

**n8n, Make and Zapier.** Each has an Apify integration with a "run Actor" action. Pick `fecklesspleb/sitemap-extractor`, paste the input JSON shown below, then read the results from the run's default dataset in the next step. To process only new URLs on a schedule, turn on the monitoring option and run it from a schedule (Apify Schedules or a trigger in your automation tool).

**Webhooks and schedules.** Apify Schedules can run the Actor on a cron, and a webhook on "run succeeded" can call your own endpoint with the run ID.

**HTTP API.** Run the Actor and get the items back in one call. This waits for the run to finish, so use it for small runs:

```bash
curl -X POST "https://api.apify.com/v2/acts/fecklesspleb~sitemap-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://www.example.com"], "includePatterns": ["/blog/"], "lastmodWithinDays": 30}'
```

**Python** (`pip install apify-client`):

```python
from decimal import Decimal
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("fecklesspleb/sitemap-extractor").call(
    run_input={
        "startUrls": [
            "https://www.example.com"
        ],
        "includePatterns": [
            "/blog/"
        ],
        "lastmodWithinDays": 30
    },
    max_total_charge_usd=Decimal("5"),  # optional spend cap
)
## apify-client 3.x returns a Run object; on 1.x/2.x use run["defaultDatasetId"]
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)
```

**JavaScript / Node.js** (`npm install apify-client`):

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fecklesspleb/sitemap-extractor').call(
    {
        "startUrls": [
            "https://www.example.com"
        ],
        "includePatterns": [
            "/blog/"
        ],
        "lastmodWithinDays": 30
    },
    { maxTotalChargeUsd: 5 }, // optional spend cap
);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

Results can also be exported from the dataset as JSON, CSV, Excel, XML or RSS from the Apify Console or the dataset API.

### FAQ

**How do I extract all URLs from a sitemap.xml?**
Paste the sitemap URL (or just the domain) in `startUrls` and run. Every `<loc>` becomes one item. Set `maxUrlsPerSite` to 0 to remove the default 1,000 cap.

**How do I find a website's sitemap?**
Paste the domain. The Actor reads `Sitemap:` lines in robots.txt, then tries `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml` and `/sitemap.xml.gz`. `RUN_SUMMARY` tells you where it found the sitemap, or that none was found.

**Can it handle sitemap index files and .gz sitemaps?**
Yes. Indexes are followed recursively (up to `maxSitemapsPerSite` files), and gzipped and plain-text sitemaps are read. Use `sitemapPatterns` to follow only some child sitemaps.

**How can I monitor a competitor's new pages?**
Turn on `onlyNewUrls`, give the monitor its own `stateStoreName`, and schedule it. Each run returns only URLs not seen before, and you only pay for those.

**Can I export sitemap URLs to CSV or Google Sheets?**
Yes. Export the dataset as CSV or Excel, or send it to Google Sheets through the n8n, Make or Zapier integration.

**Does it check whether URLs return 200 or 404?**
No. It reads the sitemap only. Feed the URL list to a crawler if you need status codes.

**Do I need an API key or login?**
No. The Actor reads public endpoints and needs no account on the source site. You only need an Apify account to run it. It obeys robots.txt and rate-limits requests per host. robots.txt is checked again on every redirect, so a URL that redirects to a disallowed path or another host is not followed. If a host's robots.txt can't be read (a server error, a network failure or a page that isn't a robots file), that host is skipped for the run and `RUN_SUMMARY` says why. A missing robots.txt means no rules.

### Related Actors

- [RSS Feed Reader](https://apify.com/fecklesspleb/rss-feed-reader): RSS and Atom feed items as clean JSON, with feed discovery from a website URL.
- [Hacker News Scraper](https://apify.com/fecklesspleb/hacker-news-scraper): Hacker News stories and comments, plus parsed "Who is hiring?" job posts.
- [ATS Jobs Scraper](https://apify.com/fecklesspleb/ats-jobs-scraper): jobs from Greenhouse, Lever, Ashby, Workable, Recruitee and Workday boards in one schema.
- [Remote Jobs Aggregator](https://apify.com/fecklesspleb/remote-jobs-aggregator): remote jobs from Remote OK, We Work Remotely and Jobicy, de-duplicated.

# Changelog

This Actor's version history is a separate document: https://apify.com/fecklesspleb/sitemap-extractor/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

A site (https://www.nasa.gov) to auto-discover its sitemaps, or a direct sitemap URL. Leave empty, or keep the sample NASA sitemap, for a default-mode run: up to 40 sample URLs from https://www.nasa.gov/wp-sitemap.xml at the normal per-URL price.

## `includePatterns` (type: `array`):

Keep URLs matching any of these regular expressions, for example /blog/.

## `excludePatterns` (type: `array`):

Drop URLs matching any of these regular expressions.

## `sitemapPatterns` (type: `array`):

For big sitemap indexes, follow only child sitemaps matching these, for example post-sitemap.

## `lastmodAfter` (type: `string`):

Keep URLs whose lastmod is on or after this date, YYYY-MM-DD. URLs with no lastmod are dropped unless you turn on Keep URLs without a lastmod.

## `lastmodWithinDays` (type: `integer`):

Keep URLs whose lastmod is within the last N days. Overrides the date above. URLs with no lastmod are dropped unless you turn on Keep URLs without a lastmod.

## `keepUndatedUrls` (type: `boolean`):

Off by default: when a date filter is set, URLs with no lastmod in the sitemap are dropped, since their age can't be checked. Turn this on to keep them. On sites whose sitemaps have no lastmod, that returns every URL (up to the limits), and each one is billed.

## `maxUrlsPerSite` (type: `integer`):

0 = no limit.

## `maxTotalUrls` (type: `integer`):

0 = no limit.

## `maxSitemapsPerSite` (type: `integer`):

Safety cap on sitemap files read per site.

## `includeAlternates` (type: `boolean`):

Add language alternate URLs listed in the sitemap.

## `onlyNewUrls` (type: `boolean`):

Return only URLs not seen in earlier runs with the same state store.

## `stateStoreName` (type: `string`):

Named key-value store for monitoring. Use a different name per monitor.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.nasa.gov/wp-sitemap.xml"
  ],
  "keepUndatedUrls": false,
  "maxUrlsPerSite": 1000,
  "maxTotalUrls": 0,
  "maxSitemapsPerSite": 1000,
  "includeAlternates": false,
  "onlyNewUrls": false,
  "stateStoreName": "sitemap-extractor-state"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.nasa.gov/wp-sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("fecklesspleb/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://www.nasa.gov/wp-sitemap.xml"] }

# Run the Actor and wait for it to finish
run = client.actor("fecklesspleb/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.nasa.gov/wp-sitemap.xml"
  ]
}' |
apify call fecklesspleb/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fecklesspleb/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/QPY9zYbFcdAvoTLlU/builds/wzG5W7Zf8YCYFFQNU/openapi.json