# Sitemap URL Extractor (`fleet261114/sitemap-url-extractor`) Actor

List every URL in a website's XML sitemaps, with lastmod, changefreq and priority. Finds sitemaps via robots.txt, follows sitemap indexes, reads .gz files. Filter by pattern or date.

- **URL**: https://apify.com/fleet261114/sitemap-url-extractor.md
- **Developed by:** [Fleet](https://apify.com/fleet261114) (community)
- **Categories:** SEO tools, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.35 / 1,000 url returneds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor

Get the full list of URLs a website publishes in its XML sitemaps, with each URL's `lastmod`, `changefreq` and `priority`, in one dataset you can export to CSV, Excel or JSON.

Give it a domain and it finds the sitemaps itself: it reads the `Sitemap:` lines in robots.txt, or tries `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml` and `/wp-sitemap.xml`. It follows sitemap index files to every child sitemap, reads gzip-compressed `.xml.gz` sitemaps and plain-text sitemaps, and removes duplicate URLs.

It fetches **only robots.txt and sitemap files**, never the pages themselves, so it is fast and light on the website: a site with 20,000 URLs typically takes well under a minute.

### Use it for

- **SEO audits and migrations**: the complete list of URLs a site wants indexed, before and after a move.
- **Content inventories**: every blog post or product page, with last-modified dates.
- **Change tracking**: run it on a schedule with "Only URLs modified after" to see what changed recently.
- **Feeding other tools**: a clean URL list to pass to a crawler, a link checker or an AI pipeline.

### Input

| Field | What it does |
|---|---|
| **Websites or sitemap URLs** (required) | `example.com`, `https://example.com`, or a sitemap URL such as `https://example.com/post-sitemap.xml` (also `.xml.gz`, `.txt`). |
| Keep URLs matching | Regular expressions (case-insensitive); keep a URL if any matches, e.g. `/blog/`. |
| Drop URLs matching | Regular expressions; drop a URL if any matches, e.g. `/tag/`. |
| Only URLs modified after | A date. URLs without a `lastmod` are dropped when this is set. |
| Max URLs per site | Stop after this many URLs for each site. 0 = no limit. |
| Only URLs on the same host | Drop sitemap entries pointing to other hosts. |

### Output

```json
{
  "url": "https://wordpress.org/news/2014/04/smith/",
  "lastmod": "2014-04-15T21:49:40Z",
  "changefreq": null,
  "priority": null,
  "sitemapUrl": "https://wordpress.org/news/video-sitemap-1.xml",
  "site": "https://wordpress.org",
  "scrapedAt": "2026-11-15T04:20:26.365Z"
}
```

`lastmod`, `changefreq` and `priority` are exactly as the site wrote them (or `null`). A `RUN_SUMMARY` record in the run's key-value store lists, per site, the sitemap files read, URLs listed and returned, sitemaps that failed (with the HTTP status or reason) and any that robots.txt disallows.

### Pricing

Pay per result: **$0.0005 per URL returned** ($0.50 per 1,000 URLs), plus Apify's small run-start fee. URLs removed by your filters or as duplicates are not charged. Large sites can have millions of URLs: use the filters, "Max URLs per site", or a maximum cost per run, which the Actor respects and stops at cleanly.

### Limits and honest notes

- It lists what the sitemaps say. Pages the site leaves out of its sitemaps are not found, and a sitemap can contain stale or broken URLs: this Actor does not visit the pages to check them.
- Sites with no sitemap, or whose sitemap is blocked or behind bot protection, return nothing for that site; the reason is in `RUN_SUMMARY` and the status message, and nothing is charged for that site.
- It obeys robots.txt for the sitemap files it fetches (rules for `*`, or rules naming `sitemap-url-extractor`).
- At most 5,000 sitemap files are read per site per run.
- No personal data is collected: only URLs and the sitemap fields.

### Use it from the API, a schedule or an AI agent

**API.** Start a run and get the results in one call (replace `YOUR_APIFY_TOKEN`):

```bash
curl -X POST "https://api.apify.com/v2/actors/fleet261114~sitemap-url-extractor/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://wordpress.org"], "maxUrlsPerSite": 1000}'
```

This waits up to 5 minutes. For bigger jobs, start a run with `POST /v2/acts/fleet261114~sitemap-url-extractor/runs` and read its dataset when it finishes. Apify's JavaScript and Python clients wrap the same calls.

**Schedules and integrations.** Create a task with your input and attach an Apify schedule (for example daily). Use Apify's integrations (webhooks, Make, Zapier, n8n, Google Sheets and others) to send the results onward.

**AI agents (MCP).** Add this Actor as a tool in any MCP client (Claude, Cursor and others) through Apify's hosted MCP server:

```
https://mcp.apify.com?tools=fleet261114/sitemap-url-extractor
```

The agent sees the input schema and the field descriptions above and can call the Actor directly. Set a maximum cost per run in the call options to cap spending.

### FAQ

**Does it crawl the website?** No. It reads only robots.txt and the sitemap files, never the pages, so it is fast and puts almost no load on the site.

**Why did a site return nothing?** It has no sitemap at robots.txt or the usual paths, or blocks automated requests. `RUN_SUMMARY` says which; nothing is charged for that site.

**Can I get only recently changed pages?** Yes: set "Only URLs modified after". This relies on the site's `lastmod` values being accurate.

**What does a typical run cost?** At $0.0005 per URL: 1,000 URLs cost about $0.50; 20,000 URLs about $10. Use the filters, "Max URLs per site" or a maximum cost per run to stay in budget.

### Questions and problems

Open an issue on the Actor's **Issues** tab with the run link.

# Actor input Schema

## `startUrls` (type: `array`):

A domain or homepage (example.com): sitemaps are found through robots.txt, or at /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml. Or a sitemap URL directly (https://example.com/post-sitemap.xml, .xml.gz or .txt).

## `includePatterns` (type: `array`):

Regular expressions, case-insensitive. Example: /blog/ . Empty = keep all.

## `excludePatterns` (type: `array`):

Regular expressions, case-insensitive. Example: ?page= or /tag/

## `lastmodAfter` (type: `string`):

Keep only URLs whose sitemap lastmod is on or after this date (YYYY-MM-DD). URLs without lastmod are dropped when this is set.

## `maxUrlsPerSite` (type: `integer`):

Stop reading a site's sitemaps after this many URLs are returned. 0 = no limit.

## `sameHostOnly` (type: `boolean`):

Drop sitemap entries that point to other hosts (e.g. a CDN or a sister site).

## Actor input object example

```json
{
  "startUrls": [
    "https://wordpress.org"
  ],
  "includePatterns": [],
  "excludePatterns": [],
  "maxUrlsPerSite": 0,
  "sameHostOnly": false
}
```

# Actor output Schema

## `urls` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://wordpress.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("fleet261114/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://wordpress.org"] }

# Run the Actor and wait for it to finish
run = client.actor("fleet261114/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://wordpress.org"
  ]
}' |
apify call fleet261114/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fleet261114/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/8E6V4DfY56EzVSorf/builds/CmiFaIAYBkR0h2fJs/openapi.json