# Sitemap URL Extractor: Nested, Gzip & Filters (`drop_data_solutions/sitemap-url-extractor`) Actor

Get the page URLs a website lists in its sitemaps. Give a domain, homepage, robots.txt or sitemap URL: it finds the sitemaps, follows nested indexes, reads .xml.gz, RSS and text sitemaps, filters by regex or lastmod date, and falls back to a light crawl when there's no sitemap.

- **URL**: https://apify.com/drop_data_solutions/sitemap-url-extractor.md
- **Developed by:** [drop_data_solutions](https://apify.com/drop_data_solutions) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.08 / 1,000 extracted urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Get the page URLs a website lists in its sitemaps. Give **Sitemap URL Extractor** a domain, homepage, `robots.txt` or sitemap URL. It finds the sitemaps, follows nested sitemap indexes, reads gzipped (`.xml.gz`), RSS/Atom and plain-text sitemaps, and returns a clean, de-duplicated list of URLs with `lastmod`, `changefreq`, `priority`, hreflang alternates and the sitemap each URL came from.

If a site has no sitemap, the run still succeeds: the report says so and (optionally) a light, robots.txt-respecting crawl lists the site's internal links instead.

### What can Sitemap URL Extractor do?

- 🔎 **Auto-discovers sitemaps:** reads `Sitemap:` lines in robots.txt, then tries `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemaps.xml`, `/sitemap/sitemap.xml` and `/sitemap.txt`. HTML "soft 404" pages returned with status 200 are detected and skipped.
- 🌳 **Follows nested sitemap indexes** up to 10 levels deep (default 6), fetching each sitemap once even when several indexes list it.
- 🗜️ **Reads the common sitemap formats:** XML urlsets and indexes, `.xml.gz` (detected by content, not file name), RSS 2.0, Atom and plain text. Common XML mistakes (stray `&`, bad entities, CDATA, odd namespace prefixes) are tolerated.
- 🎯 **Filters:** include/exclude regex, last-modified date range (`2026-01-31` or `30 days`), max URLs per run and per site, same-host only. With a date filter, whole child sitemaps older than your start date are skipped, which saves time and money.
- 🌍 **Extensions:** hreflang alternates (`xhtml:link`) and, optionally, image, video and Google News entries.
- 🕸️ **No-sitemap fallback:** a polite breadth-first crawl of same-host links (50 pages, 2 clicks deep by default) that respects `robots.txt` Disallow rules.
- 🧯 **Reliable and polite:** at most 2 requests per host at a time, a connect and idle timeout on every request (so a stalled server can't hang the run), retries with exponential backoff that honour `Retry-After`, and Apify Proxy only when a site blocks direct requests.
- 📉 **Low memory:** 50 MB sitemaps are parsed as they stream in; the default is 256 MB of memory.
- 🧾 **Per-site report:** how sitemaps were found, which failed and why (404, bot protection, DNS error, not a sitemap), and how many URLs were filtered or de-duplicated.

### What can I use a sitemap URL list for?

- **SEO audits:** compare the URLs in the sitemap with what's indexed, find stale `lastmod` dates, check hreflang coverage.
- **Crawl seeding:** feed the URL list to a scraper (Website Content Crawler, Cheerio/Playwright scrapers) instead of crawling blindly.
- **Change monitoring:** schedule weekly with `lastmodFrom: "7 days"` to get only pages that changed.
- **Content inventory and migrations:** a full list of pages before a redesign or CMS move.
- **AI and RAG pipelines:** collect the exact page list of a docs site or blog before ingesting it.

### How to get a website's URLs from its sitemap

1. Open the Actor and enter a site in **Start URLs** (e.g. `https://www.sitemaps.org`) or a bare domain in **Domains**.
2. Optional: add **Include/Exclude** patterns (e.g. exclude `/tag/`) or a **Modified on or after** date.
3. Click **Start**. Small sites finish in seconds.
4. Open the **Output** tab and download the list as JSON, CSV, Excel or HTML, or read it through the API.

#### Example input

```json
{
  "startUrls": [{ "url": "https://www.nytimes.com/sitemaps/new/news.xml.gz" }],
  "domains": ["apify.com", "wordpress.org"],
  "excludePatterns": ["/tag/", "\\?page="],
  "lastmodFrom": "30 days",
  "maxUrls": 50000
}
```

### Input

| Field | What it does |
| --- | --- |
| Start URLs | Homepages, `robots.txt` files or sitemap URLs (with `https://`). Mix freely. |
| Domains | Bare domains, one per line (`example.com`). |
| Auto-discover sitemaps | On by default. |
| Maximum URLs / per site | 0 = no limit (the run still stops at your Maximum cost per run). |
| Include / Exclude (regex) | e.g. include `/blog/`, exclude `/tag/`. |
| Modified on or after / before | ISO date or relative (`7 days`). |
| Keep URLs without a lastmod date | When a date filter is set. Default on. |
| Include hreflang alternates / media | Output options. |
| Crawl the site when no sitemap exists | Default on; max pages 50, depth 2. |
| Proxy configuration and mode | Default: Apify Proxy only when blocked. |

### Output

One dataset item per unique URL:

```json
{
  "url": "https://apify.com/store",
  "lastmod": "2026-10-20T00:00:00.000Z",
  "changefreq": "daily",
  "priority": 0.8,
  "sourceSitemap": "https://apify.com/sitemap-0.xml",
  "alternates": [{ "hreflang": "de", "href": "https://apify.com/de/store" }],
  "domain": "apify.com",
  "discoveredBy": "sitemap"
}
```

Crawled URLs have `discoveredBy: "crawl"`, `sourceSitemap: null` and `foundOn`. The `OUTPUT` record in the key-value store holds the per-site report:

```json
{
  "urlsExtracted": 20000, "sites": 2, "sitesWithUrls": 1, "sitemapsRead": 19,
  "domains": [
    { "domain": "apify.com", "status": "ok", "discoveredBy": "robots.txt", "sitemapsFoundCount": 19, "urlsOutput": 20000, "sitemapsFailed": [] },
    { "domain": "example.com", "status": "no-sitemap", "notes": ["This site publishes no sitemap at robots.txt or the usual locations."] }
  ]
}
```

Site statuses: `ok`, `partial`, `ok-all-filtered`, `blocked`, `no-sitemap`, `no-sitemap-crawl-empty`, `empty`, `failed`, `invalid-input`.

### How much does it cost to extract sitemap URLs?

Pay per URL extracted: you pay only for URLs that end up in your dataset (after filters and de-duplication), plus a tiny start fee per run. Prices fall with your Apify plan, from $0.20 per 1,000 URLs on the free plan to $0.08 per 1,000 on Gold and above. Set **Maximum cost per run** and the Actor stops cleanly at the limit. Use **Maximum URLs** or a date filter to preview a huge site cheaply.

### Use it from Python, JavaScript or the API

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("drop_data_solutions/sitemap-url-extractor").call(
    run_input={"domains": ["sitemaps.org"], "maxUrls": 1000}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item.get("lastmod"))
```

Schedule it, chain it into a scraper with an integration, or call it from AI agents through the Apify MCP server.

### Tips

- **Only want what changed?** Set **Modified on or after** to `7 days` and schedule the Actor weekly.
- **Blocked sites:** some sites answer 403 to datacenter IPs, even through Apify's datacenter proxy (in our tests: zalando.de, and Wikipedia's sitemap endpoint). They are reported as `blocked`. A Residential proxy may get through, but it isn't guaranteed.
- **Millions of URLs:** set **Maximum URLs** or a date filter first.

### FAQ

#### What if the website has no sitemap?

The report marks the site `no-sitemap`. With **Crawl the site when no sitemap exists** on (default), a light same-host crawl lists internal links instead.

#### Does it read sitemap index files and .xml.gz?

Yes. Nested indexes are followed recursively, and gzip is detected from the file's bytes, so mislabeled files work too.

#### Can it get past bot protection?

Not always. The Actor doesn't solve CAPTCHAs or JavaScript challenges. It retries rate limits politely and switches to Apify Proxy when a site blocks direct requests. If the site still refuses, the report marks it `blocked` and the run carries on with your other sites.

#### Is it legal to extract URLs from sitemaps?

Sitemaps are published so machines can read them. This Actor reads only public sitemap files and, in the fallback, public pages allowed by robots.txt. Make sure your use of the data complies with the site's terms and applicable law.

### Support

Found a site that doesn't work? Open an issue on the **Issues** tab with the URL. Built and maintained by drop_data_solutions.

# Actor input Schema

## `startUrls` (type: `array`):

Website URLs, robots.txt files or sitemap URLs (.xml, .xml.gz, .txt, RSS/Atom), with https://. For a website the Actor finds its sitemaps automatically. Bare domains go in the Domains field below.

## `domains` (type: `array`):

Bare domains or hosts, one per line, e.g. example.com or blog.example.com (https is tried first, then http). Same as adding them to Start URLs.

## `discoverSitemaps` (type: `boolean`):

For websites and domains, read robots.txt and, if it lists no sitemap, try common locations (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml, /sitemap.xml.gz and more).

## `maxUrls` (type: `integer`):

Stop after this many unique URLs. 0 = no limit (the run still stops at your Maximum cost per run).

## `maxUrlsPerDomain` (type: `integer`):

Stop each site after this many unique URLs. 0 = no limit.

## `includePatterns` (type: `array`):

Keep only URLs that match at least one of these regular expressions (case-insensitive), e.g. /blog/ or \\.html$.

## `excludePatterns` (type: `array`):

Drop URLs that match any of these regular expressions, e.g. /tag/ or \\?page=.

## `lastmodFrom` (type: `string`):

Keep only URLs whose sitemap lastmod is on or after this date. Use a date (2026-01-31) or a relative value (7 days, 3 months). Child sitemaps whose index lastmod is older are skipped entirely, which saves time.

## `lastmodTo` (type: `string`):

Keep only URLs whose sitemap lastmod is on or before this date.

## `keepUrlsWithoutLastmod` (type: `boolean`):

When a date filter is set, keep URLs whose sitemap entry has no lastmod. Turn off to keep only dated URLs.

## `sameHostOnly` (type: `boolean`):

Drop sitemap entries that point to other domains (www. is ignored when comparing).

## `includeAlternates` (type: `boolean`):

Add each URL's language versions (xhtml:link hreflang) as an alternates list.

## `includeMedia` (type: `boolean`):

Add image:image, video:video and news:news entries from the sitemap to each URL.

## `crawlIfNoSitemap` (type: `boolean`):

If no sitemap is found, follow the site's internal links (same host, robots.txt respected) to list URLs.

## `crawlMaxPages` (type: `integer`):

How many HTML pages to open per site in the fallback crawl. Same-host links found on them are output, except links disallowed by robots.txt and file links (images, PDFs, CSS/JS). URL limits still apply.

## `crawlMaxDepth` (type: `integer`):

How many clicks away from the start page to go.

## `proxyConfiguration` (type: `object`):

Used only when a site blocks direct requests (HTTP 401/403/429 or a bot challenge), unless you change Proxy mode. Choose Residential for sites behind strong bot protection.

## `proxyMode` (type: `string`):

on-block: go direct, switch to the proxy only when blocked (cheapest). always: every request through the proxy. never: no proxy.

## `maxConcurrency` (type: `integer`):

Sitemaps downloaded at once across all sites. At most 2 requests run against the same host at a time.

## `maxSitemapDepth` (type: `integer`):

How many levels of sitemap indexes to follow.

## `requestTimeoutSecs` (type: `integer`):

Give up on a request that is silent for this long. Every request has a connect and idle timeout, so a stalled server can't hold up the run indefinitely.

## `maxRequestRetries` (type: `integer`):

Retries on 429/5xx and network errors, with exponential backoff (1s, 2s, 4s…) that honours Retry-After.

## `userAgent` (type: `string`):

Optional custom User-Agent header. Defaults to a current desktop Chrome.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    },
    {
      "url": "https://www.sitemaps.org/sitemap.xml"
    }
  ],
  "domains": [],
  "discoverSitemaps": true,
  "maxUrls": 1000,
  "maxUrlsPerDomain": 0,
  "includePatterns": [],
  "excludePatterns": [],
  "keepUrlsWithoutLastmod": true,
  "sameHostOnly": false,
  "includeAlternates": true,
  "includeMedia": false,
  "crawlIfNoSitemap": true,
  "crawlMaxPages": 50,
  "crawlMaxDepth": 2,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "proxyMode": "on-block",
  "maxConcurrency": 4,
  "maxSitemapDepth": 6,
  "requestTimeoutSecs": 30,
  "maxRequestRetries": 3
}
```

# Actor output Schema

## `urls` (type: `string`):

One item per unique URL.

## `summary` (type: `string`):

OUTPUT record: totals and, for each site, status (ok, partial, blocked, no-sitemap, failed), how sitemaps were discovered, sitemaps found and failed with reasons.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        },
        {
            "url": "https://www.sitemaps.org/sitemap.xml"
        }
    ],
    "maxUrls": 1000,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("drop_data_solutions/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://apify.com" },
        { "url": "https://www.sitemaps.org/sitemap.xml" },
    ],
    "maxUrls": 1000,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("drop_data_solutions/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    },
    {
      "url": "https://www.sitemaps.org/sitemap.xml"
    }
  ],
  "maxUrls": 1000,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call drop_data_solutions/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,drop_data_solutions/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/e12ukK8Ad5rBmIWc1/builds/NgnIRP6lNPjc4zN5h/openapi.json