# Sitemap Scraper (`clintsa/sitemap-scraper`) Actor

Extract all URLs from any website's sitemap.xml with robots.txt discovery, gzip support, full metadata, and automatic retries

- **URL**: https://apify.com/clintsa/sitemap-scraper.md
- **Developed by:** [Andy Besos](https://apify.com/clintsa) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

The Sitemap Scraper extracts every URL from any website's sitemap.xml — including nested sitemap indexes, gzipped sitemaps, and sitemaps discovered via robots.txt. It returns each URL with full metadata: lastmod, changefreq, priority, and the source sitemap file it came from.

No browser, no API key, no proxy required. The actor works with plain HTTP requests and handles 429 rate limits with automatic exponential backoff. It streams large sitemaps without loading them entirely into memory, so it stays fast and reliable even on sites with hundreds of thousands of URLs.

### Why use this Sitemap Scraper?

- **robots.txt discovery** — automatically finds sitemaps declared in robots.txt, plus falls back to common paths like `/sitemap.xml`
- **Recursive sitemap index handling** — follows nested `<sitemapindex>` files to any depth, processing all child sitemaps
- **Gzip support** — transparently decompresses `.xml.gz` sitemaps without extra configuration
- **Full metadata per URL** — extracts `lastmod`, `changefreq`, `priority`, and `sourceSitemap` (which sitemap file each URL came from)
- **Source sitemap tracking** — every URL includes the exact sitemap file it was found in, essential for debugging and auditing large sites
- **429 retry with exponential backoff** — automatically retries rate-limited requests up to 5 times with increasing delays
- **Streaming XML parsing** — uses a streaming parser for memory efficiency on large sitemaps
- **URL deduplication** — automatically deduplicates URLs found in multiple sitemaps
- **Optional HTTP status check** — enable HEAD requests to verify each URL returns a valid status code
- **Change monitoring** — track new, updated, and removed URLs across scheduled runs

In our 22 August 2026 comparison of the first 20 sitemap-related Apify Store results, the market leader `apify/sitemap-extractor` had a 17.7% failure rate (220/1,241 runs in 30 days). This actor targets zero failures through proper retry logic and error handling.

### Pricing

$0.001 per run plus $0.0004 per URL extracted.

| URLs extracted | Estimated cost |
| ---: | ---: |
| 1 | $0.0014 |
| 100 | $0.041 |
| 1,000 | $0.401 |
| 10,000 | $4.001 |

The per-result model means small sites cost almost nothing, while large sites pay proportionally. No hidden proxy or compute surcharges.

Prices above cover the Actor's chargeable run and result events. Standard Apify charges for retaining or downloading data after a run may apply.

### How to use this Actor

Provide one or more website base URLs. The actor automatically discovers sitemaps via robots.txt and common paths. You can also paste direct sitemap URLs.

#### Quick start — extract all URLs from a website

```json
{
  "targets": ["https://apify.com"]
}
```

The prefilled example returns one URL to keep the first test fast and cheap. Set `maxItemsPerTarget: 0` to get everything.

#### Direct sitemap URL

```json
{
  "targets": ["https://example.com/sitemap.xml"]
}
```

#### Multiple targets with status checking

```json
{
  "targets": ["https://apify.com", "https://example.com"],
  "includeStatusCheck": true,
  "maxItemsPerTarget": 500
}
```

### Monitor new, updated, and removed records

Every successful run stores a snapshot in a named key-value store. Run the Actor on a schedule with the same `monitorId` and targets to get a change feed.

```json
{
  "targets": ["https://example.com"],
  "monitorId": "daily-check",
  "onlyChangesSince": "2026-08-01T00:00:00.000Z"
}
```

The first run creates the baseline and labels every record `new`. Later runs compare against it:

| `changeType` | Meaning |
| --- | --- |
| `new` | URL appeared for the first time |
| `updated` | URL metadata changed (e.g. `lastmod` changed); `changedFields` lists them |
| `unchanged` | URL and metadata are the same as last run |
| `removed` | URL disappeared from the sitemap |

When `onlyChangesSince` is set, unchanged records are excluded. A removal is emitted once, on the run that first detects it.

### API example

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/daXenlo~sitemap-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "targets": ["https://apify.com"],
    "maxItemsPerTarget": 10
  }'
```

For longer runs, start the Actor asynchronously and read its default dataset after the run succeeds.

### Input options

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `targets` | string\[] | required | Website URLs or direct sitemap URLs |
| `maxItemsPerTarget` | integer | 0 (all) | Limit URLs per target; `0` returns everything |
| `maxDepth` | integer | 5 | Max recursion depth for nested sitemap indexes |
| `maxSitemaps` | integer | 100 | Max sitemap files to process per target |
| `timeoutMillis` | integer | 30000 | HTTP request timeout in milliseconds |
| `concurrency` | integer | 3 | Number of concurrent HTTP requests |
| `includeStatusCheck` | boolean | false | Run HEAD requests for HTTP status codes |
| `monitorId` | string | "default" | Namespace for change tracking |
| `onlyChangesSince` | ISO timestamp | null | Filter to only return changes since this timestamp |
| `useApifyProxy` | boolean | false | Enable Apify Proxy (usually not needed) |

### Output

One dataset item per extracted URL:

```json
{
  "target": "https://apify.com",
  "url": "https://apify.com/blog",
  "lastmod": "2026-08-15",
  "changefreq": "daily",
  "priority": 0.8,
  "sourceSitemap": "https://apify.com/sitemap.xml",
  "statusCode": null,
  "sourceUrl": "https://apify.com",
  "scrapedAt": "2026-08-22T10:00:00.000Z",
  "changeType": "new",
  "changeDetectedAt": "2026-08-22T10:00:00.000Z",
  "firstSeenAt": "2026-08-22T10:00:00.000Z",
  "lastSeenAt": "2026-08-22T10:00:00.000Z"
}
```

The run also writes a `SUMMARY` record with requested, successful, and failed target counts, the exported URL count, change totals, and completion time.

### Use cases

- **SEO audits** — extract all indexable URLs a site exposes via sitemaps for coverage analysis
- **Site migrations** — generate a complete URL inventory to build redirect maps before cutover
- **Content inventory** — catalog all pages with lastmod dates for freshness analysis
- **Broken link discovery** — combine with HTTP status checks to find 404s in sitemaps
- **Competitive research** — map a competitor's content structure and publication frequency
- **Crawl seed lists** — produce clean, deduplicated URL lists for downstream scrapers or LLM pipelines

### FAQ

#### How does it discover sitemaps?

The actor first checks `robots.txt` for `Sitemap:` directives. If none are found, it tries common paths like `/sitemap.xml`, `/sitemap_index.xml`, and `/sitemap.xml.gz`. You can also provide direct sitemap URLs in the `targets` field.

#### Do I need an account or API key for the target website?

No. Sitemaps are public XML files served over HTTP. The actor only reads publicly accessible sitemap endpoints.

#### Does it work with sitemap index files?

Yes. The actor recursively follows `<sitemapindex>` entries to any depth (configurable via `maxDepth`). Each nested sitemap is fetched and parsed, and all URLs are collected.

#### What about gzipped sitemaps?

Fully supported. The actor transparently decompresses `.xml.gz` files using Node's built-in zlib.

#### How does it handle rate limiting?

The actor automatically retries 429 responses up to 5 times with exponential backoff (1s, 2s, 4s, 8s, 16s). You can also reduce `concurrency` to be gentler on the target server.

#### Can I check if URLs in the sitemap are still live?

Yes. Enable `includeStatusCheck` to run HTTP HEAD requests on each URL. The `statusCode` field in the output tells you the HTTP response code.

#### Can this Actor detect removed URLs?

It reports a removal when a URL present in the previous snapshot is absent from the next successful run using the same `monitorId`.

### Limitations and responsible use

- The actor only reads publicly accessible sitemap XML files. It does not crawl page content or bypass authentication.
- Sitemap data is provided by the website owner and may not reflect the actual site structure. Some URLs in sitemaps may return errors or redirects.
- The actor respects rate limits and uses backoff. It does not attempt to bypass anti-bot protections.
- Use this tool in compliance with the target website's terms of service.

### Support

Report a problem or request a feature through the Issues tab on this Actor's page. Include the run ID and the input you used.

# Actor input Schema

## `targets` (type: `array`):

Required. One or more website base URLs (e.g. `https://example.com`). The actor automatically discovers sitemaps via robots.txt and common paths (`/sitemap.xml`, `/sitemap_index.xml`). You can also paste direct sitemap URLs like `https://example.com/sitemap.xml`. URLs are normalized automatically.

## `maxItemsPerTarget` (type: `integer`):

Limit the number of URLs returned per target. Set to `0` for all URLs. The prefilled health-check input returns one URL.

## `maxDepth` (type: `integer`):

How deep to follow nested sitemap indexes. `1` = only the top-level sitemap. `3` = up to 3 levels of sitemap indexes. Increase for sites with deeply nested sitemap structures.

## `maxSitemaps` (type: `integer`):

Safety limit on the total number of sitemap files to fetch per target. Prevents runaway on sites with thousands of nested sitemaps. Set to `0` for unlimited.

## `timeoutMillis` (type: `integer`):

Timeout per HTTP request in milliseconds. Increase for slow sites.

## `concurrency` (type: `integer`):

Number of concurrent HTTP requests for fetching sitemaps. Higher is faster but may trigger rate limits. `1` is safest.

## `includeStatusCheck` (type: `boolean`):

When enabled, performs an HTTP HEAD request for each extracted URL to get the HTTP status code. Adds the `statusCode` field to output. Slower but useful for finding broken URLs.

## `monitorId` (type: `string`):

Persistent namespace for change tracking. Keep the same value for the same targets and filters. Use another value for an independent monitor or schedule.

## `onlyChangesSince` (type: `string`):

Optional ISO 8601 timestamp such as `2026-08-01T00:00:00.000Z`. When set, the dataset contains only records detected as new, updated, or removed since that time. The first run creates the baseline.

## `useApifyProxy` (type: `boolean`):

Sitemaps are public XML and normally work without a proxy. Enable only as a network fallback if you encounter IP-based blocking.

## `proxyConfiguration` (type: `object`):

Optional proxy configuration used only when `useApifyProxy` is enabled.

## Actor input object example

```json
{
  "targets": [
    "https://apify.com"
  ],
  "maxItemsPerTarget": 1,
  "maxDepth": 5,
  "maxSitemaps": 100,
  "timeoutMillis": 30000,
  "concurrency": 3,
  "includeStatusCheck": false,
  "monitorId": "default",
  "useApifyProxy": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "targets": [
        "https://apify.com"
    ],
    "maxItemsPerTarget": 1
};

// Run the Actor and wait for it to finish
const run = await client.actor("clintsa/sitemap-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "targets": ["https://apify.com"],
    "maxItemsPerTarget": 1,
}

# Run the Actor and wait for it to finish
run = client.actor("clintsa/sitemap-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "targets": [
    "https://apify.com"
  ],
  "maxItemsPerTarget": 1
}' |
apify call clintsa/sitemap-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,clintsa/sitemap-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/LflNswvWXkx77M8Yk/builds/hLU11AK4W10JAgxvh/openapi.json