# Sitemap URL Extractor & Checker (`veedel_digital/sitemap-extractor`) Actor

Get every URL from a website's XML sitemaps: finds sitemaps via robots.txt, follows sitemap indexes, reads .xml.gz and text sitemaps. Returns lastmod, priority and hreflang, optionally checks each URL for 404s and redirects. No proxies.

- **URL**: https://apify.com/veedel_digital/sitemap-extractor.md
- **Developed by:** [Tim Tonat](https://apify.com/veedel_digital) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Sitemap URL Extractor & Checker do?

This Actor **extracts every URL from a website's XML sitemaps**. Enter a homepage, and it finds the sitemaps on its own (via `robots.txt` or the usual paths like `/sitemap.xml`), follows **sitemap indexes**, unpacks **.xml.gz** files and reads **text sitemaps**. You get **one row per URL** with `lastmod`, `changefreq`, `priority`, hreflang alternates and image count.

Optionally it **checks the HTTP status of every URL** and finds 404s, redirects and server errors that should never be in a sitemap.

No proxies and no browser, so runs are fast and cheap, even for sitemaps with hundreds of thousands of URLs.

### Why use this sitemap extractor?

- **Get a complete URL list of any website** in seconds, without crawling it
- **SEO audits**: find sitemap URLs that return 404, redirect or are blocked. Google treats these as sitemap errors
- **Content inventories and migrations**: export all URLs with their last modification date to Excel or Google Sheets
- **Monitor competitors**: schedule the Actor and see which pages a site adds, using `lastmod`
- **Feed other tools**: pass the URL list to a PageSpeed, SEO or scraping Actor
- **Filter by section** with include/exclude globs, e.g. only `/blog/**`
- **Pay per URL**: you control the cost with *Max URLs*

### What data does it return?

| Field | Description |
|---|---|
| `url` | URL listed in the sitemap |
| `lastmod`, `changefreq`, `priority` | Values from the sitemap, `null` if missing |
| `alternates` | hreflang alternates (`xhtml:link`) |
| `images` | Number of image entries for the URL |
| `sitemap` | Sitemap file the URL was found in |
| `statusCode` | HTTP status code (only with *Check HTTP status*) |
| `redirectTo` | Redirect target, if the URL redirects |
| `issues` | `REDIRECT`, `HTTP_404`, `HTTP_500`, … or `UNREACHABLE` |
| `error` | Reason if a sitemap could not be read |

The **Summary** (key-value store record `SUMMARY`) lists every sitemap file found with its URL count and errors, plus duplicates, URLs without `lastmod`, filtered URLs and status code counts.

### How to extract all URLs from a sitemap

1. Click **Try for free**.
2. Enter one or more homepages or sitemap URLs in **Websites or sitemap URLs**.
3. Set **Max URLs**.
4. Enable **Check HTTP status of every URL** if you want to find broken URLs.
5. Click **Start** and download the results as JSON, CSV, Excel or HTML.

### How much does it cost?

This Actor uses **pay per event**, and **platform usage is included**:

- a small price **per extracted URL**
- plus a price **per checked URL**, only if you enable the status check

The current prices are shown in the **Pricing** tab. URLs that can't be reached during the status check are not charged for the check. With *Max URLs* you always know the maximum cost in advance.

### Input

See the **Input** tab for all options.

```json
{
    "startUrls": ["https://example.com", "https://example.org/sitemap_index.xml"],
    "maxUrls": 10000,
    "checkStatus": true,
    "includeGlobs": ["https://example.com/blog/**"],
    "excludeGlobs": ["**/tag/**"]
}
```

### Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

```json
{
    "url": "https://example.com/blog/sitemap-guide/",
    "lastmod": "2026-10-01T08:30:00+00:00",
    "changefreq": "weekly",
    "priority": 0.8,
    "alternates": [{ "lang": "de", "url": "https://example.com/de/blog/sitemap-guide/" }],
    "images": 2,
    "sitemap": "https://example.com/post-sitemap.xml",
    "statusCode": 301,
    "redirectTo": "https://example.com/blog/sitemaps/",
    "issues": ["REDIRECT"],
    "error": null
}
```

### Related Actors

Use the URL list as input for the other audits:

- [SEO Site Audit & Broken Link Checker](https://apify.com/veedel_digital/seo-audit): crawl a whole website and find on-page SEO issues and broken pages
- [PageSpeed Insights & Core Web Vitals Bulk Audit](https://apify.com/veedel_digital/pagespeed-audit): Lighthouse scores and real-user Core Web Vitals for many URLs
- [Structured Data, Schema.org & Open Graph Extractor](https://apify.com/veedel_digital/structured-data): extract JSON-LD, schema.org types and Open Graph tags from many pages
- [PDF to Text with OCR](https://apify.com/veedel_digital/pdf-to-text): extract text from PDFs in bulk, scanned pages included

### FAQ

#### How does it find the sitemap?

It reads all `Sitemap:` lines in `robots.txt`. If there are none, it tries `/sitemap.xml`, `/sitemap_index.xml` and `/wp-sitemap.xml`. You can always enter the sitemap URL directly.

#### Why do some URLs show HTTP 403 in the status check?

Some large sites block automated requests with a bot protection and answer 403 to everything that isn't a browser. The URL is probably fine for visitors. The status check is most useful for your own sites and client sites.

#### Are there limits?

Sitemap files can be up to 50 MB (the limit of the sitemap protocol), and sitemap indexes are followed up to 5 levels deep. Each URL is returned only once, duplicates are counted in the summary.

#### Something does not work

Please open an issue in the **Issues** tab with the website and the run ID. You can also start the Actor programmatically from the **API** tab.

# Actor input Schema

## `startUrls` (type: `array`):

Homepages (the sitemap is found via robots.txt or the usual paths) or direct links to sitemap files (.xml, .xml.gz, .txt).

## `maxUrls` (type: `integer`):

Maximum number of URLs to return (each returned URL is charged).

## `checkStatus` (type: `boolean`):

Request every URL and report its status code and redirect target. Finds 404s, redirects and server errors listed in your sitemap. Charged separately per checked URL.

## `includeGlobs` (type: `array`):

Optional globs, e.g. https://example.com/blog/\*\* . `**` matches anything, `*` anything except /.

## `excludeGlobs` (type: `array`):

Optional globs, e.g. **/tag/** .

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev"
  ],
  "maxUrls": 10000,
  "checkStatus": false
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset with all URLs from the sitemaps

## `summary` (type: `string`):

Sitemaps found, duplicates, URLs without lastmod and status counts

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("veedel_digital/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://crawlee.dev"] }

# Run the Actor and wait for it to finish
run = client.actor("veedel_digital/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev"
  ]
}' |
apify call veedel_digital/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,veedel_digital/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3czTgLMypd2ObEIUm/builds/b41GLrxsli0h2Nnoo/openapi.json