# Sitemap URL Extractor & Sitemap Validator (`kelp_forest_labs/sitemap-extractor`) Actor

Get every URL from a site's sitemaps with lastmod, changefreq and priority. Finds sitemaps via robots.txt, follows sitemap indexes, reads .xml.gz and text sitemaps, removes duplicates and flags invalid entries. An unreadable sitemap is billed as one URL; blocked hosts are free.

- **URL**: https://apify.com/kelp_forest_labs/sitemap-extractor.md
- **Developed by:** [kelp forest](https://apify.com/kelp_forest_labs) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Give it domains (or sitemap URLs). It finds every sitemap and returns **one row per unique page URL** with
`lastmod`, `changefreq`, `priority` and validation warnings.

### Use cases

- Build a URL list for a crawl, a migration or a content audit.
- Find stale pages by `lastmod`.
- Check that a new sitemap is valid before submitting it to search engines.
- Feed page lists to RAG or monitoring pipelines.

### What it does

- Discovers sitemaps from `robots.txt` `Sitemap:` lines, falling back to `/sitemap.xml` and `/sitemap_index.xml`
  (also when every sitemap robots.txt lists is broken; the broken ones are still reported).
- Follows sitemap indexes, reads gzip (`.xml.gz`) and plain-text sitemaps, removes duplicate URLs.
- Flags invalid `lastmod` dates, invalid `changefreq`, out-of-range `priority`, and URLs on a different host.
- Skips malformed entries (a `<loc>` or robots.txt `Sitemap:` line that is not a valid absolute URL) and reports
  each as a free row, while keeping every valid URL in the same file.
- Reports each broken sitemap (404, invalid XML, timeout) as its own row so you can fix it.
- Identifies itself with a named user agent and waits between sitemap files (default 0.5 s).
- A `SUMMARY` record per run: URLs per domain, `capped` with the reason (`maxUrlsPerDomain`,
  `maxSitemapsPerDomain` or `charge_limit`), sitemaps read and failed, and whether the fallback was used.
- Rows are saved in batches of up to 100; when your maximum charge for the run is reached, only the rows that were
  paid for are saved and the run stops.

### Pricing (pay per event)

| Event | Price |
|---|---|
| `url` (one unique URL row) | US$0.0005 |

**Read errors are billed like a URL row:** a sitemap that can't be fetched or parsed (404, invalid XML, timeout), and
a domain where no readable sitemap is found, are each billed as one `url` event (US$0.0005), following Apify's
pay-per-event guidance for input that looks valid but leads to an error. A domain is billed at most once for "no readable
sitemap found", and not at all for it when one of its broken sitemaps was already billed.
**Still free:** blocked or private addresses and DNS failures (nothing is fetched), invalid entries skipped inside a good
sitemap, and unexpected errors on our side. Each row says `not charged` when it is free.
Use `maxUrlsPerDomain` to cap spend.

Each run also has Apify's run-start charge of US$0.00005 per GB of run memory (US$0.00005 at the default 512 MB),
charged when the run starts, even if every domain fails.

### How to use

1. Open the actor in Apify Console and fill in the input form (or paste the JSON example below into the JSON editor).
2. Optional: set a maximum charge for the run under the run options to cap what a run can cost.
3. Click **Start**. Results appear in the **Output** tab as they are produced; export them as JSON, CSV, Excel or HTML.

You can also start runs from the Apify API or CLI, run the actor on a schedule (for example weekly), and send
results to other apps with Apify integrations and webhooks.

### Input example

```json
{"domains": ["docs.apify.com"], "maxUrlsPerDomain": 1000}
```

### Output example

```json
{"domain": "https://docs.apify.com", "url": "https://docs.apify.com/api", "lastmod": null, "changefreq": null, "priority": null, "sitemap": "https://docs.apify.com/sitemap_base.xml", "sitemapKind": "urlset", "warnings": [], "status": "ok"}
```

### FAQ

**Does it crawl the website?**
No. It reads only robots.txt and the sitemap files, which sites publish for machines.

**What if a site has no sitemap?**
You get one row saying no readable sitemap was found, with the locations tried. It is billed as one URL row
(US$0.0005); a domain that is blocked, private or does not resolve is free.

**How do I control cost?**
Set `maxUrlsPerDomain`. 1,000 URLs cost US$0.50. Each run adds Apify's run-start charge
(US$0.00005 per GB of memory, US$0.00005 at the default 512 MB).

**Can I give a sitemap URL directly?**
Yes, put the sitemap URL in `domains` or `startUrls`.

### Limits: what this actor does not do

- It does not crawl pages or check that each URL responds (that is a link checker's job).
- No private or loopback targets; redirects are re-checked at every hop.
- Each sitemap file must download within 60 seconds and be at most 50 MB uncompressed (the sitemap protocol
  limit); otherwise it is reported as an error row (billed as one URL row) and the next one goes ahead.

### Changelog

- October 8, 2026: a sitemap that can't be fetched or parsed, and a domain with no readable sitemap, are now
  billed as one URL row each (US$0.0005). Blocked, private and unresolvable targets stay free.
- 1.0 (October 2026): first public version.

# Actor input Schema

## `domains` (type: `array`):

e.g. example.com, https://example.com or https://example.com/sitemap.xml

## `startUrls` (type: `array`):

Same as domains, in Apify's start-URL format.

## `maxUrlsPerDomain` (type: `integer`):

Stops after this many unique URLs per domain (each URL row is billed).

## `maxSitemapsPerDomain` (type: `integer`):

Limit on sitemap files followed through indexes.

## `delaySecs` (type: `number`):

Politeness delay between requests to the same site.

## Actor input object example

```json
{
  "domains": [
    "docs.apify.com"
  ],
  "maxUrlsPerDomain": 100,
  "maxSitemapsPerDomain": 50,
  "delaySecs": 0.5
}
```

# Actor output Schema

## `results` (type: `string`):

One row per URL found in the site's sitemaps (lastmod, changefreq, priority, source sitemap), plus error rows (a sitemap or domain that could not be read is billed as one URL; blocked targets are free).

## `summary` (type: `string`):

URLs found per domain, whether the result was capped (capReason: maxUrlsPerDomain, maxSitemapsPerDomain or charge_limit), sitemaps read and failed, and whether the /sitemap.xml fallback was used.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "docs.apify.com"
    ],
    "maxUrlsPerDomain": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("kelp_forest_labs/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": ["docs.apify.com"],
    "maxUrlsPerDomain": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("kelp_forest_labs/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "docs.apify.com"
  ],
  "maxUrlsPerDomain": 100
}' |
apify call kelp_forest_labs/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kelp_forest_labs/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/eXfVZ2GvpCiw4O55M/builds/qeuwVyUtsokLx77NO/openapi.json