# Sitemap Scraper - Extract All URLs to JSON (`enezli/sitemap-scraper`) Actor

Extract all URLs from any XML sitemap into clean JSON, including lastmod, changefreq and priority. Automatically follows sitemap index files and gzip compression. No API key required.

- **URL**: https://apify.com/enezli/sitemap-scraper.md
- **Developed by:** [Turgay NANTA](https://apify.com/enezli) (community)
- **Categories:** Agents, MCP servers, Developer tools
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Scraper

Turn any website's sitemap into clean JSON: every URL with lastmod, changefreq and priority. Follows sitemap index files and gzip. One click, no required fields, no LLM.

### What it does

Almost every site on the web publishes a **sitemap** — the machine-readable list of its own URLs, declared in `robots.txt` and kept current by the site itself. It is the cheapest, most complete and most polite way to learn what pages a site has. This actor points at any domain and returns that list as clean JSON: canonical URL, last modification date, declared change frequency and crawl priority. It discovers the sitemap automatically (robots.txt first, then the common paths), resolves **sitemap index files recursively**, and decompresses `.gz` sitemaps — the three things that break naive scripts. No API key, no proxy, no browser.

**Why this one:**

- **Automatic discovery: robots.txt first, then the common sitemap paths** — no guessing on your side
- ****Sitemap index files resolved recursively**, with a loop guard and `source_sitemap` traceability**
- **`.gz` sitemaps decompressed transparently**
- **Reads the site's own published file** — no page rendering, no proxy, no API key
- **No LLM anywhere** — deterministic output, predictable costs, no hallucinated fields
- **Clean by default** — canonical URLs (tracking parameters stripped), parsed numbers, merged duplicates

### Quick start (no code)

1. Click **Try for free** / **Start** — every field has a working default, nothing is required.
2. (Optional) change **query** to what you need.
3. Open the **Dataset** tab when the run finishes → export as JSON, CSV or Excel.

### Input

| Field | Required | Default | Description |
|---|---|---|---|
| `query` | no | `https://apify.com` | Any website address or a direct sitemap URL. Discovery is automatic: robots.txt first, then `/sitemap.xml`, `/sitemap_index.xml` and the other common paths. |
| `maxResults` | no | `20` | How many URLs to return (max 500). Large sites declare thousands, so this controls both time and cost. |
| `maxResults` | no | `20` | Maximum clean results (capped at 500) |
| `enrich` | no | `false` | Deterministic enrichment per record — see below |
| `monitor` | no | `false` | Compare with the previous run, flag NEW records only |

Example input:

```json
{
  "query": "https://www.bbc.com",
  "maxResults": 50
}
```

### Output

Real example record (from a live run):

```json
{
  "url": "http://www.bbc.co.uk/afrique/institutionelles/2014/09/140924_test.shtml",
  "title": "140924 test",
  "lastmod": "2014-09-24T14:32:16+00:00",
  "source_sitemap": "https://www.bbc.com/afrique/sitemap.xml",
  "domain": "www.bbc.co.uk",
  "completeness": 0.33
}
```

The final `_summary` row carries run totals (`total_clean`, `deduped`, `enriched`); in monitor mode a `_changes` row lists keys new since the last run.

**Field reference**

| Field | Meaning |
|---|---|
| `url` | Canonical page URL from the sitemap `<loc>` element (deduplication key) |
| `title` | Label **derived from the URL path** — sitemaps carry no page title and this actor does not open pages, so this is a derivation, not the page's real `<title>` |
| `lastmod` | Last modification date declared by the site, when present |
| `changefreq / priority` | Declared change frequency and crawl priority, when present |
| `source_sitemap` | Which sitemap file the URL came from — index files are resolved recursively, so every URL stays traceable |
| `domain / completeness` | Canonical host and a 0–1 fill score (with `enrich`) |
| `price` / `price_text` | Parsed numeric value + original text, when the source publishes one |
| `completeness` | 0–1 filled-fields score (with `enrich`) |

### Use cases

- **Site audits** — get the full URL inventory of a site before a migration, redesign or SEO review.
- **Content monitoring** — run on a schedule with monitor mode and get alerted only on NEW pages.
- **Crawl seeding** — feed a clean, de-duplicated URL list into your own crawler instead of discovering links page by page.
- **Freshness analysis** — use `lastmod` to find stale sections of a large site.
- **AI agents** — let an agent answer 'what pages does this site publish about X?' via MCP, without crawling.

### Enrichment (optional, charged only when it produces something)

Set `enrich: true` and every record additionally gets: e-mail addresses extracted from the description (when present), the canonical `domain` of the record's URL, and a `completeness` score (0–1, how many core fields are filled). Deterministic — the same input always yields the same output — and you are **only charged for records that actually got enriched**. Records where enrichment adds nothing are free.

### Monitor mode — change alerts on a schedule

Set `monitor: true` and the actor compares the current run with the previous one (per-actor named storage) and flags **only NEW records**. Combine with [Apify Schedules](https://docs.apify.com/platform/schedules) for a daily/hourly watch: the `_changes` summary row lists what appeared since the last run, and the `change-alert` event is charged **per new record only** — an unchanged run costs you almost nothing.

### Use it from your code

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("EnezLi/sitemap-scraper").call(run_input={  "query": "https://www.bbc.com",  "maxResults": 50 })
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

**JavaScript**

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });
const { defaultDatasetId } = await client.actor('EnezLi/sitemap-scraper').call({  "query": "https://www.bbc.com",  "maxResults": 50 });
const { items } = await client.dataset(defaultDatasetId).listItems();
console.log(items);
```

**curl**

```bash
curl -X POST "https://api.apify.com/v2/acts/EnezLi~sitemap-scraper/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{  "query": "https://www.bbc.com",  "maxResults": 50 }'
```

### Use it with AI agents (MCP)

This actor is agent-ready: it appears in Apify's **AGENTS / MCP servers** catalog, so any MCP-capable assistant (Claude, custom agents, LangGraph tools) can discover and call it with a one-line tool call — zero required fields means an agent can run it safely with defaults. Connect your agent to the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp) and ask for live data in natural language.

### Pricing — Pay-Per-Event, start is free

| Event | When charged |
|---|---|
| Actor start | **Free ($0)** — try it with one click |
| `result` | Per clean result returned |
| `enrichment` | Only per record that actually got enriched |
| `change-alert` | Monitor mode: per NEW record since the previous run |

No subscription, no minimum. Volume discounts apply automatically through Apify account tiers (up to −44% on GOLD). Typical run cost example: 20 results ≈ a few cents total — you can predict your bill from the numbers above before you run.

### Is this legal?

This actor collects **publicly available data only** — the same information any visitor sees in a browser, via public endpoints. It does not bypass logins, collect private personal data, or store credentials. You are responsible for using the output in compliance with the source site's terms and the laws that apply to you (e.g. GDPR when the output contains personal data).

### Support & feedback

Found a bug, need another field, or want a variant for a related platform? **Open an issue on the Issues tab** — issues are monitored and answered, and frequently-requested fields get added to the standard output. The actor is maintained as part of a scraper family built on one shared, tested core (bugs fixed once are fixed everywhere).

### Changelog

- **0.1** (2026-07) — initial public release: search, dedup, optional enrichment, monitor mode, PPE pricing.

### Limitations (honest ones)

Only what the site declares is returned. Sitemaps are self-reported: a page missing from the sitemap will not appear, and `lastmod` is the site's claim, not a verified fact. Page titles and page content are out of scope — this actor reads the sitemap, it does not open pages.

### FAQ

**What if the site has no sitemap?**

The run finishes cleanly with zero results and a short note. Nothing is invented — a missing sitemap is reported as missing, not filled with guesses.

**Is `title` the real page title?**

No, and the schema says so explicitly. Sitemaps do not carry titles and this actor does not fetch pages, so `title` is derived from the URL path. For root URLs it is left empty rather than guessed.

**Does it handle sitemap index files?**

Yes — recursively, up to 3 levels, with a loop guard so a self-referencing index cannot spin forever. Each URL records which file it came from.

**Is scraping sitemaps allowed?**

A sitemap is a file the site publishes for automated consumption and announces in robots.txt. Reading it is the intended use. This actor sends one polite request per file and identifies itself in the User-Agent.

**Do I need an API key or account on the source platform?**

No. The actor uses public endpoints — you only need your Apify account.

**Does it use AI / an LLM?**

No. The core is fully deterministic: same input, same output, no hallucinations, no per-token costs.

**Can I run it on a schedule?**

Yes — use Apify Schedules; combine with monitor mode to pay only for what's new.

**What's the maximum number of results?**

500 per run (memory-safe cap). Run multiple queries or schedule runs for more.

**How is my bill calculated?**

Only from the events in the Pricing table — start is free, and there is no subscription.

# Actor input Schema

## `query` (type: `string`):

Any website address (https://example.com) or a direct sitemap URL. The actor finds the sitemap automatically: it reads robots.txt first, then falls back to /sitemap.xml and the other common paths. Sitemap index files are resolved recursively and .gz files are decompressed.

## `maxResults` (type: `integer`):

Maximum number of URLs to return (capped at 500). Large sites publish index files with thousands of entries, so this is the knob that controls both time and cost.

## `enrich` (type: `boolean`):

Adds deterministic enrichment per record: canonical domain and a completeness score. Charged per enriched record (see Pricing).

## `monitor` (type: `boolean`):

Compares this run with the previous one and flags NEW records only. Ideal for scheduled runs — get alerted when something changes. Charged per change.

## Actor input object example

```json
{
  "query": "https://apify.com",
  "maxResults": 20,
  "enrich": false,
  "monitor": false
}
```

# Actor output Schema

## `results` (type: `string`):

Clean, normalized and de-duplicated records scraped from Sitemap, as JSON. Each item follows the dataset schema (see storages.dataset.fields in actor.json).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "https://apify.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("enezli/sitemap-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "query": "https://apify.com" }

# Run the Actor and wait for it to finish
run = client.actor("enezli/sitemap-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "https://apify.com"
}' |
apify call enezli/sitemap-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,enezli/sitemap-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0Ncvbxk9QbjScdC5D/builds/j3U5MuN7ScoxwMM3W/openapi.json