Sitemap Scraper

Turn any website's sitemap into clean JSON: every URL with lastmod, changefreq and priority. Follows sitemap index files and gzip. One click, no required fields, no LLM.

What it does

Almost every site on the web publishes a sitemap — the machine-readable list of its own URLs, declared in robots.txt and kept current by the site itself. It is the cheapest, most complete and most polite way to learn what pages a site has. This actor points at any domain and returns that list as clean JSON: canonical URL, last modification date, declared change frequency and crawl priority. It discovers the sitemap automatically (robots.txt first, then the common paths), resolves sitemap index files recursively, and decompresses .gz sitemaps — the three things that break naive scripts. No API key, no proxy, no browser.

Why this one:

• Automatic discovery: robots.txt first, then the common sitemap paths — no guessing on your side
• Sitemap index files resolved recursively, with a loop guard and source_sitemap traceability
• .gz sitemaps decompressed transparently
• Reads the site's own published file — no page rendering, no proxy, no API key
• No LLM anywhere — deterministic output, predictable costs, no hallucinated fields
• Clean by default — canonical URLs (tracking parameters stripped), parsed numbers, merged duplicates

Quick start (no code)

1. Click Try for free / Start — every field has a working default, nothing is required.
2. (Optional) change query to what you need.
3. Open the Dataset tab when the run finishes → export as JSON, CSV or Excel.

Input

Field	Required	Default	Description
query	no	https://apify.com	Any website address or a direct sitemap URL. Discovery is automatic: robots.txt first, then /sitemap.xml, /sitemap_index.xml and the other common paths.
maxResults	no	20	How many URLs to return (max 500). Large sites declare thousands, so this controls both time and cost.
maxResults	no	20	Maximum clean results (capped at 500)
enrich	no	false	Deterministic enrichment per record — see below
monitor	no	false	Compare with the previous run, flag NEW records only

Example input:

{
  "query": "https://www.bbc.com",
  "maxResults": 50
}

Output

Real example record (from a live run):

{
  "url": "http://www.bbc.co.uk/afrique/institutionelles/2014/09/140924_test.shtml",
  "title": "140924 test",
  "lastmod": "2014-09-24T14:32:16+00:00",
  "source_sitemap": "https://www.bbc.com/afrique/sitemap.xml",
  "domain": "www.bbc.co.uk",
  "completeness": 0.33
}

The final _summary row carries run totals (total_clean, deduped, enriched); in monitor mode a _changes row lists keys new since the last run.

Field reference

Field	Meaning
url	Canonical page URL from the sitemap <loc> element (deduplication key)
title	Label derived from the URL path — sitemaps carry no page title and this actor does not open pages, so this is a derivation, not the page's real <title>
lastmod	Last modification date declared by the site, when present
changefreq / priority	Declared change frequency and crawl priority, when present
source_sitemap	Which sitemap file the URL came from — index files are resolved recursively, so every URL stays traceable
domain / completeness	Canonical host and a 0–1 fill score (with enrich)
price / price_text	Parsed numeric value + original text, when the source publishes one
completeness	0–1 filled-fields score (with enrich)

Use cases

• Site audits — get the full URL inventory of a site before a migration, redesign or SEO review.
• Content monitoring — run on a schedule with monitor mode and get alerted only on NEW pages.
• Crawl seeding — feed a clean, de-duplicated URL list into your own crawler instead of discovering links page by page.
• Freshness analysis — use lastmod to find stale sections of a large site.
• AI agents — let an agent answer 'what pages does this site publish about X?' via MCP, without crawling.

Enrichment (optional, charged only when it produces something)

Set enrich: true and every record additionally gets: e-mail addresses extracted from the description (when present), the canonical domain of the record's URL, and a completeness score (0–1, how many core fields are filled). Deterministic — the same input always yields the same output — and you are only charged for records that actually got enriched. Records where enrichment adds nothing are free.

Monitor mode — change alerts on a schedule

Set monitor: true and the actor compares the current run with the previous one (per-actor named storage) and flags only NEW records. Combine with Apify Schedules for a daily/hourly watch: the _changes summary row lists what appeared since the last run, and the change-alert event is charged per new record only — an unchanged run costs you almost nothing.

Use it from your code

Python

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("EnezLi/sitemap-scraper").call(run_input={  "query": "https://www.bbc.com",  "maxResults": 50 })
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });
const { defaultDatasetId } = await client.actor('EnezLi/sitemap-scraper').call({  "query": "https://www.bbc.com",  "maxResults": 50 });
const { items } = await client.dataset(defaultDatasetId).listItems();
console.log(items);

curl

curl -X POST "https://api.apify.com/v2/acts/EnezLi~sitemap-scraper/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{  "query": "https://www.bbc.com",  "maxResults": 50 }'

Use it with AI agents (MCP)

This actor is agent-ready: it appears in Apify's AGENTS / MCP servers catalog, so any MCP-capable assistant (Claude, custom agents, LangGraph tools) can discover and call it with a one-line tool call — zero required fields means an agent can run it safely with defaults. Connect your agent to the Apify MCP server and ask for live data in natural language.

Pricing — Pay-Per-Event, start is free

Event	When charged
Actor start	Free ($0) — try it with one click
result	Per clean result returned
enrichment	Only per record that actually got enriched
change-alert	Monitor mode: per NEW record since the previous run

No subscription, no minimum. Volume discounts apply automatically through Apify account tiers (up to −44% on GOLD). Typical run cost example: 20 results ≈ a few cents total — you can predict your bill from the numbers above before you run.

Is this legal?

This actor collects publicly available data only — the same information any visitor sees in a browser, via public endpoints. It does not bypass logins, collect private personal data, or store credentials. You are responsible for using the output in compliance with the source site's terms and the laws that apply to you (e.g. GDPR when the output contains personal data).

Support & feedback

Found a bug, need another field, or want a variant for a related platform? Open an issue on the Issues tab — issues are monitored and answered, and frequently-requested fields get added to the standard output. The actor is maintained as part of a scraper family built on one shared, tested core (bugs fixed once are fixed everywhere).

Changelog

• 0.1 (2026-07) — initial public release: search, dedup, optional enrichment, monitor mode, PPE pricing.

Limitations (honest ones)

Only what the site declares is returned. Sitemaps are self-reported: a page missing from the sitemap will not appear, and lastmod is the site's claim, not a verified fact. Page titles and page content are out of scope — this actor reads the sitemap, it does not open pages.

FAQ

What if the site has no sitemap?

The run finishes cleanly with zero results and a short note. Nothing is invented — a missing sitemap is reported as missing, not filled with guesses.

Is title the real page title?

No, and the schema says so explicitly. Sitemaps do not carry titles and this actor does not fetch pages, so title is derived from the URL path. For root URLs it is left empty rather than guessed.

Does it handle sitemap index files?

Yes — recursively, up to 3 levels, with a loop guard so a self-referencing index cannot spin forever. Each URL records which file it came from.

Is scraping sitemaps allowed?

A sitemap is a file the site publishes for automated consumption and announces in robots.txt. Reading it is the intended use. This actor sends one polite request per file and identifies itself in the User-Agent.

Do I need an API key or account on the source platform?

No. The actor uses public endpoints — you only need your Apify account.

Does it use AI / an LLM?

No. The core is fully deterministic: same input, same output, no hallucinations, no per-token costs.

Can I run it on a schedule?

Yes — use Apify Schedules; combine with monitor mode to pay only for what's new.

What's the maximum number of results?

500 per run (memory-safe cap). Run multiple queries or schedule runs for more.

How is my bill calculated?

Only from the events in the Pricing table — start is free, and there is no subscription.