# Sitemap Extractor: Every URL, Recursive, Reliable (`thoob/sitemap-extractor`) Actor
Reads sitemap.xml, sitemap index files, .gz compressed sitemaps, and robots.txt Sitemap directives, and returns one clean row per URL with lastmod, changefreq, and priority. Billed only per delivered URL.
- **URL**: https://apify.com/thoob/sitemap-extractor.md
- **Developed by:** [Pono Data](https://apify.com/thoob) (community)
- **Categories:** Developer tools
- **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
$0.08 / 1,000 extracted urls
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):
```bash
npm install apify-client
```
In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):
```bash
pip install apify-client
```
In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):
````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash
In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).
If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## Sitemap Extractor
Give it a sitemap URL, a robots.txt URL, or a site root. It returns one clean row
per URL, following sitemap index files, `.gz` compressed sitemaps, and the
`Sitemap:` directives in robots.txt. Every row carries the sitemap it came from,
so any row is verifiable at its source.
### What it does that other actors split across three
- **Finds the sitemap for you.** Give a bare site root and it probes the common sitemap locations, robots.txt, and the homepage's ``, then extracts what it finds. Other actors either find the sitemap or extract one, not both.
- **Validates each URL (optional).** Turn on status checking and every row gets an `httpStatus` and a `live` flag, checked concurrently. Filter to live URLs only for a list with no dead links.
- **Reads the sitemap extensions.** Image, news, video, and hreflang-alternate data are pulled out per URL when the sitemap carries them, not dropped.
Plus the basics done right: index recursion, `.gz`, lastmod filtering, dedupe, and a `sourceUrl` on every row.
### Input
- **Start URLs**: sitemap URLs, robots.txt URLs, or site roots. A site root is resolved by probing the common locations, robots.txt, and the homepage link.
- **Max delivered URLs**: cap on delivered rows (0 means no cap).
- **Max sitemap-index depth**: how deep to follow nested index files.
- **Only URLs modified on/after**: optional `YYYY-MM-DD` lastmod filter (entries with no lastmod are kept).
- **Include changefreq and priority**: toggle the optional hint fields.
- **Validate URL status codes**: add `httpStatus` and `live` to every row.
- **Deliver only live URLs**: with validation on, drop dead URLs to the free `rejected` dataset.
### Output
One row per URL: `loc`, `lastmod`, `changefreq`, `priority`, `host`, `depth`, the
optional `httpStatus` and `live`, the image / news / video / hreflang extension
fields when present, `sourceUrl` (the sitemap it came from), `retrievedAt`,
`confidence`, `dataSource`.
### How it works
Sitemaps are published by sites for machines to read. The actor fetches them with
a declared User-Agent, decompresses `.gz`, follows index files up to your depth
limit, dedupes by URL, and never invents a field: a value is emitted only if it is
present in the sitemap. A supplied sitemap that fails to fetch or parse is recorded
in the free `rejected` dataset; a probe of a guessed location that misses is
silent. Status checks, when on, run concurrently so validation stays fast.
### Billing
Pay per delivered URL row. Failed or unparseable sitemaps, and dead URLs when you
filter to live only, cost nothing. Status validation is included at the same
per-URL price.
### Coverage
Global. Targets in any country are processed. The one exclusion is jurisdictions
under US sanctions (Cuba, Iran, North Korea, Syria, Russia, Belarus, Venezuela,
Myanmar, matched by country-code TLD), which are written to the free `rejected`
dataset and never billed.
### Opt out
A domain owner can ask us to skip their domain at https://ponodata.com/opt-out .
Suppressed domains are returned by nothing and never billed.
### Sample output
A real run extracting URLs from a sitemap, with optional live-status validation:
| URL | HTTP | live | source |
| --- | --- | --- | --- |
| https://developer.mozilla.org/en-US/ | 200 | True | developer.mozilla… |
| https://developer.mozilla.org/en-US/404 | 200 | True | developer.mozilla… |
| https://developer.mozilla.org/en-US/about | 200 | True | developer.mozilla… |
| https://developer.mozilla.org/en-US/advertising | 200 | True | developer.mozilla… |
Every URL carries its `sourceUrl` (the sitemap it came from), for example `https://developer.mozilla.org/sitemaps/en-us/sitemap.xml.gz`.
### See also
More clean, pay-only-for-results data tools from Pono Data:
- [URL Metadata & OpenGraph Extractor](https://apify.com/thoob/url-metadata-extractor) - title, OpenGraph, and meta tags per page
- [Bulk DNS Lookup](https://apify.com/thoob/dns-bulk-lookup) - DNS records plus SPF, DMARC, and CAA
- [Domain WHOIS via RDAP](https://apify.com/thoob/rdap-domain-lookup) - registration data, structured from RDAP
Full catalog: https://apify.com/thoob
# Actor input Schema
## `startUrls` (type: `array`):
Sitemap URLs (https://site.com/sitemap.xml), robots.txt URLs, or site roots (https://site.com). For a site root the actor probes the common sitemap locations, robots.txt, and the homepage's . Sitemap index files and .gz compressed sitemaps are followed automatically.
## `maxUrls` (type: `integer`):
Safety cap on delivered and billed URL rows. 0 means no cap. The platform spend cap (ACTOR_MAX_TOTAL_CHARGE_USD) is honored regardless.
## `maxDepth` (type: `integer`):
How deep to follow nested sitemap index files. 0 means only the sitemaps you supply.
## `lastmodAfter` (type: `string`):
Optional. Keep only entries whose lastmod is on or after this date. Entries with no lastmod are kept.
## `includeSitemapMeta` (type: `boolean`):
Include the optional changefreq and priority fields from each entry.
## `checkStatus` (type: `boolean`):
Check each extracted URL's HTTP status (HEAD with a GET fallback) and add httpStatus and live to every row. Adds time on large sitemaps; runs concurrently.
## `onlyLive` (type: `boolean`):
When validating status, deliver and bill only URLs that respond 2xx or 3xx. Dead URLs go to the free rejected dataset. Ignored when status validation is off.
## Actor input object example
```json
{
"startUrls": [
"https://www.cloudflare.com/sitemap.xml"
],
"maxUrls": 0,
"maxDepth": 5,
"includeSitemapMeta": true,
"checkStatus": false,
"onlyLive": false
}
````
# Actor output Schema
## `urls` (type: `string`):
One row per URL discovered across the supplied sitemaps, with optional status and extension data.
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"startUrls": [
"https://www.cloudflare.com/sitemap.xml"
]
};
// Run the Actor and wait for it to finish
const run = await client.actor("thoob/sitemap-extractor").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = { "startUrls": ["https://www.cloudflare.com/sitemap.xml"] }
# Run the Actor and wait for it to finish
run = client.actor("thoob/sitemap-extractor").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"startUrls": [
"https://www.cloudflare.com/sitemap.xml"
]
}' |
apify call thoob/sitemap-extractor --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=thoob/sitemap-extractor",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
```json
{
"openapi": "3.0.1",
"info": {
"title": "Sitemap Extractor: Every URL, Recursive, Reliable",
"description": "Reads sitemap.xml, sitemap index files, .gz compressed sitemaps, and robots.txt Sitemap directives, and returns one clean row per URL with lastmod, changefreq, and priority. Billed only per delivered URL.",
"version": "0.0",
"x-build-id": "nff9sN6g5yAmuAcoA"
},
"servers": [
{
"url": "https://api.apify.com/v2"
}
],
"paths": {
"/acts/thoob~sitemap-extractor/run-sync-get-dataset-items": {
"post": {
"operationId": "run-sync-get-dataset-items-thoob-sitemap-extractor",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/acts/thoob~sitemap-extractor/runs": {
"post": {
"operationId": "runs-sync-thoob-sitemap-extractor",
"x-openai-isConsequential": false,
"summary": "Executes an Actor and returns information about the initiated run in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/runsResponseSchema"
}
}
}
}
}
}
},
"/acts/thoob~sitemap-extractor/run-sync": {
"post": {
"operationId": "run-sync-thoob-sitemap-extractor",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
}
},
"components": {
"schemas": {
"inputSchema": {
"type": "object",
"required": [
"startUrls"
],
"properties": {
"startUrls": {
"title": "Start URLs",
"minItems": 1,
"type": "array",
"description": "Sitemap URLs (https://site.com/sitemap.xml), robots.txt URLs, or site roots (https://site.com). For a site root the actor probes the common sitemap locations, robots.txt, and the homepage's . Sitemap index files and .gz compressed sitemaps are followed automatically.",
"items": {
"type": "string"
}
},
"maxUrls": {
"title": "Max delivered URLs",
"minimum": 0,
"maximum": 1000000,
"type": "integer",
"description": "Safety cap on delivered and billed URL rows. 0 means no cap. The platform spend cap (ACTOR_MAX_TOTAL_CHARGE_USD) is honored regardless.",
"default": 0
},
"maxDepth": {
"title": "Max sitemap-index depth",
"minimum": 0,
"maximum": 20,
"type": "integer",
"description": "How deep to follow nested sitemap index files. 0 means only the sitemaps you supply.",
"default": 5
},
"lastmodAfter": {
"title": "Only URLs modified on/after (YYYY-MM-DD)",
"type": "string",
"description": "Optional. Keep only entries whose lastmod is on or after this date. Entries with no lastmod are kept."
},
"includeSitemapMeta": {
"title": "Include changefreq and priority",
"type": "boolean",
"description": "Include the optional changefreq and priority fields from each entry.",
"default": true
},
"checkStatus": {
"title": "Validate URL status codes",
"type": "boolean",
"description": "Check each extracted URL's HTTP status (HEAD with a GET fallback) and add httpStatus and live to every row. Adds time on large sitemaps; runs concurrently.",
"default": false
},
"onlyLive": {
"title": "Deliver only live URLs",
"type": "boolean",
"description": "When validating status, deliver and bill only URLs that respond 2xx or 3xx. Dead URLs go to the free rejected dataset. Ignored when status validation is off.",
"default": false
}
}
},
"runsResponseSchema": {
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"actId": {
"type": "string"
},
"userId": {
"type": "string"
},
"startedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"finishedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"status": {
"type": "string",
"example": "READY"
},
"meta": {
"type": "object",
"properties": {
"origin": {
"type": "string",
"example": "API"
},
"userAgent": {
"type": "string"
}
}
},
"stats": {
"type": "object",
"properties": {
"inputBodyLen": {
"type": "integer",
"example": 2000
},
"rebootCount": {
"type": "integer",
"example": 0
},
"restartCount": {
"type": "integer",
"example": 0
},
"resurrectCount": {
"type": "integer",
"example": 0
},
"computeUnits": {
"type": "integer",
"example": 0
}
}
},
"options": {
"type": "object",
"properties": {
"build": {
"type": "string",
"example": "latest"
},
"timeoutSecs": {
"type": "integer",
"example": 300
},
"memoryMbytes": {
"type": "integer",
"example": 1024
},
"diskMbytes": {
"type": "integer",
"example": 2048
}
}
},
"buildId": {
"type": "string"
},
"defaultKeyValueStoreId": {
"type": "string"
},
"defaultDatasetId": {
"type": "string"
},
"defaultRequestQueueId": {
"type": "string"
},
"buildNumber": {
"type": "string",
"example": "1.0.0"
},
"containerUrl": {
"type": "string"
},
"usage": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "integer",
"example": 1
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
},
"usageTotalUsd": {
"type": "number",
"example": 0.00005
},
"usageUsd": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "number",
"example": 0.00005
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
}
}
}
}
}
}
}
}
```