# Sitemap URL Extractor: All Pages, New & Removed URLs (`urlwright/sitemap-url-extractor`) Actor

Get every URL of a website: from sitemaps (index, gzip) or a polite crawl when there is none. Flags new and removed pages between runs, checks HTTP status, many domains per run. Respects robots.txt. $0.15–$0.30 per 1,000 URLs.

- **URL**: https://apify.com/urlwright/sitemap-url-extractor.md
- **Developed by:** [Urlwright](https://apify.com/urlwright) (community)
- **Categories:** Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.15 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor: All Pages, New & Removed URLs

**Urlwright** — *Little tools for the web.*

Plain-HTTP Apify Actor that discovers sitemaps, extracts URLs (indexes, nested sitemaps, `.gz`), optionally crawls the same host (or optional subdomains) when sitemaps are missing/empty, optionally diffs against the previous run, and optionally checks HTTP status. One bad domain never fails the whole run; a `RUN_SUMMARY` is written to the key-value store.

Default User-Agent: `UrlwrightBot/1.0 (sitemap; +https://urlwright.com/bot)` (`UrlwrightBot` for robots.txt matching).

### What it does

1. Accept domains, site origins, or direct sitemap URLs.
2. Discover sitemaps via `robots.txt` `Sitemap:` lines, then common paths if needed.
3. Parse urlsets and sitemap indexes (bounded depth), including gzip.
4. Optional **crawl fallback** (`crawlFallback`: `off` | `whenNoSitemap` | `always`): BFS for `href` + `rel=canonical` only — no page body stored, no personal data. Emits discovered in-scope links **and** each successfully crawled HTML page (including the start page) as `source: crawl` rows. **Always** honors `robots.txt` + crawl-delay per host (even if `respectRobotsTxt` is false for sitemaps). Optional `includeSubdomains` allows the same registrable domain (Public Suffix List, including private suffixes such as `github.io` / `vercel.app`). Crawl URL rows are flushed in batches during the crawl when change detection is off.
5. Optional **change detection** (`compareWithPrevious`): named key-value store per domain, `changeStatus` = `new` | `removed` | `unchanged`. Use with Apify Schedules; set `onlyChanges` to emit only new/removed (cheaper). Subdomain crawl URLs stay under the start domain’s snapshot.
6. Optionally HEAD/GET each URL for status, redirects, and final URL — **in blocks of 100**: check a block, push it, charge for that block, then continue (so a timeout mid-run still yields rows).
7. Dataset rows are written in **batches of 500** (trimmed to remaining pay-per-event / `maxUrls` budget).

### Input example

Store console prefill uses a small sample (`startUrls`: `https://docs.apify.com`, `maxUrls`: 100). The `maxUrls` default for real runs remains **10000**.

```json
{
  "startUrls": ["https://docs.apify.com", "https://other.example/sitemap.xml"],
  "checkStatus": true,
  "maxUrls": 5000,
  "maxSitemapDepth": 5,
  "crawlFallback": "whenNoSitemap",
  "includeSubdomains": false,
  "maxCrawlPages": 100,
  "maxCrawlDepth": 2,
  "compareWithPrevious": true,
  "onlyChanges": true,
  "concurrency": 5,
  "requestTimeoutSecs": 20,
  "requestsPerSecond": 2,
  "respectRobotsTxt": true
}
```

### Output

#### Output example (dataset item)

```json
{
  "url": "https://example.com/about",
  "lastmod": "2026-02-01",
  "changefreq": "monthly",
  "priority": "0.5",
  "sourceSitemap": "https://example.com/sitemap.xml",
  "source": "sitemap",
  "changeStatus": "new",
  "statusCode": 200,
  "finalUrl": "https://example.com/about",
  "redirectChain": ["https://example.com/about-old"]
}
```

`source` is `sitemap` or `crawl`.

#### Run summary

Key-value store record `RUN_SUMMARY` contains per-domain errors, crawl/change counts, totals, and truncation metadata:

- **`truncated`** — `true` when the run stopped early and did not finish the full inventory.
- **`stoppedReason`** (optional) — why it stopped:
  - `charge-limit` — pay-per-event budget / `maxTotalChargeUsd` exhausted.
  - `max-urls` — `maxUrls` cap reached.
  - `timeout` — cooperative stop shortly before the Actor run deadline so `RUN_SUMMARY` can still be written. A domain stopped for this reason is marked **`truncated`** (not `skippedReason: 'no-results-timeout'`, which is the separate 60s empty-domain rule).
- Per-domain **`truncated: true`** when that domain was incompletely written.
- **`changes.new` / `changes.removed` / `changes.unchanged`** — counts of dataset rows **actually written** with that `changeStatus`.
- **`changes.inventory.{new,removed,unchanged}`** — diff over the **found inventory**, capped by `maxUrls` when the domain was truncated (same inventory budget as on non-compare runs; large sites are not loaded in full just for compare). With `onlyChanges`, written `unchanged` is 0 while `inventory.unchanged` can be greater than 0.
- **`partial` / `removedCandidates`** — still set on incomplete compare runs (`removedCandidates` = `inventory.removed`); snapshot is not updated and removed rows are not emitted until the domain is complete.

**Named snapshot stores:** change-detection snapshots are written to separate named key-value stores (`c04-url-snapshot-<hostname>-<hash>`), not to the default store. The default store holds `RUN_SUMMARY`.

**Input dedup:** `startUrls` entries that resolve to the same target are processed once (first wins). Duplicate domain/origin inputs (same normalized origin) and duplicate direct sitemap URLs (same normalized URL) are skipped — no fetches, no charges — with a warning and `skippedDuplicate: true` in `RUN_SUMMARY`. A domain input and a direct sitemap URL for the same host are **not** duplicates of each other.

#### Scheduled change monitoring

1. Enable `compareWithPrevious` (and usually `onlyChanges`).
2. Create an [Apify Schedule](https://docs.apify.com/platform/actors/running/tasks#schedules) for this Actor / task.
3. Each run updates a named store and tags URLs with `changeStatus`. Store naming (Apify-safe, ≤63 chars):
   - **Domain/origin inputs** — `c04-url-snapshot-<hostname>-<hash-of-host>` (unchanged; existing scheduled monitors keep their store).
   - **Direct sitemap URL inputs** — same host slug, but the trailing hash is of the normalized sitemap URL, so two sitemap URLs on the same host get separate stores and do not overwrite each other’s snapshots.
     Per run, each snapshot store is read/written at most once (at most one `domain-compared` charge per store). Snapshots are written only when the domain inventory is complete; incomplete runs set `changes.partial`, keep written `changes.removed` at the rows actually emitted (usually 0; candidate count in `removedCandidates` / `inventory.removed`), and leave the previous snapshot unchanged.

### Pricing

Pay per event (USD). Recommended minimum **Max total charge** per run: **$0.02**.

| Event | When | FREE | BRONZE | SILVER | GOLD |
|---|---|---:|---:|---:|---:|
| `url-extracted` (primary) | Each dataset row, including synthetic `changeStatus: removed` rows | 0.0003 | 0.00025 | 0.0002 | 0.00015 |
| `url-status-checked` | After save, only when a real HTTP status was obtained (incl. 4xx/5xx) | 0.0003 | 0.00025 | 0.0002 | 0.00015 |
| `crawled-page` | Each successfully fetched HTML page during crawl fallback | 0.0025 | 0.002 | 0.0017 | 0.0015 |
| `domain-compared` | Once per domain after a complete snapshot update when a previous snapshot existed (not the first run; also when `onlyChanges` finds no diffs) | 0.002 | 0.002 | 0.002 | 0.002 |
| `domain-empty` | Once per input domain that yielded no URL rows (unreachable, no sitemap, blocked by robots.txt, crawl-delay skip, no-results timeout, empty sitemaps, or crawl that produced zero rows — e.g. non-HTML / disallowed start page with no other URLs) | 0.0005 | 0.0005 | 0.0005 | 0.0005 |
| `apify-actor-start` | Once per run | 0.0002 | 0.0002 | 0.0002 | 0.0002 |

Platinum and Diamond pay the Gold price.

**Bronze examples:** $0.25 / 1,000 URLs · $0.25 / 1,000 status checks · $2 / 1,000 crawled pages · $0.002 per monitoring compare · $0.0005 per empty domain.

### Limitations

- No browser / JavaScript-rendered sitemaps or SPAs.
- No personal data fields — only URLs and HTTP metadata (crawl stores no HTML body).
- `respectRobotsTxt` (default on) controls sitemap discovery, sitemap extraction, and status checks. **Crawl always respects `robots.txt`** (and crawl-delay) per host, including redirect hops and each subdomain’s own robots.txt.
- **Crawl-delay cap:** if `robots.txt` sets a crawl-delay **above 10 seconds** for our User-Agent, the domain is skipped (no sitemap/status/crawl requests beyond `robots.txt`; `skippedReason: 'crawl-delay'` in `RUN_SUMMARY`). Subdomains with an excessive crawl-delay are not crawled when `includeSubdomains` is on.
- **No-result timeout:** if a domain yields no URLs within **60 seconds**, processing stops (`skippedReason: 'no-results-timeout'`).
- **`requestsPerSecond`** minimum is **0.5** (default 2). Default memory 256 MB; max memory **512 MB**.
- **Private-network block:** if the start URL itself is rejected (hostname resolves to a private or special-use address, an IP literal, an IPv4-mapped IPv6 address, or a redirect onto one of those), the domain is `ssrf_blocked` in `RUN_SUMMARY` (`errorCode: SSRF_BLOCKED`). It is not reported as a robots.txt failure, “No sitemap found”, or `domain-empty`, and no event is charged. A nested sitemap or a later crawl page that is blocked does not change the rest of the domain; only that blocked request is skipped and not charged.
- **robots.txt fetch failures (RFC 9309):** network errors and HTTP 5xx when fetching `robots.txt` are treated as **disallow-all** (for crawl and, when `respectRobotsTxt` is on, for sitemap discovery/extraction/status). HTTP 4xx on robots.txt is allow-all. Such failures mark the domain inventory incomplete (`changes.partial`) so snapshots are not overwritten with false “everything removed” diffs. A normal empty discovery (“No sitemap found”) is separate from these reliability failures. A start URL blocked as `ssrf_blocked` is not this case.
- Crawl stays on the same hostname by default; with `includeSubdomains: true` it may follow subdomains of the same registrable domain (never foreign domains; private multi-tenant suffixes like `*.github.io` / `*.vercel.app` are separate registrable domains). HTML download capped at 2 MB per page. Link extraction ignores `<script>` / `<style>` contents and non-http(s) schemes (`javascript:`, `data:`, `mailto:`, `tel:`, `vbscript:`).
- **Content-Type:** crawl downloads a page body only when `Content-Type` is `text/html` or `application/xhtml+xml`. **Missing `Content-Type` is treated as non-HTML** (body not downloaded, page not charged as `crawled-page`).
- Before a crawl, part of the remaining crawl pay-per-event budget is reserved for later URL extraction charges. This reserve applies **only** to the pay-per-event budget, never to `maxCrawlPages`.
- Sitemap download/decompress capped at 50 MB per file.
- Very large sites: use `maxUrls` / `maxSitemapDepth` / `maxCrawlPages` / `maxCrawlDepth` to bound cost and runtime.
- Status checks are polite (per-host rate limit + crawl-delay) and run in blocks of 100 interleaved with dataset pushes.

### FAQ

#### Why is my dataset empty?

Common causes: the domain was unreachable or had no usable sitemap; crawl fallback was off or produced no in-scope HTML links; robots.txt / crawl-delay blocked the host; the 60s no-result timeout fired; or `onlyChanges` found nothing new or removed. Check `RUN_SUMMARY` for per-domain errors, `skippedReason`, and `domain-empty` charges.

#### What does a run with 1,000 URLs cost?

On **Bronze**, start + 1,000 extracted URLs (no status checks, no crawl): `0.0002 + 1000 × 0.00025 = $0.2502`. Status checks and crawled pages are charged separately if enabled.

#### Does the Actor respect robots.txt?

Yes for crawl — always, including crawl-delay, regardless of `respectRobotsTxt`. For sitemap discovery, sitemap extraction, and status checks, `respectRobotsTxt` defaults to on (you can turn it off for those paths only).

#### How does change detection work?

Enable `compareWithPrevious` (and usually `onlyChanges`). Each complete run stores a URL snapshot in a named key-value store and tags dataset rows with `changeStatus` (`new` / `removed` / `unchanged`). Schedule the Actor so successive runs compare against the previous snapshot. Incomplete runs do not overwrite the previous snapshot.

# Actor input Schema

## `startUrls` (type: `array`):

List of domains (example.com), site origins (https://example.com), or direct sitemap URLs (https://example.com/sitemap.xml).

## `checkStatus` (type: `boolean`):

When enabled, perform a polite HEAD (GET fallback) status check per extracted URL.

## `maxUrls` (type: `integer`):

Hard cap on URLs written to the dataset across the whole run.

## `maxSitemapDepth` (type: `integer`):

How deep to follow nested sitemap indexes (0 = only leaf urlsets from the first level).

## `crawlFallback` (type: `string`):

When to BFS-crawl for href/canonical links: off | whenNoSitemap (also if sitemap yields 0 URLs) | always. Crawl always honors robots.txt + crawl-delay per host. Default whenNoSitemap.

## `includeSubdomains` (type: `boolean`):

When true, crawl (including redirect hops) may follow subdomains of the same registrable domain (Public Suffix List, e.g. beispiel.de → shop.beispiel.de). Foreign domains are never followed. Each subdomain uses its own robots.txt. Default false = exact hostname only.

## `maxCrawlPages` (type: `integer`):

Max HTML pages to fetch during crawl fallback (PPE crawled-page charged per successful page).

## `maxCrawlDepth` (type: `integer`):

Max link depth from the site origin during crawl fallback (0 = homepage only).

## `compareWithPrevious` (type: `boolean`):

Store the URL set per domain in a named key-value store and set changeStatus (new|removed|unchanged). Ideal for Apify Schedules.

## `onlyChanges` (type: `boolean`):

When compareWithPrevious is on, write only new and removed URLs (skip unchanged). Cheaper for scheduled monitoring.

## `concurrency` (type: `integer`):

Max parallel HTTP requests for status checks, sitemap fetches, and crawl pages.

## `requestTimeoutSecs` (type: `integer`):

Timeout per HTTP request.

## `requestsPerSecond` (type: `number`):

Per-host polite rate limit (minimum 0.5). Combined with robots.txt crawl-delay when present (stricter wins). Domains whose robots.txt crawl-delay exceeds 10 seconds are skipped.

## `respectRobotsTxt` (type: `boolean`):

Honor Disallow / crawl-delay for sitemap discovery, sitemap URL extraction, and status checks. Crawl fallback always respects robots.txt (and crawl-delay) per host, regardless of this option. Always recommended.

## `userAgent` (type: `string`):

HTTP User-Agent used for all requests.

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com"
  ],
  "checkStatus": false,
  "maxUrls": 100,
  "maxSitemapDepth": 5,
  "crawlFallback": "whenNoSitemap",
  "includeSubdomains": false,
  "maxCrawlPages": 100,
  "maxCrawlDepth": 2,
  "compareWithPrevious": false,
  "onlyChanges": false,
  "concurrency": 5,
  "requestTimeoutSecs": 20,
  "requestsPerSecond": 2,
  "respectRobotsTxt": true,
  "userAgent": "UrlwrightBot/1.0 (sitemap; +https://urlwright.com/bot)"
}
```

# Actor output Schema

## `urls` (type: `string`):

Dataset items: sitemap/crawl URLs with optional status and changeStatus fields.

## `runSummary` (type: `string`):

JSON summary of the run (per-domain errors, crawl/change counts, totals, truncation).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com"
    ],
    "maxUrls": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("urlwright/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com"],
    "maxUrls": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("urlwright/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com"
  ],
  "maxUrls": 100
}' |
apify call urlwright/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,urlwright/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gheu8Z8YGwyeCh8Lc/builds/oTxifAX6VBqgwgrDT/openapi.json