# Sitemap URL Extractor — every URL in a site's sitemaps (`subimpact/sitemap-url-extractor`) Actor

Extract every URL from a site's XML sitemaps: robots.txt discovery, nested sitemap indexes, gzip and RSS/Atom feeds, with lastmod, changefreq, priority, image count and hreflang alternates. Optional HTTP status check per URL. $0.30 per 1,000 URLs.

- **URL**: https://apify.com/subimpact/sitemap-url-extractor.md
- **Developed by:** [subimpact](https://apify.com/subimpact) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor — every URL in a site's sitemaps

Get **every URL on a website** as JSON, from the source that websites publish for exactly this
purpose: their sitemaps. Point it at a domain and it finds the sitemaps itself through
`robots.txt`, follows sitemap indexes to their children, unpacks `.xml.gz`, and reads RSS, Atom
and plain-text sitemap files too.

One dataset row per URL, with the sitemap it came from, its `lastmod`, `changefreq`, `priority`,
image count and hreflang alternates. Optionally add one HTTP status code per URL so you can spot
404s and redirects in the same run.

Pay only for what you get: **$0.30 per 1,000 URLs**. No start fee, no monthly minimum.

- **Sitemap discovery done for you.** `robots.txt` `Sitemap:` lines first (all of them, mixed case
  and relative paths included), then `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml` and
  `/wp-sitemap.xml`. Direct sitemap URLs are also accepted and read as-is.
- **Nested indexes, gzip and feeds.** Sitemap indexes are followed up to 5 levels deep (Google's
  own limit), `.xml.gz` files are gunzipped, and RSS/Atom feeds listed as sitemaps are parsed as
  URL sources.
- **The extra fields nobody exports.** `lastmod`, `changefreq`, `priority`, the number of
  `<image:image>` entries per URL (plus the image URLs themselves) and the `xhtml:link` hreflang
  alternates that multilingual sites publish.
- **One bad sitemap never kills a run.** A sitemap that 404s, times out or turns out to be an HTML
  error page becomes one free `error` row and the run carries on with the next sitemap.
- **Filters before you pay.** Regex include/exclude patterns and a `lastmodAfter` cutoff decide
  what is delivered, so you are only charged for URLs that reach your dataset.
- **Built for big sitemaps.** Sitemaps are parsed as a stream: a 50,000-URL sitemap file never has
  to fit in memory, and delivery starts while the file is still downloading. Measured peak memory
  in the runs below: about 100 MB for 1,000 URLs, 125 MB for 20,000.

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `startUrls` | array of strings | prefill `["https://www.smashingmagazine.com"]` | Website roots (`https://example.com`) or direct sitemap URLs (`https://example.com/sitemap.xml`). At least one |
| `includePatterns` | array of strings | `[]` | Keep only URLs matching at least one JavaScript regular expression, e.g. `/blog/`. Empty = keep everything |
| `excludePatterns` | array of strings | `[]` | Drop URLs matching any regular expression, e.g. `\.(jpg\|png\|pdf)$`. Applied after the includes |
| `lastmodAfter` | string | `""` | ISO date (`2026-01-31` or `2026-01-31T12:00:00Z`). Keeps entries whose `lastmod` is on or after it; entries with no `lastmod` are dropped while this is set |
| `maxUrls` | integer | `1000` | Hard cap on delivered URL rows, 1 to 1,000,000. The run stops at the cap |
| `checkStatus` | boolean | `false` | Send a HEAD request (GET fallback when the server refuses HEAD) per delivered URL and fill `statusCode`. Billed separately |
| `maxSitemaps` | integer | `200` | How many sitemap files one run may read, index children included, 1 to 10,000 |
| `requestTimeoutSecs` | integer | `60` | How long to wait for a sitemap or `robots.txt` response, 5 to 300 seconds |
| `proxyConfiguration` | object | not set | Optional Apify proxy for sites that block datacenter IPs. Sitemaps are public, so the default is a direct connection |

Invalid input fails the run immediately with a clear message, before any request and before any
charge: a `maxUrls` of 0, an unparseable regex, a `lastmodAfter` that is not an ISO date, or a
start URL like `mailto:...`. A start URL that is not usable is also delivered as one free `error`
row while the other start URLs still run.

### Example inputs

A whole site, the pre-filled example (1,000 URLs maximum):

```json
{
  "startUrls": ["https://www.smashingmagazine.com"]
}
```

Only blog posts, no images or PDFs, 2,000 URLs:

```json
{
  "startUrls": ["https://blog.apify.com"],
  "includePatterns": ["^https://blog\\.apify\\.com/"],
  "excludePatterns": ["\\.(jpg|jpeg|png|webp|pdf)$"],
  "maxUrls": 2000
}
```

Sitemap content that changed this year, with a status code per URL:

```json
{
  "startUrls": ["https://example.com/sitemap.xml"],
  "lastmodAfter": "2026-01-01",
  "checkStatus": true,
  "maxUrls": 500
}
```

A sitemap on a site that only serves datacenter IPs through a proxy:

```json
{
  "startUrls": ["https://example.com"],
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}
```

### Output

| Field | Type | Notes |
|---|---|---|
| `url` | string or null | Absolute URL from the sitemap, fragment stripped. `null` on error rows |
| `sitemapUrl` | string or null | The sitemap file this URL came from (after redirects) |
| `lastmod` | string or null | The `<lastmod>` value exactly as published (W3C datetime, date-only, or RSS `pubDate` / Atom `updated`) |
| `changefreq` | string or null | `always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly`, `never` |
| `priority` | number or null | The sitemap `<priority>`, 0 to 1 |
| `images` | integer or null | How many `<image:image>` entries the sitemap lists for this URL (`0` when it lists none) |
| `imageUrls` | array or null | The `<image:loc>` values for this URL, in sitemap order |
| `alternates` | array or null | `xhtml:link rel="alternate"` entries: `{ hreflang, href }` per language variant |
| `statusCode` | integer or null | With `checkStatus`: the first HTTP status code, redirects **not** followed (so `301`/`302` shows a redirect). `null` when the check is off or failed |
| `scrapedAt` | string | ISO 8601 timestamp of delivery |
| `status` | string | `ok` for a URL row, `error` for a sitemap or discovery step that could not be read |
| `error` | string or null | Why a sitemap could not be read (HTTP status, timeout, "not a sitemap", empty). `null` on URL rows |

Real rows from the live runs on 2026-11-15 (`node test/live.mjs https://blog.apify.com/sitemap-posts.xml 20`):

```json
{
  "url": "https://blog.apify.com/black-friday-checklist/",
  "sitemapUrl": "https://blog.apify.com/sitemap-posts.xml",
  "lastmod": "2026-11-13T15:02:27.000Z",
  "changefreq": null,
  "priority": null,
  "images": 1,
  "imageUrls": ["https://storage.ghost.io/c/f2/6e/f26ec999-9a90-4aee-a0d4-9b3ca2bb668f/content/images/2026/11/black-friday-ecommerce-checklist-feature-2.webp"],
  "alternates": null,
  "statusCode": null,
  "scrapedAt": "2026-11-15T10:02:25.764Z",
  "status": "ok",
  "error": null
}
```

A site with no sitemap at all — `node test/live.mjs https://example.com 5` — delivers one free
error row instead of failing the run:

```json
{
  "url": null,
  "sitemapUrl": null,
  "lastmod": null,
  "changefreq": null,
  "priority": null,
  "images": null,
  "imageUrls": null,
  "alternates": null,
  "statusCode": null,
  "scrapedAt": "2026-11-15T10:00:12.004Z",
  "status": "error",
  "error": "No sitemap found for https://example.com. Tried: https://example.com/sitemap.xml: HTTP 404; https://example.com/sitemap_index.xml: HTTP 404; https://example.com/sitemap-index.xml: HTTP 404; https://example.com/wp-sitemap.xml: HTTP 404"
}
```

### Pricing

| Event | Price | Charged when |
|---|---|---|
| `url-extracted` | $0.0003 per URL ($0.30 per 1,000) | a URL row was pushed to the dataset |
| `url-status-checked` | $0.0005 per URL ($0.50 per 1,000) | `checkStatus` is on and a status code came back for a delivered URL |

Nothing else is charged. Error rows are free. URLs removed by your filters, duplicates, and URLs
that never got delivered because the cap or the run's max charge stopped the run are free too — a
row is pushed first and charged second, so you never pay for a row you did not receive.

Examples: 1,000 URLs = **$0.30**; 10,000 URLs = **$3.00**; 1,000 URLs with a status check =
**$0.80**. Apify platform usage (a default 1,000-URL run takes a few seconds and peaks around
100 MB) is counted separately by Apify.

### How it works

1. **Discovery.** For each start URL the run reads `robots.txt` and takes every `Sitemap:` line
   (all of them, not just the first). If there is none, it probes `/sitemap.xml`,
   `/sitemap_index.xml`, `/sitemap-index.xml` and `/wp-sitemap.xml` in that order and stops at the
   first file that really contains sitemap records. Only when all four fail is one error row
   delivered, so a site without sitemaps does not fill your dataset with noise.
2. **Traversal.** A sitemap index (`<sitemapindex>`) queues its children, up to 5 levels deep and
   up to `maxSitemaps` files. Each sitemap URL is fetched once per run, so a sitemap linked from
   two places is not read twice.
3. **Parsing.** The response body is streamed through a tolerant XML tokenizer: `<urlset>`,
   `<sitemapindex>`, RSS `<item>` and Atom `<entry>` documents are recognised by their root
   element, with the image and `xhtml:link` extensions. Real sitemaps often contain raw `&` and
   stray markup, so a malformed document is parsed rather than thrown away. Bytes are decoded as
   UTF-8 across chunk boundaries, so an accented or non-Latin URL that is split between two network
   chunks still comes out whole. A body that is not XML
   is read as a plain-text sitemap (one URL per line). An HTML page — a soft 404 — is reported as
   "not a sitemap", never as zero URLs.
4. **Delivery.** URLs are normalised (whitespace stripped, relative `<loc>` resolved, fragments
   removed), deduplicated, filtered, then pushed to the dataset in batches of up to 500 rows as
   the file streams in — each batch is pushed first and charged second. A 50,000-URL sitemap
   therefore starts producing rows immediately instead of after the download.
5. **Stop conditions.** `maxUrls` (counted in delivered rows), `maxSitemaps`, an
   `ACTOR_MAX_PAID_DATASET_ITEMS` limit, or Apify's max-charge limit for the run. Whichever comes
   first, the run stops cleanly, the open response is destroyed and the dataset keeps everything
   already delivered.

`RUN_SUMMARY` in the run's key-value store records the counts for the run: sitemaps read, URLs
seen, delivered, duplicates, each filter reason, and error rows.

### Reliability, blocks and limits

- **No proxy needed by default.** Sitemaps are public files and the live runs below were done
  directly, without a proxy: `blog.apify.com` (999 URLs from 5 sitemaps, 0.2 s, 98 MB peak),
  `www.apify.com` (a 22-child sitemap index), `www.nytimes.com` (a gzipped sitemap index with
  thousands of children: 1,000 URLs in 1.5 s, 103 MB) and `example.com` (no sitemap: one free
  error row in 0.2 s). None of them blocked the run; every response was HTTP 200.
- **If a site does block datacenter IPs**, set `proxyConfiguration` (an Apify proxy or your own).
  The actor never retries forever: 408/429/5xx responses are retried twice with backoff, a 404 is
  not retried, a stalled connection is dropped after 30 seconds without data, and any sitemap that
  still fails becomes a free error row.
- **Size guardrails:** 100 MB is the decompressed ceiling for a single sitemap (Google's own limit
  is 50 MB), 5 levels of nested indexes, `maxSitemaps` 200 by default, and `maxUrls` 1,000 by
  default so a first run costs cents.
- **Redirects:** sitemap and `robots.txt` requests follow up to 5 redirects; `statusCode` checks
  do not.

### What it does not do

- It does not crawl your pages: it reads sitemaps. A site with no sitemap has no URLs to return.
- Sitemaps that only exist behind JavaScript rendering are not discovered (sitemap files are XML,
  not rendered pages).
- `checkStatus` is a HEAD request (GET only when the server refuses HEAD), so it reports the HTTP
  status, not the page content, its title or whether the page is soft-404 content.
- hreflang alternates are reported on the row of the URL that declares them; they are not expanded
  into extra rows, and they are not charged as URLs.
- Sitemaps are decoded as UTF-8, which the sitemap protocol requires; a sitemap served in a legacy
  8-bit encoding (ISO-8859-1) would deliver mangled accented characters in its URLs.
- Sitemap entries are reported as published. If a sitemap lists a URL that has since been deleted,
  the row still appears — use `checkStatus` to see the 404.

### Development

```bash
npm install
npm test          # offline: node --test, fixtures only, no network
node test/live.mjs https://blog.apify.com 25        # live smoke test, not part of npm test
node test/live.mjs https://blog.apify.com 10 --check-status
```

The offline suite covers sitemap dialects (`test/fixtures/`), chunk boundaries, gzip, plain text,
50,000-URL documents, the discovery and traversal rules, the billing order (push before charge,
failed pushes never charged, error rows free, cap counted in delivered rows), input validation and
the manifest/ dataset-schema limits Apify enforces at deploy time. `test/live.mjs` runs the real
fetch-and-parse path against one live site and prints the row count, one full row and peak memory;
it exits non-zero when nothing is delivered.

### Run monitoring and privacy

If a run fails or times out, this Actor tells its developer the run ID and status, so we can fix problems fast. It never sends your input, your results or your account data.

# Actor input Schema

## `startUrls` (type: `array`):

Website roots (https://example.com) or direct sitemap URLs (https://example.com/sitemap.xml). For a site root the actor reads robots.txt for Sitemap: lines and then tries /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml and /wp-sitemap.xml.

## `includePatterns` (type: `array`):

Keep only URLs matching at least one of these regular expressions. Example: /blog/ to keep blog URLs only. Empty = keep everything.

## `excludePatterns` (type: `array`):

Drop URLs matching any of these regular expressions. Example: .(jpg|png|pdf)$ or /tag/. Excludes are applied after includes.

## `lastmodAfter` (type: `string`):

ISO date (2026-01-31 or 2026-01-31T12:00:00Z). Keeps only sitemap entries whose <lastmod> is on or after this date. Entries without a lastmod are dropped when this is set.

## `maxUrls` (type: `integer`):

Hard cap on delivered URL rows (1-1,000,000). The run stops as soon as it is reached, so a default run stays cheap. Charged at $0.0003 per delivered URL.

## `checkStatus` (type: `boolean`):

Send a HEAD request (GET fallback) to each delivered URL and fill in statusCode, so you can spot 404s and redirects. Billed separately at $0.50 per 1,000 checked URLs.

## `maxSitemaps` (type: `integer`):

How many sitemap files (index children included) one run may read (1-10,000). Default 200 is plenty for almost every site.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for a sitemap or robots.txt response (5-300 seconds).

## `proxyConfiguration` (type: `object`):

Optional. Sitemaps are public, so no proxy is used by default. Add an Apify proxy for sites that block datacenter IPs.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.smashingmagazine.com"
  ],
  "includePatterns": [],
  "excludePatterns": [],
  "lastmodAfter": "",
  "maxUrls": 1000,
  "checkStatus": false,
  "maxSitemaps": 200,
  "requestTimeoutSecs": 60
}
```

# Actor output Schema

## `urls` (type: `string`):

One dataset item per URL: url, sitemapUrl, lastmod, changefreq, priority, image count, hreflang alternates and an optional status code.

## `runSummary` (type: `string`):

Counts for the run: sitemaps read, URLs delivered, duplicates, filtered URLs and error rows.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.smashingmagazine.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("subimpact/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://www.smashingmagazine.com"] }

# Run the Actor and wait for it to finish
run = client.actor("subimpact/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.smashingmagazine.com"
  ]
}' |
apify call subimpact/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,subimpact/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/w06OASML14iz01vix/builds/UtYhE9LM94uu4GAPR/openapi.json
