# Sitemap URL Extractor (`jakob_jerich/sitemap-url-extractor`) Actor

Extract every URL from any website's XML sitemaps: auto-discovers sitemaps via robots.txt, follows sitemap indexes and .gz files, returns lastmod, hreflang, images, robots.txt status and optional HTTP status codes. $0.30 per 1,000 URLs.

- **URL**: https://apify.com/jakob_jerich/sitemap-url-extractor.md
- **Developed by:** [Jakob Jerič](https://apify.com/jakob_jerich) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.30 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor

Get every URL from a website's XML sitemaps. Enter a website (or a sitemap URL). The Actor finds the sitemaps through `robots.txt` and the usual locations, follows sitemap indexes and gzipped `.xml.gz` files, and returns one clean row per page. Each row has `lastmod`, `changefreq`, `priority`, hreflang alternates, image and video counts, and whether `robots.txt` allows the URL. HTTP status codes are optional.

No browser and no proxies, so it is fast and cheap: **$0.30 per 1,000 URLs**, status check included.

### What it does

- **Finds sitemaps automatically.** It reads `Sitemap:` lines in `robots.txt`, then tries `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml` and `/sitemap.txt`.
- **Follows nested sitemap indexes** of any depth, and handles **gzipped sitemaps** (`.xml.gz`).
- **Reads every sitemap format**: standard XML, Google image, video and news extensions, `xhtml:link` hreflang, RSS and Atom feeds, and plain-text URL lists.
- **Finds new or updated pages.** Filter by `lastmod` with a date ("2026-01-31") or a period ("7 days"). Child sitemaps older than the cut-off are skipped without being downloaded.
- **Include and exclude filters** with regular expressions, e.g. only `/blog/` or no `/tag/`.
- **robots.txt check for each URL.** It flags sitemap URLs that `robots.txt` blocks, a common SEO mistake.
- **Optional HTTP status check.** A lightweight HEAD request records the **original** status code. Redirects are not followed, so 301/302/404 URLs in your sitemap stand out.
- **Handles errors well.** Retries with backoff on timeouts, 429 and 5xx responses. Unreachable sites or missing sitemaps don't fail the run. Each problem is explained in the `SUMMARY` record.

### Use cases

- **SEO audits**: find sitemap URLs that redirect, return 404, or are blocked by robots.txt.
- **Crawl seeding**: give your scraper or RAG/LLM pipeline a complete, de-duplicated URL list instead of crawling link by link.
- **Content monitoring**: schedule a daily run with `lastmodAfter: "1 day"` to see a competitor's new pages, products or articles.
- **Site migrations**: export all old URLs to build redirect maps, then check them afterwards.
- **Inventory and research**: count pages by section, language (hreflang) or update frequency.

### Input

| Field | Description | Default |
|---|---|---|
| `startUrls` | Websites (`https://example.com`) or direct sitemap URLs (`.xml`, `.xml.gz`, index, feed, `.txt`) | required |
| `maxUrlsPerSite` | Max URLs saved per start URL (0 = no limit). This also caps your cost. | 10000 |
| `includePatterns` | Keep only URLs that match one of these regexes | – |
| `excludePatterns` | Drop URLs that match any of these regexes | – |
| `lastmodAfter` | Keep URLs modified on or after a date or period (`2026-01-31`, `7 days`, `3 months`) | – |
| `dropUrlsWithoutLastmod` | Skip URLs that have no lastmod | false |
| `sameDomainOnly` | Keep only URLs on the start URL's domain (www. is ignored) | true |
| `checkStatus` | Send a HEAD request to each URL and record its status code and redirect target | false |
| `statusConcurrency` | Parallel status requests | 5 |
| `maxSitemapFiles` | Safety limit on sitemap files read per site | 500 |
| `discoverSitemaps` | Try the usual sitemap locations if robots.txt declares none | true |

#### Example input

```json
{
  "startUrls": ["https://apify.com", "https://www.gov.uk/sitemap.xml"],
  "maxUrlsPerSite": 5000,
  "excludePatterns": ["/tag/"],
  "lastmodAfter": "30 days",
  "checkStatus": true
}
```

### Output

Each saved URL is one dataset row. This row is from a real run on `https://www.nasa.gov` with `checkStatus` on. Every row has the same fields, with `null` when the sitemap doesn't provide a value:

```json
{
  "url": "https://www.nasa.gov/directorates/armd/aetc/about-aetc-ground-facilities/",
  "startUrl": "https://www.nasa.gov",
  "sitemapUrl": "https://www.nasa.gov/wp-sitemap-posts-post-1.xml",
  "lastmod": "2026-09-18T12:40:35.000Z",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "videoCount": 0,
  "alternates": [],
  "newsTitle": null,
  "newsPublicationDate": null,
  "robotsAllowed": true,
  "statusCode": 301,
  "redirectLocation": "https://www.nasa.gov/directorates/armd/aetc",
  "statusError": null
}
```

`statusCode`, `redirectLocation` and `statusError` are filled only when `checkStatus` is on. A plain date in the sitemap (`2026-10-20`) is kept as a plain date.

The key-value store record **`SUMMARY`** lists, for each start URL: whether robots.txt was found, the sitemaps discovered, how many sitemap files were parsed, URLs found and saved, and a plain-English reason for any problem (e.g. "HTTP 403 (the website refuses automated access to this file)").

You can download the dataset as JSON, CSV, Excel or XML, or get it through the Apify API, webhooks, Zapier, Make or MCP.

### Pricing

Pay per event: **$0.30 per 1,000 URLs saved** ($0.0003 per URL). The HTTP status check costs nothing extra. Sites without a sitemap and URLs removed by your filters cost nothing. Set `maxUrlsPerSite` or a maximum cost per run to stay within budget; the Actor stops cleanly when the limit is reached.

Examples: a 50-page brochure site costs $0.015. A 2,000-page blog costs $0.60. A 100,000-URL e-commerce sitemap costs $30, or less with filters.

### FAQ

**Does it crawl the website?** No. It reads only the sitemaps the site publishes for robots, plus HEAD requests if you turn on status checking. That makes it fast and light on the target site.

**What if a site has no sitemap?** The run still succeeds. `SUMMARY` says that no sitemap was found and you are not charged. For sites without sitemaps, use a link crawler instead.

**Why does a site return HTTP 403?** Some sites block automated clients. This Actor identifies itself honestly and does not bypass bot protection or use residential proxies, so such sites are reported, not scraped.

**My sitemap is on another host (CDN) than my pages. Why are rows missing?** `sameDomainOnly` compares URLs with the start URL's domain. Start from the website address instead, or turn `sameDomainOnly` off.

**Does it respect robots.txt?** Yes. Guessed sitemap locations and status checks are skipped for paths that robots.txt disallows. Every row includes `robotsAllowed`.

**How many URLs can it handle?** Each sitemap file can be up to 100 MB uncompressed (the protocol limit is 50 MB). URLs are de-duplicated in memory, so for runs with several hundred thousand URLs or more, raise the memory to 1 GB.

**Can I get only new pages since my last run?** Yes. Schedule the Actor with `lastmodAfter` set to the period between runs, e.g. `"1 day"`. Accuracy depends on how carefully the site maintains `lastmod`.

### Limits

- Only URLs listed in sitemaps are returned. Pages missing from the sitemap are not discovered.
- `lastmod`, `changefreq` and `priority` are taken as the site declares them.
- Status checks use HEAD (falling back to GET when HEAD isn't supported). A few servers answer HEAD differently from GET.
- No login, cookies or bot-protection bypass.

Questions or a feature request? Open an issue on the Actor's Issues tab.

# Actor input Schema

## `startUrls` (type: `array`):

Website addresses (e.g. https://www.example.com) or direct sitemap URLs (e.g. https://www.example.com/sitemap.xml, .xml.gz, sitemap index, RSS/Atom feed or plain-text URL list). For a website, sitemaps are discovered from robots.txt and the usual locations.

## `maxUrlsPerSite` (type: `integer`):

Stop after saving this many URLs for each start URL. Use 0 for no limit. You pay per saved URL, so this also caps your cost.

## `includePatterns` (type: `array`):

Optional. Keep only URLs that match at least one of these regular expressions (case-insensitive), e.g. "/blog/" or "/products/.+". Leave empty to keep all.

## `excludePatterns` (type: `array`):

Optional. Drop URLs that match any of these regular expressions (case-insensitive), e.g. "/tag/" or "\\?page=".

## `lastmodAfter` (type: `string`):

Optional. Keep only URLs whose sitemap <lastmod> is on or after this date. Accepts an absolute date (2026-01-31) or a relative period ("7 days", "2 weeks", "3 months"). URLs without a lastmod are kept unless you enable the option below. Great for finding new or updated pages.

## `dropUrlsWithoutLastmod` (type: `boolean`):

If enabled, URLs that have no <lastmod> date in the sitemap are skipped.

## `sameDomainOnly` (type: `boolean`):

Keep only URLs on the same domain as the start URL (www. is ignored). Disable to also keep URLs on other domains or subdomains that appear in the sitemap.

## `checkStatus` (type: `boolean`):

If enabled, sends a lightweight HEAD request to every saved URL and records the HTTP status code and redirect target (redirects are not followed, so you see 301/302 issues in your sitemap). URLs disallowed by robots.txt are not requested. Slower: roughly 5–20 URLs per second.

## `statusConcurrency` (type: `integer`):

How many status-check requests run in parallel. Keep it low to be polite to the website.

## `maxSitemapFiles` (type: `integer`):

Safety limit on how many sitemap files (including nested sitemap indexes) are read per start URL.

## `discoverSitemaps` (type: `boolean`):

If robots.txt declares no sitemap, try the usual locations: /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml and /sitemap.txt.

## Actor input object example

```json
{
  "startUrls": [
    "https://apify.com",
    "https://www.gov.uk/sitemap.xml"
  ],
  "maxUrlsPerSite": 1000,
  "includePatterns": [],
  "excludePatterns": [],
  "dropUrlsWithoutLastmod": false,
  "sameDomainOnly": true,
  "checkStatus": false,
  "statusConcurrency": 5,
  "maxSitemapFiles": 500,
  "discoverSitemaps": true
}
```

# Actor output Schema

## `urls` (type: `string`):

All saved URLs with lastmod, changefreq, priority, hreflang, image/video counts, robots.txt status and optional HTTP status.

## `summary` (type: `string`):

SUMMARY record: sitemaps discovered, files parsed, URL counts and the reason for any problem, per start URL.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://apify.com",
        "https://www.gov.uk/sitemap.xml"
    ],
    "maxUrlsPerSite": 1000
};

// Run the Actor and wait for it to finish
const run = await client.actor("jakob_jerich/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        "https://apify.com",
        "https://www.gov.uk/sitemap.xml",
    ],
    "maxUrlsPerSite": 1000,
}

# Run the Actor and wait for it to finish
run = client.actor("jakob_jerich/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://apify.com",
    "https://www.gov.uk/sitemap.xml"
  ],
  "maxUrlsPerSite": 1000
}' |
apify call jakob_jerich/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,jakob_jerich/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/UhMCqCSEcLB84vdZl/builds/BeyuMPByAbBkDB0aC/openapi.json