# Sitemap URL Extractor (`sampon/sitemap-url-extractor`) Actor

List every URL a website publishes in its XML sitemaps, found automatically through robots.txt. Filter by text or lastmod date, or get only the URLs added since your last run.

- **URL**: https://apify.com/sampon/sitemap-url-extractor.md
- **Developed by:** [Sampon](https://apify.com/sampon) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor

Give it a website and it returns every URL the site lists in its XML sitemaps, one row per URL. You don't need to know where the sitemap lives: the Actor reads the `Sitemap:` lines in robots.txt, falls back to the usual places such as `/sitemap.xml` and `/sitemap_index.xml`, and then follows sitemap indexes and `.gz` files down to the last URL.

Typical uses: an inventory of a site before an SEO audit or a migration, a list of pages to feed into another scraper, or a daily check for pages a competitor has just published.

### What you put in

The only required field is the website (or a sitemap address, if you already have one):

```json
{
  "startUrls": [{ "url": "https://crawlee.dev/" }],
  "maxItems": 5000
}
```

You can list several websites in one run.

| Setting | What it does |
|---|---|
| `maxItems` | Stops after this many URLs in total. The default is 50 so a first run stays quick; raise it to get the whole sitemap. |
| `includePatterns` / `excludePatterns` | Keep only URLs that contain one of your texts (for example `/blog/`), or drop URLs that contain one (for example `/tag/`). Case does not matter. |
| `patternsAreRegex` | Read the two lists above as regular expressions instead of plain text. |
| `lastmodSince` | Keep only URLs whose `lastmod` is on or after a date (`2026-01-31`) or within a period (`7 days`, `2 weeks`, `3 months`). |
| `keepUrlsWithoutLastmod` | With a date filter, URLs that have no `lastmod` are dropped unless you turn this on. |
| `onlyNewUrls` | Returns only URLs that were not in the sitemap on your previous run. See below. |
| `rememberUrls` | Keeps every row but marks each one with `isNew: true` or `false`. |
| `maxSitemaps` | Safety limit on the number of sitemap files read in one run (default 500). |

### What you get

One row per unique URL:

```json
{
  "url": "https://crawlee.dev/blog",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "images": 0,
  "videos": 0,
  "news": 0,
  "hreflangCount": 0,
  "hreflangs": null,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "startUrl": "https://crawlee.dev/",
  "isNew": null
}
```

- `lastmod`, `changefreq` and `priority` are copied as the site publishes them. Many sites leave some of them out, and those come back as `null`.
- `images`, `videos` and `news` count the image, video and news sitemap entries attached to the URL.
- `hreflangs` lists the language codes of the alternate versions declared for the URL (`x-default` is left out), and `hreflangCount` is their number.
- `sitemapUrl` tells you which sitemap file the URL came from, which helps when a site splits its sitemap by section.
- `isNew` is filled only when change tracking is on.

A URL that appears in more than one sitemap is returned once.

#### The run summary

Next to the table, the run saves one `SUMMARY` record in its key-value store: how many sitemap files were read, failed or skipped, how many URLs were found and saved, and for each sitemap file where it was found (robots.txt, a sitemap index, a usual location or your input) and what happened to it. When a file can't be used, the reason is written there and the run carries on with the rest. Typical reasons are `HTTP 404`, "the server returned an HTML page instead of a sitemap", "broken XML" (URLs read before the break are kept) and "disallowed by robots.txt".

### Getting only new URLs

Turn on `onlyNewUrls` and run the same input on a schedule. The first run returns everything and remembers it; each later run returns only the URLs that have appeared since. The memory is a key-value store in your own account, named by `stateStoreName`, with one record per website. It holds short fingerprints of the URLs, not the URLs themselves. Use a different name to keep separate histories.

In this mode every sitemap file is read even after `maxItems` is reached, so that the memory stays complete. On large sites that makes the run longer.

### Price

Free for now. You only pay Apify's usual platform usage. In our test, a run that read one sitemap of about 3,900 URLs and saved 50 took about 3 seconds, which costs a small fraction of a cent. A per-URL price may be added later. If that happens, it will be shown on this page before it applies to you.

### How it treats the sites it reads

- It only reads robots.txt and the sitemap files a site publishes for search engines. It never opens the pages listed in them.
- It follows robots.txt. A sitemap in a location that robots.txt disallows is skipped, and so is everything on a site whose robots.txt answers with a server error or "forbidden". A crawl delay in robots.txt is respected.
- By default it makes at most two requests at a time and waits at least half a second between requests to the same site. You can slow it down further, but not beyond four parallel requests.

### Limits

- It reads XML sitemaps, sitemap indexes, gzip-compressed sitemaps and plain-text sitemaps (one URL per line). RSS and Atom feeds are not read.
- A site with no sitemap in robots.txt and none at the usual locations returns no rows. The summary says so.
- Files larger than 50 MB after decompression are skipped, as the sitemap standard sets that limit.
- The result is what the site says in its sitemap. Pages missing from the sitemap are not found, and a `lastmod` date is only as accurate as the site keeps it.
- A sitemap index nested more than five levels deep is not followed further.

This Actor is an independent tool. It is not made by, or affiliated with, any of the websites it reads or the companies behind sitemap standards.

# Actor input Schema

## `startUrls` (type: `array`):

A website home page (for example https://crawlee.dev/) or the address of a sitemap or sitemap index. Several can be listed in one run.

## `maxItems` (type: `integer`):

Stop after saving this many URLs in total. Raise it to get the whole sitemap.

## `includePatterns` (type: `array`):

Keep only URLs that contain at least one of these texts, such as /blog/ or /products/. Case does not matter. Leave empty to keep everything.

## `excludePatterns` (type: `array`):

Drop URLs that contain any of these texts, such as ?page= or /tag/.

## `patternsAreRegex` (type: `boolean`):

Turn on to write the two lists above as regular expressions (Python syntax, case-insensitive), for example /p/\d+$.

## `lastmodSince` (type: `string`):

Keep only URLs whose lastmod date is on or after this point. Write a date such as 2026-01-31, or a period such as 7 days, 2 weeks or 3 months. Leave empty to ignore dates.

## `keepUrlsWithoutLastmod` (type: `boolean`):

When a date filter is set, URLs that have no lastmod in the sitemap are dropped. Turn this on to keep them.

## `onlyNewUrls` (type: `boolean`):

Remembers the URLs seen for each website and returns only URLs that were not there last time. The first run returns everything and sets the baseline. Every sitemap is read in this mode, even after the maximum is reached, so the memory stays complete.

## `rememberUrls` (type: `boolean`):

Keeps all rows but fills the isNew column (true or false) by comparing with earlier runs. Not needed when the option above is on.

## `stateStoreName` (type: `string`):

Name of the key-value store in your account that holds the remembered URLs. Use a different name to keep separate histories, for example one per project.

## `maxSitemaps` (type: `integer`):

Safety limit on how many sitemap files are read per run, including index files.

## `maxConcurrency` (type: `integer`):

How many sitemap files to download at the same time. Capped at 4 to stay gentle on the website.

## `requestDelayMs` (type: `integer`):

Minimum time between two requests to the same website. A longer crawl delay in the site's robots.txt always wins.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/"
    }
  ],
  "maxItems": 50,
  "patternsAreRegex": false,
  "keepUrlsWithoutLastmod": false,
  "onlyNewUrls": false,
  "rememberUrls": false,
  "stateStoreName": "sitemap-url-extractor-state",
  "maxSitemaps": 500,
  "maxConcurrency": 2,
  "requestDelayMs": 500
}
```

# Actor output Schema

## `results` (type: `string`):

One row per URL found in the sitemaps, with lastmod, change frequency, priority, media counts and the sitemap it came from.

## `summary` (type: `string`):

How many sitemaps were read, failed or skipped (with the reason for each) and how many URLs were found.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("sampon/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://crawlee.dev/" }] }

# Run the Actor and wait for it to finish
run = client.actor("sampon/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev/"
    }
  ]
}' |
apify call sampon/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sampon/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Zf3XNKd8aR7tObI9R/builds/QJ2M0LRmnQPn6Rhg1/openapi.json