# Sitemap Scraper: Every URL on Any Site (`glitchbound/sitemap-scraper`) Actor

Get every URL a website publishes, from a bare domain. Finds the sitemap through robots.txt or the usual paths, follows nested sitemap index files to any depth, unpacks .xml.gz, and returns each URL with lastmod, changefreq and priority. Filter by path regex or by what changed since a date.

- **URL**: https://apify.com/glitchbound/sitemap-scraper.md
- **Developed by:** [Daniel Meshulam](https://apify.com/glitchbound) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.75 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Scraper: Every URL on Any Site

Give it a domain. Get back every URL that site publishes, with the date each one
last changed.

```json
{ "startUrls": ["apify.com"], "maxUrlsPerSite": 5000 }
```

You do not have to find the sitemap first. That is the part that actually takes
time, and it is the part this Actor does for you.

### Where the other sitemap tools stop

| | Typical Store Actor | This one |
|---|---|---|
| Input | you paste the exact sitemap URL | a bare domain is enough |
| robots.txt | ignored | read first, it is the authoritative answer |
| Sitemap index files | first level only | followed to any depth |
| `.xml.gz` | fails or returns binary | unpacked |
| No sitemap at all | empty run, no explanation | one `error` row saying so, **unbilled** |

That third row is the one that quietly costs people money elsewhere. A sitemap
index is a sitemap *of sitemaps*; a site with 400,000 URLs typically publishes an
index pointing at 40 files of 10,000 each. Parsing only the top level returns
about 40 rows and looks like it worked.

### What each row gives you

| Field | |
|---|---|
| `url` | the page URL |
| `path` | just the path, for filtering and grouping |
| `lastmod` | normalised to UTC ISO, sites write it five different ways |
| `changefreq`, `priority` | as declared, when present |
| `discoveredVia` | `robots.txt` or which path was guessed |
| `sitemap` | which sitemap file this URL came from |

### Pull only what changed

```json
{ "startUrls": ["example.com"], "changedSince": "2026-07-01" }
```

Re-crawling a whole site to find the 30 pages that moved is the usual waste.
`changedSince` filters on `lastmod`, so you get the 30.

### Filter by section

```json
{
  "startUrls": ["example.com"],
  "includePattern": "^/blog/",
  "excludePattern": "/(tag|author|page)/"
}
```

Both are regular expressions tested against the path.

### Use cases

- **SEO audits**: full URL inventory, find orphan and stale pages
- **Migration QA**: diff the sitemap before and after a replatform
- **Feeding a crawler**: a clean seed list beats spidering blindly
- **Content monitoring**: schedule it and watch `lastmod` move
- **AI/RAG pipelines**: enumerate a documentation site before ingesting it

### Pricing

Charged per URL returned. A site with no sitemap, a sitemap that 404s, or one
that fails to parse produces an `error` row and costs **$0.00**: you are never
billed for a lookup that gave you nothing.

### Do you need an API key? No. Is there a free tier? Yes.

The two things people search for in this category, measured, are **free** and
**API key**: `sitemap scraper`, `sitemap scraper python`.

So, plainly:

- **No API key.** Nothing to register for, nothing to rotate, no key to leak in
  a repo. The source is a public API published deliberately by its owner.
- **No proxy setup.** The source does not bot-wall datacenter addresses, so the
  default works.
- **Pay per result, not per month.** There is no subscription and no minimum.
  Rows that error or that your filters drop are **not charged**.
- **Free to try.** Run it with the prefilled input and see real rows before
  deciding anything.

### Notes

- URLs come from what the site *declares*. A sitemap can be stale or incomplete;
  this reports it faithfully rather than pretending otherwise.
- `changedSince` drops URLs that carry no `lastmod`, because there is no honest
  way to decide whether they changed.
- Up to 200 sitemap files are fetched per site, which covers roughly two million
  URLs and stops a self-referencing sitemap from looping forever.

### FAQ

#### Do I have to know the sitemap URL?

No, and that is the point. Give it a bare domain and it reads `robots.txt` first,
which is the authoritative answer, then falls back to the usual paths. Every row
carries `discoveredVia` so you can see which route found it.

#### What if the site has no sitemap?

You get one `error` row saying so, and it is not charged. An empty dataset with no
explanation is the failure mode elsewhere.

#### Does it handle sitemap index files?

Yes, to any depth. This is the difference that matters: a sitemap index is a
sitemap *of sitemaps*, and a site with 400,000 URLs typically publishes an index
pointing at 40 files of 10,000. A tool that parses only the top level returns
about 40 rows and looks like it worked.

#### Are gzipped sitemaps supported?

Yes. `.xml.gz` is unpacked. Large sites commonly serve only the compressed form.

#### How do I find just the pages that changed?

`changedSince: "2026-07-01"` filters on `lastmod`, so you get the pages that moved
instead of re-crawling the whole site to find them. Note that URLs carrying no
`lastmod` at all are dropped by that filter, because there is no honest way to
decide whether they changed.

#### Can I limit it to one section of a site?

Yes. `includePattern` and `excludePattern` are regular expressions tested against
the path: `^/blog/` to keep a section, `/(tag|author|page)/` to drop pagination
and taxonomy noise.

#### Is a sitemap the same as everything on the site?

No, and the distinction matters for an audit. These are the URLs the site
*declares*. A sitemap can be stale, incomplete, or list pages that now 404. This
reports it faithfully rather than implying it is a crawl.

#### How large a site can it handle?

Up to 200 sitemap files per site, which covers roughly two million URLs and also
stops a self-referencing sitemap from looping forever.

# Actor input Schema

## `startUrls` (type: `array`):

Sites to pull the sitemap from, one per line. A bare domain like apify.com is enough, since the sitemap is discovered from robots.txt or the usual paths. You can also paste a sitemap URL directly.

## `maxUrlsPerSite` (type: `integer`):

Stop after this many URLs per site. Nested sitemap index files are followed automatically until this cap is hit. The form is prefilled with 50 so a first run costs cents; the cap when this field is left out entirely is 5000.

## `includePattern` (type: `string`):

Regular expression tested against the URL path. Example: ^/blog/ keeps only blog posts.

## `excludePattern` (type: `string`):

Regular expression tested against the URL path. Example: /(tag|author|page)/ drops paginated and taxonomy pages.

## `changedSince` (type: `string`):

Only URLs whose <lastmod> is on or after this date (YYYY-MM-DD). URLs with no lastmod are dropped. Use it to pull just what changed since your last crawl.

## `proxyConfiguration` (type: `object`):

Proxy settings.

## `maxItems` (type: `integer`):

A hard ceiling on rows for the entire run, across every target. The per-target limits above cap each one separately, so fifty targets at twenty each is still a thousand rows; this caps the total. The run stops cleanly when it is reached and nothing beyond that point is fetched or charged. Leave empty for no ceiling.

## Actor input object example

```json
{
  "startUrls": [
    "apify.com"
  ],
  "maxUrlsPerSite": 50,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

One row per URL, with the sitemap file it came from.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "apify.com"
    ],
    "maxUrlsPerSite": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("glitchbound/sitemap-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["apify.com"],
    "maxUrlsPerSite": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("glitchbound/sitemap-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "apify.com"
  ],
  "maxUrlsPerSite": 50
}' |
apify call glitchbound/sitemap-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,glitchbound/sitemap-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/cefChfXmiHPbJUXAq/builds/domFWIh21p9V8gDkf/openapi.json