# Website Sitemap Extractor - Find Sitemaps & Extract URLs (`clearfetch/website-sitemap-extractor`) Actor

Give it a domain and it finds the sitemaps itself, from robots.txt, the homepage or the usual paths, follows sitemap index files, unpacks gzipped ones, and returns every URL with lastmod, priority, hreflang alternates and images. No proxy, no login.

- **URL**: https://apify.com/clearfetch/website-sitemap-extractor.md
- **Developed by:** [Nada Hanad](https://apify.com/clearfetch) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 website processeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website Sitemap Extractor - Find Sitemaps & Extract URLs

Give it a domain. It finds the sitemaps itself, follows index files, unpacks gzipped ones, and returns every URL
with its last-modified date, hreflang alternates and images. **$0.002 per website plus $0.02 per 1,000 URLs.**
Websites with no sitemap are free.

### Why this one

Most sitemap tools ask you for the sitemap URL. If you already knew where it was, you would not need the tool.
This one starts from a bare domain and looks in the three places a sitemap actually lives:

1. **robots.txt**, where the standard says it should be declared.
2. **A link on the homepage**, which some content systems use instead.
3. **The usual paths**, from `/sitemap.xml` to `/wp-sitemap.xml` and `/sitemap_index.xml`.

The output tells you which of the three worked, so an SEO audit can flag a site that has a sitemap but never
declares it.

It also handles the parts that break naive extractors:

- **Sitemap index files**, followed recursively, because large sites split their URLs across hundreds of files.
- **Gzipped sitemaps**, detected from the file's own bytes rather than the content type, which servers get wrong.
- **Plain-text sitemaps**, which the protocol allows and almost nothing supports.
- **An HTML error page served in place of a sitemap**, reported as a failed file instead of parsed into nonsense.

### What data you get

Per URL: `loc`, `lastmod`, `changefreq`, `priority`, **hreflang alternates**, **image URLs**, and the
sitemap file it came from, which matters on sites with dozens of them.

Per website: how the sitemap was discovered, the full sitemap tree with the URL count of each file, and a summary
including the newest `lastmod` anywhere on the site, which tells you at a glance when it last changed.

### How to use

1. Paste domains into **Websites**, one per line. A direct sitemap link works too and is used as-is.
2. Optionally filter with **URL filter** (a regular expression) or **Changed since** (a date), to pull only what
   you need.
3. Run it. Each website is one row, exportable as JSON, CSV or Excel.

### Input

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `urls` | array | — | Domains, or direct sitemap links. Also accepts `url` and `startUrls`. |
| `maxUrlsPerSite` | integer | `0` | Stop after this many URLs per website. 0 means no limit. |
| `maxSitemaps` | integer | `50` | How many sitemap files to read per website. |
| `urlPattern` | string | — | Case-insensitive regular expression; only matching URLs are returned. |
| `changedSince` | string | — | A date such as `2026-01-31`; only URLs with a `lastmod` on or after it. |
| `includeUrlList` | boolean | `true` | Turn off for counts and the sitemap tree only, when auditing many sites. |
| `maxConcurrency` | integer | `5` | Websites processed in parallel. |
| `timeoutSecs` | integer | `30` | Per-request timeout. |
| `proxyConfiguration` | object | off | Optional. Not needed for most sites. |

### Output example

One website, trimmed to two URLs and three sitemap files:

```json
{
  "url": "https://vercel.com/",
  "ok": true,
  "discovery": "robots.txt",
  "sitemapCount": 1,
  "sitemaps": [
    {
      "url": "https://vercel.com/sitemap.xml",
      "ok": true,
      "type": "urlset",
      "urls": 7092,
      "children": 0,
      "depth": 0
    }
  ],
  "urlCount": 500,
  "urls": [
    {
      "loc": "https://vercel.com/docs/flags",
      "lastmod": "2026-08-29T03:11:19.144Z",
      "changefreq": null,
      "priority": null,
      "alternates": [],
      "images": [],
      "source": "https://vercel.com/sitemap.xml"
    },
    {
      "loc": "https://vercel.com/kb/ai-sdk",
      "lastmod": "2026-08-26T01:31:12.753Z",
      "changefreq": null,
      "priority": null,
      "alternates": [],
      "images": [],
      "source": "https://vercel.com/sitemap.xml"
    }
  ],
  "stats": {
    "withLastmod": 500,
    "withAlternates": 0,
    "withImages": 0,
    "newestLastmod": "2026-09-06T10:10:02.700Z"
  },
  "notes": [
    "Stopped at the \"maxUrlsPerSite\" limit of 500; the site has more."
  ],
  "elapsedMs": 330,
  "scrapedAt": "2026-09-06T10:30:50.785Z"
}
```

A website with no sitemap is reported and costs nothing:

```json
{
  "url": "https://this-domain-does-not-exist-12345.com/",
  "ok": false,
  "error": "no sitemap found in robots.txt, on the homepage, or at any of the usual paths",
  "elapsedMs": 11729,
  "scrapedAt": "2026-09-06T10:31:02.184Z"
}
```

### Pricing

- **$0.002 per website**, covering discovery and every sitemap file read.
- **$0.02 per 1,000 URLs** returned. A 500-page site costs about a cent. A 50,000-page site costs about a dollar.
- Websites with no sitemap are free.

### Use cases

- **Crawl planning**: get the URL list before you crawl, so you scrape only what exists.
- **SEO audits**: find sites whose sitemap is undeclared, whose index points at missing files, or whose
  `lastmod` dates are stale.
- **Change monitoring**: run with **Changed since** on a schedule and get only the pages updated since your last
  run.
- **Content migration**: enumerate every URL on the old site before switching over.
- **Feeding other tools**: pipe the URL list into a scraper, a screenshot tool, or a technology detector.
- **AI agents**: a tool that answers "what pages does this site have" without crawling it.

### Integrations

```bash
curl -X POST "https://api.apify.com/v2/acts/clearfetch~website-sitemap-extractor/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["apify.com"], "urlPattern": "/blog/"}'
```

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("clearfetch/website-sitemap-extractor").call(
    run_input={"urls": ["apify.com"], "changedSince": "2026-01-01"}
)

for site in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(site["url"], site["urlCount"], "URLs via", site["discovery"])
    for u in site["urls"][:5]:
        print("  ", u["lastmod"], u["loc"])
```

Works with the Apify integrations for n8n, Make, Zapier, Google Sheets, Slack and webhooks, with scheduled runs,
and with AI agents through the Apify MCP server.

### FAQ

**What if a site has no sitemap?** The row comes back with `ok: false` and an explanation, and you are not
charged for it.

**Does it check whether the URLs work?** No. This lists what a site says it publishes. Checking each URL is a
different job with a different cost profile.

**How deep does it follow index files?** As deep as they go, breadth-first, up to the **maxSitemaps** limit. If
that limit stops it early, the output says so and tells you how many files were left.

**Are gzipped sitemaps supported?** Yes, detected from the file's own magic bytes, because servers routinely
send them with the wrong content type.

**Do I need a proxy?** No. A proxy input exists for sites that block datacenter traffic.

**Is this legal?** A sitemap is a file a website publishes specifically to be read by machines. This reads it.

### Changelog

- **1.0.0** (2026-09) — first release: discovery from robots.txt, homepage links and common paths; recursive
  index following; gzip and plain-text sitemaps; hreflang alternates and images; filtering by pattern and date.

# Actor input Schema

## `urls` (type: `array`):

Domains, one per line. You do not need to know where the sitemap is: it is found from robots.txt, the homepage, or the usual paths. A direct link to a sitemap.xml works too and is used as-is.

## `maxUrlsPerSite` (type: `integer`):

Stop after this many URLs from one website. 0 means no limit.

## `maxSitemaps` (type: `integer`):

How many sitemap files to read per website. Large sites split their sitemap into hundreds of files behind an index.

## `urlPattern` (type: `string`):

Optional case-insensitive regular expression. Only URLs matching it are returned, for example "/blog/" or "\\.html$".

## `changedSince` (type: `string`):

Optional date such as 2026-01-31. Only URLs whose lastmod is on or after it are returned, which is how you pull just what changed.

## `includeUrlList` (type: `boolean`):

Turn off to get only the counts and the sitemap tree, which is useful when auditing many sites at once.

## `maxConcurrency` (type: `integer`):

Websites processed in parallel.

## `timeoutSecs` (type: `integer`):

Per-request timeout.

## `proxyConfiguration` (type: `object`):

Optional. Not needed for most sites.

## Actor input object example

```json
{
  "urls": [
    "apify.com",
    "vercel.com"
  ],
  "maxUrlsPerSite": 0,
  "maxSitemaps": 50,
  "includeUrlList": true,
  "maxConcurrency": 5,
  "timeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

One row per website: how the sitemap was discovered, every sitemap file read and whether it parsed, and the full URL list with lastmod, changefreq, priority, hreflang alternates, images and which sitemap file each URL came from. Websites with no sitemap appear with ok=false and are not charged.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "apify.com",
        "vercel.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("clearfetch/website-sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "apify.com",
        "vercel.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("clearfetch/website-sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "apify.com",
    "vercel.com"
  ]
}' |
apify call clearfetch/website-sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,clearfetch/website-sitemap-extractor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/e9r67OBgj8M4UVATz/builds/VEmF1n1xYL2gxeOlM/openapi.json