# Sitemap URL Discovery — XML Sitemap & robots.txt API (`accountable_eel/sitemap-url-discovery`) Actor

Discover every URL a website's own XML sitemap declares, starting from robots.txt's Sitemap: directive and following sitemap indexes automatically. No key, no page crawling — charged only when a working sitemap is found.

- **URL**: https://apify.com/accountable\_eel/sitemap-url-discovery.md
- **Developed by:** [Adrian Voss](https://apify.com/accountable_eel) (community)
- **Categories:** Lead generation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.80 / 1,000 successful lookups

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap URL Discovery

Discover every URL a website's own sitemap declares, starting from the `Sitemap:` directive
in `robots.txt` (falling back to the conventional `/sitemap.xml` path if none is declared).
No API key, no crawling the site page-by-page — this reads the site's own XML sitemap, the
same file search engines use to index it.

### Features

- **Robots.txt-first discovery.** Reads `Sitemap:` lines from `robots.txt` rather than
  guessing at a URL, so it follows whatever sitemap the site actually declares.
- **Sitemap index support.** If the declared sitemap is an index (`<sitemapindex>`), it
  automatically follows the first child sitemap one level deep to reach real page URLs.
- **Fallback path.** Sites with no `Sitemap:` line in `robots.txt` are still checked at the
  conventional `/sitemap.xml` location.
- **Bulk URL list.** Returns up to 500 discovered URLs per site in one row, with a
  `truncated` flag when the real sitemap has more.
- **Pay only for real discoveries.** Domains with no usable sitemap cost nothing —
  see [Pricing](#pricing).

### How to use Sitemap URL Discovery — XML Sitemap & robots.txt API

1. **In the Apify Console.** Open the actor page and click **Start** — the `items` field is already pre-filled with a working example. Results land in the run's dataset as soon as each item is found.
2. **Via the API.** Call it directly with a POST request — no Console needed once you have an API token:
   ```bash
   curl "https://api.apify.com/v2/acts/accountable_eel~sitemap-url-discovery/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
     -X POST \
     -H "Content-Type: application/json" \
     -d '{"items":["bbc.co.uk"]}'
   ```
3. **On a schedule.** Save this actor as an Apify **Task** with the input you want, then add a **Schedule** (hourly, daily, weekly) so it runs on its own — no server of your own required.

### Input

```json
{
  "items": ["nytimes.com", "bbc.co.uk"],
  "maxConcurrency": 5,
  "proxyConfiguration": { "useApifyProxy": true }
}
```

`items` is a list of domains or URLs (a bare domain is normalized to `https://` and reduced
to its root). One dataset row is returned per item; rows with `"found": false` are never
charged. `maxConcurrency` (default 5) caps parallel requests — kept conservative since this
target has no browser fallback. `proxyConfiguration` routes requests through Apify Proxy.

### Output

```json
{
  "query": "bbc.co.uk",
  "found": true,
  "data": {
    "domain": "bbc.co.uk",
    "sitemapIndexUrls": ["https://www.bbc.co.uk/sitemap.xml"],
    "sampledSitemap": "https://www.bbc.co.uk/sitemaps/https-index-uk-archive.xml",
    "urlCount": 500,
    "urls": [
      "https://www.bbc.co.uk/news/uk-12345678",
      "https://www.bbc.co.uk/sport/football/98765432"
    ],
    "truncated": true
  },
  "scrapedAt": "2026-08-20T14:03:11.000Z"
}
```

A row is only marked `found: true` — and only then billed — once a sitemap resolves and
yields at least one `<loc>` URL. A domain with no `Sitemap:` line in `robots.txt` and no
sitemap at the conventional fallback path (or a sitemap index whose first child sitemap is
unreachable) comes back as `found: false` and is never charged. `urls` is capped at 500
entries per row; `truncated: true` means the actual sitemap had more.

### Use cases

- Seed a broader crawl or scraping job with a real, authoritative list of a site's known
  URLs instead of guessing at site structure.
- Audit whether a site publishes a sitemap at all, and where it points, as part of a
  technical SEO review.
- Compare `urlCount` across a portfolio of sites to spot ones with unusually thin or
  missing sitemap coverage.
- Pull a sample of a competitor's published content/product URLs for content-gap analysis.
- Verify that a sitemap index correctly points to a working child sitemap after a site
  migration or CMS change.

### Pricing

$5 per 1,000 results, plus a $0.005 start fee. Misses (`found:false`) are never charged.

### Use it from Clay, n8n, Make, or an AI agent

This actor runs synchronously over plain HTTP — call it directly from a script, a workflow tool, or an AI agent, no Apify Console needed once you have an API token.

```bash
curl "https://api.apify.com/v2/acts/accountable_eel~sitemap-url-discovery/run-sync-get-dataset-items?token=<YOUR_TOKEN>" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"items":["bbc.co.uk"]}'
```

**n8n.** Add an HTTP Request node: Method `POST`, URL `https://api.apify.com/v2/acts/accountable_eel~sitemap-url-discovery/run-sync-get-dataset-items?token=<YOUR_TOKEN>`, Body Content Type `JSON`, JSON Body `{"items":["bbc.co.uk"]}` (swap in an expression from an earlier node for a real value).

**Clay.** Add an "HTTP API" column: Method `POST`, URL `https://api.apify.com/v2/acts/accountable_eel~sitemap-url-discovery/run-sync-get-dataset-items?token=<YOUR_TOKEN>`, Body `{"items":["{{value}}"]}`, mapping the row's value into the `items` array.

**MCP.** In Claude, Cursor, or any MCP client with the Apify MCP server, ask for "Sitemap URL Discovery | Apify" — the agent will find and run this actor.

### FAQ

**Why did a well-known site come back `found: false`?** Not every site declares a
`Sitemap:` line in `robots.txt` or serves one at `/sitemap.xml` — that's a real, correct
outcome for sites without a public sitemap, not a failure of the actor.

**Does it crawl the whole site to find URLs?** No — it only reads `robots.txt` and follows
the sitemap URL(s) declared there (or the conventional fallback path). It never crawls
arbitrary site pages.

**What happens with a sitemap index (a sitemap of sitemaps)?** The actor follows the first
child sitemap listed in the index one level deep and returns URLs from that child. It does
not aggregate every child sitemap in a large index.

**Is there a limit on how many URLs come back?** Yes — up to 500 per row. `urlCount`
reports the number found in the sampled sitemap, and `truncated: true` indicates more exist
than were returned.

**Does it respect `robots.txt` disallow rules?** No — this actor only reads the
`Sitemap:` directive from `robots.txt`, it does not evaluate `Disallow`/`Allow` crawl
rules, since it isn't crawling site pages.

**Can I pass a URL with a path instead of a bare domain?** Yes — any path is stripped down
to the domain's root before checking `robots.txt`, since sitemaps are declared per-domain,
not per-page.

# Actor input Schema

## `items` (type: `array`):

One item per line — see the item shape and examples below. Only the items we actually find are charged — never per run, and never for a miss.

## `maxConcurrency` (type: `integer`):

Parallel requests. Keep conservative — this target has no browser fallback, so getting blocked costs more than slow-and-steady.

## `proxyConfiguration` (type: `object`):

Apify Proxy config. Residential recommended for anti-bot-sensitive targets.

## Actor input object example

```json
{
  "items": [
    "bbc.co.uk"
  ],
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        "bbc.co.uk"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("accountable_eel/sitemap-url-discovery").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "items": ["bbc.co.uk"] }

# Run the Actor and wait for it to finish
run = client.actor("accountable_eel/sitemap-url-discovery").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    "bbc.co.uk"
  ]
}' |
apify call accountable_eel/sitemap-url-discovery --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,accountable_eel/sitemap-url-discovery"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ZqGaNTRvy5b1q2m0t/builds/gwnirJmjds9sUNar9/openapi.json