# Sitemap Extractor & URL Discovery (`madpai/sitemap-url-discovery`) Actor

Find the page URLs a website lists in robots.txt and XML sitemaps (sitemap indexes, gzip, text sitemaps, hreflang, lastmod), with a polite shallow-crawl fallback. Clean JSON for crawlers, SEO audits, RAG and AI agents.

- **URL**: https://apify.com/madpai/sitemap-url-discovery.md
- **Developed by:** [William Schultz](https://apify.com/madpai) (community)
- **Categories:** SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.40 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor & URL Discovery

### What does this sitemap extractor do?

Sitemap Extractor & URL Discovery finds the URLs a website publishes. Give it domains, homepages or direct sitemap URLs. It reads `robots.txt`, finds the sitemaps (including nested sitemap indexes, gzipped sitemaps, plain-text sitemaps and RSS or Atom feeds used as sitemaps), and returns one clean record per URL with its lastmod date, priority, hreflang alternates and the sitemap it came from. If a site has no usable sitemap, it can fall back to a shallow, HTTP-only crawl of same-site links from the homepage. Each record says whether the URL came from a sitemap or from that crawl.

It uses plain HTTP requests, no browser, no login and no proxies. It respects `robots.txt` and waits about a second between requests to the same site by default.

### Who is this URL extractor for?

- **SEO teams** checking which pages a site exposes to search engines and AI crawlers, or what changed recently (filter by lastmod).
- **Developers building RAG or crawl pipelines** who need a URL list before fetching content.
- **AI agents** using Apify's MCP server that need to map a site before reading it.

### Input

| Field | Type | Default | What it does |
|---|---|---|---|
| `urls` (required) | list | prefill `crawlee.dev` | Domains (`example.com`), homepages, direct sitemap URLs or RSS/Atom feed URLs. |
| `maxUrls` | integer | 1000 (prefill 100) | Total cap on URLs saved. You pay per URL saved, so this also caps cost. |
| `maxUrlsPerSite` | integer | 0 (no cap) | Stops one big site from using the whole budget. |
| `includeRegex` | text | none | Keep only URLs that match, e.g. `/blog/`. |
| `excludeRegex` | text | none | Drop URLs that match, e.g. `/tag/|/page/\d+`. |
| `lastmodAfter` | date | none | Keep only URLs modified on or after this date. URLs with no lastmod are dropped when this is set. |
| `fallbackCrawl` | boolean | true | Follow same-site links from the homepage if there's no usable sitemap. |
| `maxCrawlPages` | integer | 30 | Pages fetched per site during the fallback crawl. |
| `respectRobotsTxt` | boolean | true | Don't fetch sitemaps or pages that robots.txt disallows. Can only be turned off together with `ownSiteConfirmed`. |
| `checkRobotsForUrls` | boolean | true | Adds `allowedByRobots` to each URL, checked against the robots.txt of the URL's own host. URLs on other hosts cost one extra robots.txt request per host (at most 10 hosts; beyond that `null` with a `robotsNote`). |
| `maxSitemapsPerDomain` | integer | 300 | Safety cap on sitemap files fetched per site. |
| `requestDelayMs` | integer | 1000 | Delay between requests to the same site (about 1 request per second, longer if robots.txt sets a Crawl-delay). Shorter delays need `ownSiteConfirmed`. |
| `maxConcurrency` | integer | 5 | Sites processed in parallel (each site still gets the per-site delay). |
| `ownSiteConfirmed` | boolean | false | Tick only for sites you own or are authorised to crawl. Unlocks turning off robots.txt and delays under 1 second. |

Example input:

```json
{
  "urls": ["crawlee.dev", "https://docs.apify.com/sitemap.xml"],
  "maxUrls": 100,
  "excludeRegex": "/tag/"
}
```

### Output

One item per unique URL. Here's a real item from a test run (1 Nov 2026):

```json
{
  "url": "https://crawlee.dev/blog",
  "domain": "crawlee.dev",
  "inputUrl": "crawlee.dev",
  "discoveredVia": "sitemap",
  "sourceSitemap": "https://crawlee.dev/sitemap.xml",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "alternates": [],
  "imageCount": 0,
  "allowedByRobots": true,
  "robotsNote": null,
  "discoveredAt": "2026-11-01T20:54:56.778Z"
}
```

Each run also saves an `OUTPUT_SUMMARY` record and a `SUMMARY.md` in the key-value store. Per site, these record whether robots.txt was found, which sitemaps were fetched, any sitemap errors, URLs found versus saved, and whether the sitemap or the crawl was used. Agents and scripts can read them without paging through the dataset.

### How much does it cost to extract sitemap URLs?

It's pay per event: **$0.40 per 1,000 URLs saved** ($0.0004 per URL), charged as one `url` event for each unique URL saved to the dataset. Example: a 10,000-URL site costs $4.00 in URL events. Any other charges Apify applies are shown on the Pricing tab. Your `maxUrls` setting and your maximum charge per run both cap the cost, and the run stops when either is reached. Sites or sitemaps that fail or can't be reached save no URLs, so they add no `url` events.

### Use with AI agents

The Actor is designed to run with limited permissions, so AI agents can call it as a tool through the Apify MCP server. Example prompt: "List the blog URLs on crawlee.dev that changed since 1 October."

### Limits

- Sitemaps only list what the site owner chose to include. The fallback crawl is shallow (`maxCrawlPages`), so on sites without a sitemap you may not get every page.
- `lastmod` is whatever the site reports. Many sites leave it out or set it on every page.
- A sitemap that's blocked, requires a login or returns an error is listed in the run summary as an error. The Actor never works around it.
- Inputs that point at private, loopback, link-local or other reserved network addresses are never requested; the run summary lists them with a `blocked-private` error. Sitemap entries and redirects to such addresses are refused the same way.
- URLs beyond your caps are not saved. The run summary shows how many were found compared with how many were saved.

### FAQ

**Does it download page content?** No. It returns URLs and sitemap metadata. Pair it with a crawler to get page content.
**What happens if a site has no sitemap?** With `fallbackCrawl` on (the default), it collects same-site links from the homepage and marks those URLs `"discoveredVia": "crawl"`. With it off, that site returns no URLs, and the run summary's `noSitemapReason` field says why, for example that robots.txt lists no sitemap and `/sitemap.xml` returned HTTP 404. The same reason is added to `sitemapErrors` and the log.
**Can I monitor new pages?** Yes. Schedule it with `lastmodAfter` set to a date, or compare the URL lists from two runs.

### Legal

It collects public URLs and sitemap metadata only, and no personal data. You're responsible for using the results in line with the target site's terms.

### Changelog

See the Changelog tab (CHANGELOG.md in the source).

# Changelog

This Actor's version history is a separate document: https://apify.com/madpai/sitemap-url-discovery/changelog.md

# Actor input Schema

## `urls` (type: `array`):

Domains (example.com), homepage URLs, or direct sitemap URLs. Sitemaps are found via robots.txt, then common paths like /sitemap.xml.

## `maxUrls` (type: `integer`):

Total cap on URLs saved across all inputs. You pay per URL saved, so this also caps cost.

## `maxUrlsPerSite` (type: `integer`):

Cap per input so one big site can't use the whole budget. 0 = no per-site cap.

## `includeRegex` (type: `string`):

Keep only URLs matching this case-insensitive regular expression, e.g. /blog/.

## `excludeRegex` (type: `string`):

Drop URLs matching this case-insensitive regular expression, e.g. /tag/|/page/\d+.

## `lastmodAfter` (type: `string`):

Keep only URLs whose sitemap lastmod is on or after this date (YYYY-MM-DD). Useful for change monitoring. URLs without lastmod are dropped when set.

## `fallbackCrawl` (type: `boolean`):

If a site has no usable sitemap, follow same-site links from the homepage (HTTP only, shallow).

## `maxCrawlPages` (type: `integer`):

Pages fetched per site during the fallback crawl. Each page can reveal many URLs.

## `respectRobotsTxt` (type: `boolean`):

Don't fetch sitemaps or pages disallowed by robots.txt. Can only be turned off together with "I own these sites".

## `checkRobotsForUrls` (type: `boolean`):

Add allowedByRobots to each URL, checked against the robots.txt of the URL's own host (one extra robots.txt request per other host, max 10; otherwise null with a robotsNote).

## `maxSitemapsPerDomain` (type: `integer`):

Safety cap on sitemap files fetched per site, including nested indexes.

## `requestDelayMs` (type: `integer`):

Per-site delay between requests (about 1 request per second by default). robots.txt Crawl-delay is honoured if longer. Values under 1000 ms only apply with "I own these sites" checked.

## `maxConcurrency` (type: `integer`):

How many sites to process at once.

## `ownSiteConfirmed` (type: `boolean`):

Only check this for your own sites. It allows turning off robots.txt and using delays under 1 second.

## Actor input object example

```json
{
  "urls": [
    "crawlee.dev"
  ],
  "maxUrls": 100,
  "maxUrlsPerSite": 0,
  "fallbackCrawl": true,
  "maxCrawlPages": 30,
  "respectRobotsTxt": true,
  "checkRobotsForUrls": true,
  "maxSitemapsPerDomain": 300,
  "requestDelayMs": 1000,
  "maxConcurrency": 5,
  "ownSiteConfirmed": false
}
```

# Actor output Schema

## `urls` (type: `string`):

Dataset with one item per unique URL: url, domain, inputUrl, discoveredVia (sitemap|crawl), sourceSitemap, lastmod, changefreq, priority, alternates, imageCount, allowedByRobots (checked against each URL's own host; null = not checked), robotsNote, discoveredAt.

## `summary` (type: `string`):

JSON summary per input site: robots.txt found, sitemaps fetched, sitemap errors, a noSitemapReason when no sitemap was found, URLs found vs saved, and whether sitemaps or the crawl were used.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "crawlee.dev"
    ],
    "maxUrls": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("madpai/sitemap-url-discovery").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["crawlee.dev"],
    "maxUrls": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("madpai/sitemap-url-discovery").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "crawlee.dev"
  ],
  "maxUrls": 100
}' |
apify call madpai/sitemap-url-discovery --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,madpai/sitemap-url-discovery"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/1yFlBrBrPwglFBSF4/builds/oe5IO3sRh4WjMBvGf/openapi.json