# Sitemap URL Extractor: Bulk XML Sitemap Parser (`hillcountrybots/sitemap-url-extractor`) Actor

Extract every URL from a website's XML sitemaps: robots.txt discovery, nested sitemap indexes and gzip. Returns url, lastmod, changefreq, priority and source sitemap. Include/exclude filters. Plain HTTP, no browser. $0.50 per 1,000 URLs, failures free.

- **URL**: https://apify.com/hillcountrybots/sitemap-url-extractor.md
- **Developed by:** [Hill Country Bots](https://apify.com/hillcountrybots) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.50 / 1,000 extracted urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor: Bulk XML Sitemap Parser

Extract **every URL from a website's XML sitemaps** in one run. This is a bulk **sitemap URL extractor** and **XML sitemap parser**. The Actor reads **robots.txt**, follows **nested sitemap indexes**, and decompresses **gzip** sitemaps. Each row comes back with the URL, **lastmod**, **changefreq**, **priority** and the **source sitemap**. Optional include and exclude filters. **Plain HTTP, no browser, no login, no personal data.** **$0.50 per 1,000 extracted URLs. Failed fetches, invalid inputs and filtered URLs are free.**

### What does this sitemap URL extractor do?

You give it a website or a sitemap URL. It then:

- Discovers sitemap files from **robots.txt** (`Sitemap:` lines) and, if you want, from common paths (`/sitemap.xml`, `/sitemap_index.xml`, and their `.gz` versions)
- Follows **sitemap indexes** recursively, including gzip-compressed files
- Returns one row per unique URL: `url`, `lastmod`, `changefreq`, `priority`, `sourceSitemap`
- Applies optional **include** and **exclude** patterns (`*` wildcards or `/regex/`)
- Caps the run with `maxUrls` and `maxSitemaps`
- Refuses private, local and internal addresses

It does not crawl pages, render JavaScript, or fetch anything except robots.txt and sitemap files.

### Who is it for?

- **SEO agencies and consultants:** export a site's URL list for audits, migrations or index coverage checks
- **Developers:** turn a sitemap into a clean CSV or JSON list as a first step before a PageSpeed run or a content crawl
- **AI agents and automations:** stable JSON through the Apify API or MCP, with a predictable price per extracted URL

### Quick start

1. Paste a website (for example `https://www.sitemaps.org/`) or a direct sitemap URL. The form comes prefilled, so **Start** works right away.
2. Leave sitemap discovery on unless you already have the exact sitemap URL. Optionally add include or exclude patterns, or raise `maxUrls` for a full site (the example starts at 25 so a one-click run finishes in seconds).
3. Click **Start**, then download the results as JSON, CSV or Excel, or read them through the API.

To refresh a site's URL list on a schedule, create a Schedule in Apify Console and send the dataset to Google Sheets, or feed the `url` column into the [Bulk PageSpeed Insights Test](https://apify.com/hillcountrybots/bulk-pagespeed-checker).

#### Sample output

This table is the first five of 20 URLs from a real run on **November 10, 2026, at 9:34 PM CT** (America/Chicago), starting at sitemaps.org. You pay about **$0.50 per 1,000** extracted URLs. Failed fetches and filtered URLs are free.

![Sample output from a real Sitemap URL Extractor run](https://api.apify.com/v2/key-value-stores/2kusIBSeqRffwM3Q9/records/sitemap-sample-output.png?signature=19lgr8PJ8CmGQjJEyGdk5)

| URL | lastmod | Source sitemap |
|---|---|---|
| https://www.sitemaps.org/ | 2016-11-21 | https://www.sitemaps.org/sitemap.xml |
| https://www.sitemaps.org/protocol.html | 2022-12-15 | https://www.sitemaps.org/sitemap.xml |
| https://www.sitemaps.org/faq.html | 2016-11-21 | https://www.sitemaps.org/sitemap.xml |
| https://www.sitemaps.org/terms.html | 2016-11-21 | https://www.sitemaps.org/sitemap.xml |
| https://www.sitemaps.org/da/ | 2016-11-21 | https://www.sitemaps.org/sitemap.xml |

If no sitemap can be found or a start URL is invalid, the run writes a row with `status: "error"`, `errorCode` and `errorMessage` instead of finishing as a silent success with empty data. Those rows are not charged.

#### Example inputs

Discover and parse XML sitemaps from a website homepage (robots.txt plus common paths):

```json
{
  "startUrls": ["https://www.sitemaps.org/"],
  "discoverFromRobots": true,
  "tryCommonPaths": true,
  "maxUrls": 500
}
```

Extract URLs from a direct sitemap.xml (no discovery):

```json
{
  "startUrls": ["https://www.sitemaps.org/sitemap.xml"],
  "discoverFromRobots": false,
  "tryCommonPaths": false,
  "maxUrls": 500
}
```

Keep only matching paths with an include pattern:

```json
{
  "startUrls": ["https://www.sitemaps.org/"],
  "includePatterns": ["https://www.sitemaps.org/protocol*"],
  "maxUrls": 500
}
```

Cap a large site at 100 URLs ($0.05 maximum):

```json
{
  "startUrls": ["https://www.sitemaps.org/"],
  "discoverFromRobots": true,
  "tryCommonPaths": true,
  "maxUrls": 100
}
```

Ready-to-run example tasks (pre-filled inputs on this Actor's Store page):

- [Extract URLs from a website's XML sitemaps](https://apify.com/hillcountrybots/sitemap-url-extractor/examples/sitemap-homepage-discover)
- [Parse a direct sitemap.xml file](https://apify.com/hillcountrybots/sitemap-url-extractor/examples/sitemap-direct-xml)

### Input

| Field | Type | Default | What it does |
|---|---|---|---|
| `startUrls` | array of strings | (required, prefilled) | Website home pages and/or direct sitemap URLs. Private, local and internal addresses are refused. |
| `discoverFromRobots` | boolean | `true` | Fetch `/robots.txt` and collect every `Sitemap:` line. |
| `tryCommonPaths` | boolean | `true` | Also try `/sitemap.xml`, `/sitemap_index.xml` and their `.gz` versions. Missing files are skipped, not charged. |
| `includePatterns` | array | (none) | If set, only URLs matching at least one pattern are kept. `*` wildcards or `/regex/`. |
| `excludePatterns` | array | (none) | URLs matching any of these patterns are dropped. |
| `maxUrls` | integer | `10000` | Exact cap on unique URLs returned (and charged). Caps spend at `maxUrls` × $0.0005. |
| `maxSitemaps` | integer | `200` | Exact cap on unique sitemap files to fetch (indexes and urlsets). |
| `maxConcurrency` | integer 1-10 | `4` | Sitemap files fetched in parallel. |
| `requestTimeoutSecs` | integer | `30` | Seconds to wait on one file, then retry (up to 3 attempts). |

### Output example

```json
{
  "url": "https://www.sitemaps.org/protocol.html",
  "lastmod": "2016-11-21",
  "changefreq": "monthly",
  "priority": "0.8",
  "sourceSitemap": "https://www.sitemaps.org/sitemap.xml",
  "status": "ok",
  "errorCode": null,
  "errorMessage": null,
  "extractedAt": "2026-11-10T22:10:00.000Z"
}
```

A start URL or sitemap file that could not be used is **not charged** and looks like this:

```json
{"url": "https://example.com/missing-sitemap.xml", "status": "error", "errorCode": "HTTP_404", "errorMessage": "HTTP 404", "sourceSitemap": "https://example.com/missing-sitemap.xml"}
```

The key-value store also gets a `RUN_SUMMARY` record with totals (`extracted`, `failed`, `filtered`, `sitemapsFetched`, `indexesFollowed`).

#### Output fields (for AI agents and integrations)

- `status`: `ok` means a URL was extracted from a sitemap. `error` means a start URL or sitemap file could not be used and is not charged.
- `url`: the page URL from `<loc>`, or the rejected start/sitemap URL on an error row.
- `lastmod`, `changefreq`, `priority`: sitemap fields, when present. `priority` is kept as the original string (0.0-1.0).
- `sourceSitemap`: the sitemap file the URL was read from.
- `errorCode` is one of: `INVALID_URL`, `PRIVATE_URL`, `HTTP_<code>`, `TIMEOUT`, `NETWORK_ERROR`, `INVALID_SITEMAP`, `EMPTY_SITEMAP`.

### Use it from the API, MCP or an AI agent

- **Actor ID:** `hillcountrybots/sitemap-url-extractor`
- **One API call** returns the rows:

```bash
curl -X POST "https://api.apify.com/v2/acts/hillcountrybots~sitemap-url-extractor/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":["https://www.sitemaps.org/"],"maxUrls":1000}'
```

- **MCP (Claude, Cursor, ChatGPT and other MCP clients):** add `https://mcp.apify.com?tools=hillcountrybots/sitemap-url-extractor`.
- **For agents:** branch on `status`. Each `ok` row is one extracted URL. Error rows are free.

### How much does it cost?

**$0.50 per 1,000 extracted URLs**, which is $0.0005 per `url-extracted` event (one unique URL kept after filters). You pay nothing for:

- invalid start URLs
- failed sitemap fetches
- URLs dropped by include/exclude filters
- the run summary

The Actor's compute is included, so there's no separate platform usage charge. For example, 2,000 URLs from a large sitemap index is $1.00. To cap spend, use `maxUrls` or the run's maximum charge setting.

### Limits

- This tool reads sitemap files only. It does not crawl pages, check HTTP status of each URL, or render JavaScript.
- A URL appears in the output only if a sitemap lists it. Pages missing from the sitemap are not invented.
- `lastmod` is whatever the sitemap published; some sites omit it or keep it stale.
- Very large sites can have hundreds of sitemap files. Use `maxUrls` and `maxSitemaps` to cap the run.

### FAQ

**Do I need a sitemap URL, or will a homepage do?** A homepage is enough. The Actor reads `robots.txt` and also tries common sitemap paths.

**Does it support gzip sitemaps and sitemap indexes?** Yes. `.gz` files and nested indexes are followed, up to `maxSitemaps`.

**Does it crawl the website?** No. It only fetches robots.txt and sitemap XML.

**Does it collect personal data?** No. It returns URLs and sitemap metadata only.

**Can I feed the URLs into the PageSpeed checker?** Yes. Export the `url` column and paste it into [Bulk PageSpeed Insights Test](https://apify.com/hillcountrybots/bulk-pagespeed-checker).

### More tools from Hill Country Bots

- [Bulk PageSpeed Insights Test](https://apify.com/hillcountrybots/bulk-pagespeed-checker): Lighthouse scores and Core Web Vitals for many URLs. $3 per 1,000 audits, failures free.
- [Bulk Domain Availability Checker](https://apify.com/hillcountrybots/domain-availability-checker): Bulk available/taken checks from official registry RDAP, with registrar and expiry dates. $1 per 1,000 domains, invalid inputs and errors free.
- [Tech Stack Detector](https://apify.com/hillcountrybots/tech-stack-detector): CMS, ecommerce, analytics, CDN and frameworks for many websites. $0.50 per 1,000 sites, failures free.
- [SEO Audit Tool](https://apify.com/hillcountrybots/seo-audit): 0-100 SEO score per page, top 5 fixes and a client-ready HTML report. $2 per 1,000 pages, failures free.

### Support

Found a missed sitemap or need a field added? Open an issue on this Actor's **Issues** tab with the run ID and the website. We reply within 2 business days (US Central time). Built and maintained by **Hill Country Bots**.

# Actor input Schema

## `startUrls` (type: `array`):

Required. Website home pages and/or direct sitemap URLs (https://example.com/ or https://example.com/sitemap.xml). A domain without https:// is accepted. Private, local and internal addresses are refused (free).

## `discoverFromRobots` (type: `boolean`):

Fetch /robots.txt on each website and collect every Sitemap: line.

## `tryCommonPaths` (type: `boolean`):

If robots.txt lists no sitemap (or in addition to it), try /sitemap.xml, /sitemap_index.xml and their .gz versions on the same origin.

## `includePatterns` (type: `array`):

If set, only URLs matching at least one pattern are kept. Use \* as a wildcard (e.g. https://example.com/blog/\*) or a /regex/ in slashes. Compared against the full URL. Leave empty to keep all.

## `excludePatterns` (type: `array`):

URLs matching any of these patterns are dropped (after include filters). Same syntax as include patterns: \* wildcards or /regex/.

## `maxUrls` (type: `integer`):

Exact maximum number of unique URLs to return in this run. Caps spend: at most maxUrls x $0.0005. The example starts at 25 so a one-click run finishes in seconds; raise it for a full site. Invalid, failed and filtered rows do not count toward this cap and are not charged.

## `maxSitemaps` (type: `integer`):

Exact maximum number of unique sitemap files (indexes and urlsets) to fetch, to cap run time on huge sites. Common-path probes that 404 still count as a fetched attempt.

## `maxConcurrency` (type: `integer`):

Sitemap files fetched in parallel (1-10). Keep this modest; many sites rate-limit sitemap requests.

## `requestTimeoutSecs` (type: `integer`):

Seconds to wait for one sitemap or robots.txt response before retrying (up to 3 attempts with backoff).

## Actor input object example

```json
{
  "startUrls": [
    "https://www.sitemaps.org/"
  ],
  "discoverFromRobots": true,
  "tryCommonPaths": true,
  "maxUrls": 25,
  "maxSitemaps": 200,
  "maxConcurrency": 4,
  "requestTimeoutSecs": 30
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.sitemaps.org/"
    ],
    "maxUrls": 25
};

// Run the Actor and wait for it to finish
const run = await client.actor("hillcountrybots/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://www.sitemaps.org/"],
    "maxUrls": 25,
}

# Run the Actor and wait for it to finish
run = client.actor("hillcountrybots/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.sitemaps.org/"
  ],
  "maxUrls": 25
}' |
apify call hillcountrybots/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hillcountrybots/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/eur4OckdE3euxyjaz/builds/QRQR7y5MeVQjD8MJJ/openapi.json