# Sitemap URL Extractor: All URLs from XML Sitemaps (`echolab/sitemap-url-extractor`) Actor

Sitemap extractor and sitemap.xml parser to extract all URLs from a website. Discover robots.txt and sitemap indexes; extract lastmod, news/image/video sitemaps and hreflang. Filter, deduplicate and export CSV/JSON URL lists.

- **URL**: https://apify.com/echolab/sitemap-url-extractor.md
- **Developed by:** [Echo Lab](https://apify.com/echolab) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor — extract all URLs from a website

### What it does

This fast HTTP-only **sitemap extractor** turns websites, domains or direct sitemap links into a deduplicated URL inventory. Use it as a sitemap.xml parser, XML sitemap scraper or lightweight sitemap checker, without a browser. It reads the URLs the site publishes, not its HTML navigation.

- Discover Sitemap directives in robots.txt, then try common sitemap paths until one works.
- Follow sitemap indexes with depth and loop protection; read XML, gzip (detected by magic bytes), and plain-text URL lists.
- Extract lastmod, changefreq, priority, image/video counts, news metadata and hreflang alternates.
- Filter URLs with Python regular expressions, date and domain filters; export JSON or CSV from Apify.
- Optionally check URL status with HEAD, falling back to a immediately closed GET on 405/501.

### Use cases

- SEO audit: inventory indexed candidates and inspect lastmod and sitemap metadata.
- Content inventory: export a clean URL list to a spreadsheet.
- Site migration: prepare old URL lists for redirect mapping.
- Monitor new pages: run on a daily schedule with lastmodAfter. This filters published dates; it is not a persistent diff database.
- Seed URL lists for crawlers and RAG pipelines, which can fetch content separately.

### Input

| Field | Default | Meaning |
|---|---|---|
| startUrls | https://www.sitemaps.org | Array of website URLs/domains or direct .xml, .xml.gz, .txt links |
| maxDepth | 5 | Index depth; root is 0 |
| maxUrlsPerSite | 200 in default input | Saved URL cap per start point; set 0 for unlimited |
| maxUrls | 0 | Total saved URL cap; 0 is unlimited |
| maxSitemapsPerSite | 500 | Maximum attempted files per start point |
| maxConcurrency | 5 | Global request limit, maximum 20; per-host concurrency 2 |
| includePatterns | \[] | Python regex strings: match any to include; empty includes all |
| excludePatterns | \[] | Python regex strings: any match excludes |
| lastmodAfter | empty | ISO date/time lower bound, inclusive |
| excludeMissingLastmod | false | Drop entries without lastmod, independently of date filter |
| sameDomainOnly | true | Match registrable domain, including subdomains |
| checkStatus | false | Optional HTTP status and final redirect checks; slower |

```json
{
  "startUrls": ["https://www.sitemaps.org", "https://www.wpbeginner.com/sitemap_index.xml"],
  "maxUrlsPerSite": 2000,
  "maxUrls": 3000,
  "maxDepth": 5,
  "lastmodAfter": "2022-01-01",
  "excludePatterns": ["/tag/", "\\?"],
  "sameDomainOnly": true,
  "checkStatus": false
}
```

### Output

One useful dataset item per unique URL **across the entire run**. Overlapping start points are not billed twice. The first saved listing supplies site and sitemapUrl. Per-site SUMMARY counts describe rows actually saved for that start point.

```json
{
  "url": "https://www.sitemaps.org/protocol.html",
  "lastmod": "2022-12-15",
  "changefreq": null,
  "priority": null,
  "sitemapUrl": "https://www.sitemaps.org/sitemap.xml",
  "site": "https://www.sitemaps.org",
  "imageCount": 0,
  "videoCount": 0,
  "newsTitle": null,
  "newsPublicationDate": null,
  "alternates": [],
  "extractedAt": "2026-11-05T00:00:00+00:00"
}
```

| Field | Meaning |
|---|---|
| url | Published page URL |
| lastmod | Date/time normalized to ISO 8601 when parseable, otherwise preserved |
| changefreq | Published hint or null |
| priority | Published finite number or null |
| sitemapUrl | File that first listed the URL |
| site | Normalized input start point |
| imageCount / videoCount | Number of image/video entries |
| newsTitle / newsPublicationDate | News sitemap metadata or null |
| alternates | Array of {hreflang, href}, empty if absent |
| statusCode / redirectUrl | Only with checkStatus; null on request failure; redirectUrl null without redirect |
| extractedAt | UTC extraction timestamp |

No billable error rows are saved. Key-value record `SUMMARY` contains an array of `{site, sitemapsFound, sitemapsFailed: [{url, reason}], urls}`. Empty or failed sites are logged and no-sitemap sites are mentioned in the final status. Partial malformed files may have already emitted valid preceding URLs, with the parse failure recorded in SUMMARY.

### Pricing

You pay only for URLs saved to the dataset, plus a tiny per-run start fee; Apify platform usage is included in the price. Error logs and SUMMARY records are not dataset items. The Actor stops normally when a dataset charge limit is reached. The current price per 1,000 URLs is shown on the Pricing tab of this page.

### Tips / limits

Only what the site publishes in sitemaps can be extracted. JS-only sites without sitemaps return nothing. lastmod is the publisher's claim, not proof of freshness; missing and unparseable dates are kept unless missing dates are explicitly excluded. No persistent cross-run deduplication is performed. Invalid start points are skipped with a warning unless all are invalid. SUMMARY status distinguishes completed, stopped and skipped sites. For large inventories, use a practical maxUrls cap: around one million unique URLs per run is a planning ceiling at 256 MB, not a guarantee; long URLs and several near-50 MB sitemaps can need more memory. Raise run memory to 512 MB or split inputs for large sites. Deduplication sets grow with discovered URLs, including filtered-out URLs.

The default is a verified small sitemap at sitemaps.org, capped at 200 saved URLs for cheap daily tests. Set maxUrlsPerSite to 0 to extract an unlimited inventory. Sitemap index depth and file limits still apply. Files are capped at 50 MB uncompressed, including gzip-bomb protection. XML is iteratively parsed; DTD/entity declarations are rejected. Maximum three request attempts, exponential backoff and Retry-After support (waits capped at 30 seconds) apply to transient errors. A site that publishes unusually many duplicate entries still takes time to parse. Filters are Python regex (not globs); avoid expensive pathological expressions.

### Compliance & data source

Official public robots.txt and sitemap endpoints only. No login, cookies, proxies, CAPTCHA solving or anti-bot bypass. The sitemap is explicitly published for machines; robots.txt is used for discovery, not to crawl HTML. Optional status checks contact only URLs listed in sitemaps, with HEAD or an immediately closed GET fallback. User-Agent: `EchoLab-Apify-Actor/sitemap-url-extractor (+https://apify.com/echolab)`.

No contact details or private-person fields are collected; sitemap URLs and news metadata are copied as published. You are responsible for ensuring the selected sources and exported metadata comply with applicable terms and privacy rules.

### Support

Open an issue on the Actor's Issues tab.

More real output, worked pricing examples and a comparison with alternatives: https://echolab-tools.pages.dev/sitemap-url-extractor/

# Actor input Schema

## `startUrls` (type: `array`):

Website URLs/domains or direct .xml, .xml.gz or .txt sitemaps. Invalid entries are skipped with a warning if at least one valid start point remains. URLs are deduplicated across all start points.

## `maxDepth` (type: `integer`):

Maximum sitemap index recursion depth; root is depth 0.

## `maxUrlsPerSite` (type: `integer`):

Maximum URLs saved per start point; 0 is unlimited. Default input caps this at 200.

## `maxUrls` (type: `integer`):

Total URL limit; 0 is unlimited.

## `maxSitemapsPerSite` (type: `integer`):

Maximum sitemap files attempted per start point.

## `maxConcurrency` (type: `integer`):

Global HTTP concurrency, 1–20; per-host limit 2.

## `checkStatus` (type: `boolean`):

Optional HEAD status checks, GET fallback on 405/501. Slower.

## `sameDomainOnly` (type: `boolean`):

Keep the same registrable domain, allowing subdomains.

## `includePatterns` (type: `array`):

Python regex strings matched against URL; any include must match.

## `excludePatterns` (type: `array`):

Python regex strings; any match excludes URL.

## `lastmodAfter` (type: `string`):

ISO date/datetime lower bound; missing or unparseable lastmod is kept.

## `excludeMissingLastmod` (type: `boolean`):

Drop URLs without lastmod.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.sitemaps.org"
  ],
  "maxDepth": 5,
  "maxUrlsPerSite": 200,
  "maxUrls": 0,
  "maxSitemapsPerSite": 500,
  "maxConcurrency": 5,
  "checkStatus": false,
  "sameDomainOnly": true,
  "includePatterns": [],
  "excludePatterns": [],
  "lastmodAfter": "",
  "excludeMissingLastmod": false
}
```

# Actor output Schema

## `results` (type: `string`):

One row per unique URL per start point.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.sitemaps.org"
    ],
    "maxDepth": 5,
    "maxUrlsPerSite": 200,
    "maxUrls": 0,
    "maxSitemapsPerSite": 500,
    "maxConcurrency": 5,
    "checkStatus": false,
    "sameDomainOnly": true,
    "includePatterns": [],
    "excludePatterns": [],
    "lastmodAfter": "",
    "excludeMissingLastmod": false
};

// Run the Actor and wait for it to finish
const run = await client.actor("echolab/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://www.sitemaps.org"],
    "maxDepth": 5,
    "maxUrlsPerSite": 200,
    "maxUrls": 0,
    "maxSitemapsPerSite": 500,
    "maxConcurrency": 5,
    "checkStatus": False,
    "sameDomainOnly": True,
    "includePatterns": [],
    "excludePatterns": [],
    "lastmodAfter": "",
    "excludeMissingLastmod": False,
}

# Run the Actor and wait for it to finish
run = client.actor("echolab/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.sitemaps.org"
  ],
  "maxDepth": 5,
  "maxUrlsPerSite": 200,
  "maxUrls": 0,
  "maxSitemapsPerSite": 500,
  "maxConcurrency": 5,
  "checkStatus": false,
  "sameDomainOnly": true,
  "includePatterns": [],
  "excludePatterns": [],
  "lastmodAfter": "",
  "excludeMissingLastmod": false
}' |
apify call echolab/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,echolab/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/emNm0qXX1BNf8dHG6/builds/YIc6ZYXaRLgbxnWBu/openapi.json