# Sitemap URL Extractor: Get All URLs From Any Website (`aaronallday89/sitemap-url-extractor`) Actor

Get every URL a website lists in its sitemaps, as clean rows: URL, lastmod, changefreq, priority and source sitemap. Auto-finds sitemaps via robots.txt, follows sitemap indexes, reads .xml.gz, text and RSS sitemaps. For SEO audits, crawl lists and RAG pipelines. $1.00 per 1,000 URLs.

- **URL**: https://apify.com/aaronallday89/sitemap-url-extractor.md
- **Developed by:** [Aaron Allday](https://apify.com/aaronallday89) (community)
- **Categories:** SEO tools, Developer tools, E-commerce
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.23 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor: Get All URLs From Any Website

**Who it's for:** SEO specialists, agencies, developers and data/AI teams who need a website's full URL list without crawling it.

**The problem it solves:** Crawling a whole site just to list its pages is slow and expensive. Sitemaps already list them, but they are split across indexes, gzipped files and odd formats. This Actor reads all of them and gives you one clean, de-duplicated URL list.

**Use cases:**

- SEO audits: find stale, orphan or unexpected URLs with `lastmod`
- Build input lists for crawlers, PageSpeed audits, link checkers or RAG/LLM ingestion
- Site migrations: compare old vs new URL inventories
- Monitor competitors' new pages on a schedule

**Quick example:** `{"startUrls":[{"url":"https://crawlee.dev"}]}` → one row per URL: `url`, `lastmod`, `changefreq`, `priority`, `sourceSitemap`.

**Price:** **$1.00 per 1,000 URLs** on the Free plan, as low as $0.23 on higher plans. Comparable sitemap Actors in the Store charge $0.50 to $30 per 1,000 results on the Free plan. Discovering sitemaps, duplicates and errors are free.

Get a clean list of **every URL a website publishes in its sitemaps**, with `lastmod`, `changefreq`, `priority` and the sitemap each URL came from. Give it a homepage and it finds the sitemaps for you. Give it a sitemap and it follows every child sitemap. No browser and no proxies, so it's **fast and cheap**: tens of thousands of URLs per minute.

### What it does

- **Finds sitemaps automatically.** It reads `Sitemap:` lines from `robots.txt`. If there are none, it tries common locations: `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemap.txt`.
- **Follows sitemap indexes** to any depth (WordPress, Yoast, Rank Math, Shopify, Wix, Next.js, Docusaurus and so on).
- **Reads every common format:** XML `urlset`, `sitemapindex`, **gzipped** `.xml.gz` (detected from the file contents, not the name), plain-text sitemaps, and RSS / Atom feeds.
- **Collects extra details:** image count, video and news flags, and the number of hreflang alternates for each URL.
- **Filters:** max URLs, include/exclude regex, and a same-domain-only option.
- **Handles errors:** timeouts, retries with backoff on 429 / 5xx / network errors, removal of duplicate URLs, and a size guard for very large files. A full run summary, including any errors, is saved to the `SUMMARY` record in the key-value store.

### Use cases

- SEO audits: find orphan, stale or unexpected URLs, and check `lastmod` freshness
- Prepare URL lists for crawlers, scrapers, Lighthouse/PageSpeed audits or link checkers
- Content inventories and migration checks (old vs new site)
- Monitor competitors' new pages: schedule a run and compare datasets
- Feed URLs into other Apify Actors

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array | `https://crawlee.dev` | Website URLs (sitemaps discovered automatically) or direct sitemap URLs |
| `maxUrls` | integer | 1000 | Maximum URLs to output (0 = no limit; the run's max cost still applies) |
| `includeRegex` | string | – | Keep only URLs matching this regex (case-insensitive) |
| `excludeRegex` | string | – | Drop URLs matching this regex |
| `sameDomainOnly` | boolean | true | Drop URLs on other hosts (`www.` is ignored) |
| `discoverCommonPaths` | boolean | true | Try common sitemap paths when robots.txt lists none |
| `maxSitemaps` | integer | 500 | Safety cap on sitemap files fetched |
| `maxConcurrency` | integer | 10 | Sitemap files downloaded in parallel |
| `requestTimeoutSecs` | integer | 30 | Timeout per request (each request is retried up to 3×) |

#### Example input

```json
{
  "startUrls": [
    { "url": "https://wordpress.org" },
    { "url": "https://developer.mozilla.org/sitemap.xml" }
  ],
  "maxUrls": 20000,
  "includeRegex": "/docs/|/blog/",
  "excludeRegex": "\\.(pdf|jpg|png)$",
  "sameDomainOnly": true
}
```

### Output

One dataset item per unique URL:

```json
{
  "url": "https://crawlee.dev/blog",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sourceSitemap": "https://crawlee.dev/sitemap.xml",
  "sitemapType": "urlset",
  "sitemapDepth": 0,
  "inputUrl": "https://crawlee.dev/",
  "imageCount": 0,
  "hasVideo": false,
  "hasNews": false,
  "alternateLanguageLinks": 0
}
```

`lastmod` and similar fields are passed through exactly as the site publishes them. `null` means the sitemap doesn't include that field. You can download the results as JSON, CSV, Excel, XML or HTML, or get them through the API.

The run summary (`SUMMARY` in the key-value store) looks like this:

```json
{
  "inputs": 4, "sitemapsDiscovered": 297, "sitemapsFetched": 63, "sitemapsFailed": 0,
  "urlsFound": 73640, "urlsFiltered": 52434, "duplicates": 1206, "urlsSaved": 20000,
  "urlsPerInput": { "https://wordpress.org/": 12581, "https://developer.mozilla.org/": 7419, "https://www.python.org/": 0 },
  "inputsWithoutUrls": ["https://www.python.org/"],
  "durationSecs": 14.7, "errors": []
}
```

### Pricing

**Pay per result: from $1.00 per 1,000 URLs** ($0.001 per URL) on the Free plan, with lower prices on paid Apify plans:

| Apify plan | Price per 1,000 URLs |
|---|---|
| Free | $1.00 |
| Starter (Bronze) | $0.63 |
| Scale (Silver) | $0.45 |
| Business (Gold) and above | $0.23 |

Plus the standard Apify "Actor start" event ($0.00005 per run). **You only pay for URLs saved to your dataset.** Discovering and downloading sitemaps, filtered-out URLs, duplicates and errors are free. Platform usage (compute, storage) is included in the price. Set a **maximum cost per run** and the Actor stops cleanly when it reaches that limit. Example: 10,000 URLs on the Free plan cost about $10.00.

### Performance

In local tests, the default input took about 1 second of work (1,000 URLs from 2 sitemaps). A mixed run of 4 sites crawled 63 sitemaps, read 73,640 URLs and saved 20,000 in about 15 seconds at 256 MB memory.

### Related Actors

More no-login, pay-per-result tools from the same developer:

- [Bulk PageSpeed Insights & Lighthouse Audit](https://apify.com/aaronallday89/pagespeed-bulk-audit): Lighthouse scores and Core Web Vitals for a list of URLs.
- [Whole-Site PageSpeed Audit (Sitemap → Lighthouse)](https://apify.com/aaronallday89/whole-site-pagespeed-audit): enter a homepage and get PageSpeed audits of its pages plus a site summary.
- [SEO Meta Tags & Structured Data Auditor](https://apify.com/aaronallday89/seo-meta-structured-data-auditor): titles, descriptions, canonical, noindex, Open Graph and Schema.org rich-result checks per URL.
- [Wayback Machine Snapshot Finder](https://apify.com/aaronallday89/wayback-snapshot-finder): archived snapshots, first/last capture and every archived URL of a domain.
- [Domain Health Checker: WHOIS, DNS & SSL](https://apify.com/aaronallday89/domain-health-checker): domain age and expiry, DNS/SPF/DMARC, SSL certificate and HTTPS redirects.
- [ATS Jobs Scraper](https://apify.com/aaronallday89/ats-job-board-scraper): open jobs from Greenhouse, Lever, Ashby, Workable, SmartRecruiters, Recruitee and Personio.

#### Use together

1. Run **Sitemap URL Extractor** with `"includeRegex": "/blog/"` and `"maxUrls": 50`.
2. Pass the dataset's `url` column to **Bulk PageSpeed Insights & Lighthouse Audit** as `urls`, either by hand, with an Apify integration ("Run another Actor when this one succeeds"), or in code:

```js
const { defaultDatasetId } = await client.actor('aaronallday89/sitemap-url-extractor').call({ startUrls: [{ url: 'https://example.com' }], includeRegex: '/blog/', maxUrls: 50 });
const { items } = await client.dataset(defaultDatasetId).listItems({ fields: ['url'] });
await client.actor('aaronallday89/pagespeed-bulk-audit').call({ urls: items.map((i) => ({ url: i.url })), strategy: 'mobile' });
```

### FAQ

**The site has no sitemap. What happens?**
The run still succeeds with 0 URLs for that input. The input is listed in `SUMMARY.inputsWithoutUrls` and a warning appears in the log. This Actor only reads sitemaps and doesn't crawl links. If you need link crawling, use a crawler such as Website Content Crawler.

**Does it support gzipped sitemaps?**
Yes. Any gzip file is detected from its first bytes and decompressed automatically, whatever its name or headers say.

**Can I get URLs from subdomains or CDNs listed in the sitemap?**
Yes. Turn off `sameDomainOnly`.

**Why are there fewer URLs than in the sitemap?**
Duplicates (the same URL in several sitemaps) are removed, and your filters and `maxUrls` apply. `SUMMARY` shows `urlsFound`, `urlsFiltered` and `duplicates`.

**Does it use proxies or a browser?**
No. It makes plain HTTP requests with a polite user agent, which keeps it fast and inexpensive. Sitemaps are public files that sites publish so they can be read.

**How fresh is `lastmod`?**
It's exactly what the website publishes. Some CMSs update it on every build, so treat it as a hint.

**Is it legal?**
Sitemaps are published so machines can read them. You're still responsible for how you use the data. Respect each website's terms of service and applicable law.

### Feedback

Found a sitemap it can't read? Open an issue on the Actor's **Issues** tab with the URL and it will be fixed.

### More tools from the same developer

- [Bulk PageSpeed Insights Checker: Lighthouse & Core Web Vitals](https://apify.com/aaronallday89/pagespeed-bulk-audit): SEO agencies, web developers, site owners and performance teams
- [Whole-Site PageSpeed & Core Web Vitals Audit (via Sitemap)](https://apify.com/aaronallday89/whole-site-pagespeed-audit): SEO consultants, agencies and site owners
- [Greenhouse, Lever, Ashby & Workable Jobs Scraper (7 ATS)](https://apify.com/aaronallday89/ats-job-board-scraper): Job boards and aggregators, recruiters, sales teams prospecting on hiring signals, and market researchers
- [Bulk WHOIS, DNS & SSL Expiry Checker: Domain Health](https://apify.com/aaronallday89/domain-health-checker): IT admins, agencies managing client domains, email deliverability teams, domain investors and lead-enrichment workflows

# Actor input Schema

## `startUrls` (type: `array`):

Website URLs (e.g. https://example.com) to auto-discover sitemaps via robots.txt and common paths, or direct sitemap URLs (sitemap.xml, sitemap index, .xml.gz, .txt, RSS/Atom).

## `maxUrls` (type: `integer`):

Maximum number of URLs to output across all inputs. 0 = no limit (you are still protected by the run's maximum cost).

## `includeRegex` (type: `string`):

Only output URLs matching this regular expression (case-insensitive). Example: /blog/

## `excludeRegex` (type: `string`):

Skip URLs matching this regular expression (case-insensitive). Example: .(pdf|jpg|png)$|/tag/

## `sameDomainOnly` (type: `boolean`):

Drop URLs whose host differs from the input URL's host (www. is ignored).

## `discoverCommonPaths` (type: `boolean`):

If robots.txt lists no sitemaps, try /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz and /sitemap.txt.

## `maxSitemaps` (type: `integer`):

Safety cap on how many sitemap files (including index children) are fetched per run.

## `maxConcurrency` (type: `integer`):

How many sitemap files are downloaded in parallel.

## `requestTimeoutSecs` (type: `integer`):

Timeout per HTTP request. Failed requests (network errors, 429, 5xx) are retried up to 3 times with backoff.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxUrls": 1000,
  "sameDomainOnly": true,
  "discoverCommonPaths": true,
  "maxSitemaps": 500,
  "maxConcurrency": 10,
  "requestTimeoutSecs": 30
}
```

# Actor output Schema

## `urls` (type: `string`):

All extracted URLs with sitemap metadata (default dataset, 'URLs' view).

## `summary` (type: `string`):

JSON summary: sitemaps discovered/fetched/failed, URLs found/filtered/duplicates/saved, URLs per input, inputs without URLs, errors.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        }
    ],
    "maxUrls": 1000
};

// Run the Actor and wait for it to finish
const run = await client.actor("aaronallday89/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev" }],
    "maxUrls": 1000,
}

# Run the Actor and wait for it to finish
run = client.actor("aaronallday89/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxUrls": 1000
}' |
apify call aaronallday89/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,aaronallday89/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/GpLFTbS8b5gOZxze1/builds/D0yT9OFUquR72baZm/openapi.json