# Sitemap URL Extractor – All URLs from sitemap.xml & robots.txt (`openharvest/sitemap-url-extractor`) Actor

Extract page URLs from a website's sitemaps. Finds sitemaps via robots.txt, common paths and homepage links, then reads XML and .xml.gz sitemaps, sitemap indexes, text sitemaps, RSS and Atom feeds. Returns lastmod, changefreq, priority, images, videos and hreflang.

- **URL**: https://apify.com/openharvest/sitemap-url-extractor.md
- **Developed by:** [Enes Bulanık](https://apify.com/openharvest) (community)
- **Categories:** SEO tools, Developer tools, Lead generation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.30 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor – All URLs from sitemap.xml & robots.txt

Extract page URLs from a website's sitemaps. Finds sitemaps via robots.txt, common paths and homepage links, then reads XML and .xml.gz sitemaps, sitemap indexes, text sitemaps, RSS and Atom feeds. Returns lastmod, changefreq, priority, images, videos and hreflang.

Sitemaps are streamed, so 50 MB+ files work. Gzip is detected even when the server sends the wrong headers, broken XML is parsed as far as it goes, and every run saves a per-sitemap report of what was found and what failed.

### What you get

- **One row per URL** with the sitemap it came from, its domain and (optionally) lastmod, changefreq, priority, `image:image`, `video:video`, `news:news`, `xhtml:link` hreflang alternates, and feed titles and publish dates.
- **Sitemap discovery** – robots.txt `Sitemap:` lines, `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/news-sitemap.xml`, `/sitemap_news.xml`, `/sitemap.txt`, plus `<link rel="sitemap">` and RSS/Atom feeds from the homepage.
- **Formats and edge cases handled:**
  - gzip-compressed sitemaps served as `text/xml`, `application/octet-stream` or anything else (magic-byte detection), double-gzip, deflate
  - `http → https` and `www` redirects, relative `<loc>` links, BOM, CDATA, HTML entities
  - truncated or malformed XML (keeps every complete URL)
  - timeouts, 429 and 5xx errors (exponential-backoff retries, mid-stream resume without duplicates)
  - soft-404 pages that return HTML instead of XML (detected and skipped)
- **Filters** – include / exclude regex, same-domain only, de-duplication, global and per-site limits.
- **Speed** – plain HTTP and a streaming SAX parser, no browser. In our tests: about 500 URLs in 2 seconds, a 50 MB sitemap with 4,600 product URLs plus images and hreflang in about 20 seconds.

### Use cases

- **SEO audits** – list all indexable pages, find stale content by `lastmod`, check hreflang coverage and image sitemaps.
- **Crawling seed lists** – feed URLs into Website Content Crawler, Cheerio/Playwright scrapers or your own crawler instead of crawling blindly.
- **Content & competitor monitoring** – schedule daily runs and diff the URL list to spot new products, articles or landing pages.
- **Migrations & redirects** – export the full URL inventory before a site migration.
- **E-commerce intelligence** – collect all product URLs (and product images) from a store's sitemap.
- **AI / RAG pipelines** – get a clean list of documentation or blog URLs to ingest.

### How to use

1. Click **Try for free**.
2. Add one or more websites (`https://example.com` or just `example.com`) or direct sitemap/feed URLs to **Websites or sitemap URLs**.
3. Optionally set **Max URLs**, regex filters, or turn off metadata for a lean URL list.
4. Click **Start**. Download the results as CSV, Excel, JSON, XML or HTML, or fetch them via API.

### Input

| Field | Default | Description |
|---|---|---|
| `startUrls` | `https://apify.com` | Domains or direct sitemap / sitemap index / `.xml.gz` / `.txt` / RSS / Atom URLs. |
| `maxUrls` | `500` | Total URL limit (`0` = unlimited). |
| `maxUrlsPerSite` | `0` | Optional cap per start URL so one big site doesn't use the whole budget. |
| `followSitemapIndex` | `true` | Follow child sitemaps in sitemap indexes. |
| `includeRobotsTxtDiscovery` | `true` | Use `Sitemap:` lines from robots.txt. |
| `tryCommonPaths` | `true` | Probe common sitemap locations (missing ones are skipped silently). |
| `discoverFromHomepage` | `true` | If robots.txt has no sitemap, look for `<link rel="sitemap">` and RSS/Atom feeds on the homepage. |
| `urlIncludeRegex` / `urlExcludeRegex` | – | Case-insensitive regex filters on page URLs, e.g. `/blog/` or `/tag/|/author/`. |
| `sameDomainOnly` | `false` | Drop URLs on other domains (subdomains are kept). |
| `includeMeta` | `true` | lastmod, changefreq, priority, images, videos, news, alternates, title. |
| `dedupe` | `true` | Return each URL once (duplicates are never charged). |
| `saveSummary` | `true` | Save a per-sitemap report to the key-value store as `OUTPUT`. |
| `maxConcurrency`, `maxSitemaps`, `requestTimeoutSecs`, `maxRetries`, `proxyConfiguration` | | Advanced tuning. |

Example:

```json
{
    "startUrls": [{ "url": "https://www.ikea.com/sitemaps/prod-et-EE_1.xml" }, { "url": "wordpress.org" }],
    "maxUrls": 5000,
    "maxUrlsPerSite": 2500,
    "urlExcludeRegex": "/(tag|author)/"
}
```

### Output

One dataset row per URL. Real example from IKEA's product sitemap (lists shortened):

```json
{
    "url": "https://www.ikea.com/ee/et/p/yxstaby-vitriinkarp-maengupuldile-valge-70620011/",
    "sourceSitemap": "https://www.ikea.com/sitemaps/prod-et-EE_1.xml",
    "domain": "www.ikea.com",
    "lastmod": "2026-10-07T05:23:26.915Z",
    "changefreq": "weekly",
    "priority": 1,
    "images": [
        { "url": "https://www.ikea.com/ee/et/images/products/yxstaby-vitriinkarp-maengupuldile-valge__1505544_pe1008573_s5.jpg", "title": null, "caption": null }
    ],
    "alternates": [
        { "hreflang": "en-ee", "url": "https://www.ikea.com/ee/en/p/yxstaby-gaming-controller-display-box-white-70620011/" },
        { "hreflang": "de-de", "url": "https://www.ikea.com/de/de/p/yxstaby-schaukasten-fuer-gaming-controller-weiss-70620011/" }
    ],
    "videos": [],
    "news": null,
    "title": null,
    "sitemapType": "urlset"
}
```

With `includeMeta: false` you get a lean list:

```json
{ "url": "https://apify.com/", "sourceSitemap": "https://apify.com/sitemap/pages.xml", "domain": "apify.com" }
```

- `sitemapType` is `urlset`, `rss`, `atom` or `text`.
- `videos` items: `title`, `description`, `thumbnailUrl`, `contentUrl`, `playerUrl`, `duration`, `publicationDate`...
- `news`: `publicationName`, `language`, `publicationDate`, `title`, `keywords`.
- For RSS/Atom feeds, `lastmod` is the item's publish/update date and `title` its headline.

#### Run summary (key-value store `OUTPUT`)

```json
{
    "totalUrls": 500,
    "stopReason": "maxUrls (500) reached",
    "durationSecs": 1.7,
    "sitemapsProcessed": 9,
    "sitemapsFailed": 0,
    "duplicatesSkipped": 0,
    "httpRequests": 10,
    "sites": [{ "input": "https://apify.com", "robotsTxtSitemaps": ["https://apify.com/sitemap.xml"], "urls": 500 }],
    "sitemaps": [
        { "sitemapUrl": "https://apify.com/sitemap.xml", "discoveredVia": "robots.txt", "type": "sitemapindex", "urls": 0, "childSitemaps": 16, "status": "done" },
        { "sitemapUrl": "https://apify.com/sitemap/pages.xml", "discoveredVia": "index:https://apify.com/sitemap.xml", "type": "urlset", "urls": 500, "status": "stopped" }
    ]
}
```

### Pricing

**Pay per event: $0.0003 per URL = $0.30 per 1,000 URLs.** Platform usage is included.

- Only unique URLs saved to the dataset are charged – duplicates, filtered URLs and sitemap index entries are free.
- Set **Maximum cost per run** in the run options; the Actor stops cleanly at your budget.

Examples: 500 URLs = $0.15 · 10,000 URLs = $3 · 100,000 URLs = $30.

### Use with AI agents (MCP)

Add the Actor to the [Apify MCP server](https://mcp.apify.com) (`https://mcp.apify.com?tools=openharvest/sitemap-url-extractor`) or let agents find it with `search-actors` ("sitemap"). An agent can pass a bare domain; it does not need to know where the sitemap is.

Example `call-actor` input for "List the blog posts on example.com that were updated this year":

```json
{
    "startUrls": [{ "url": "example.com" }],
    "urlIncludeRegex": "/blog/",
    "maxUrls": 500
}
```

The agent reads `url` and `lastmod` and filters by date.

### FAQ

**The site has no sitemap. What happens?**
The Actor tries robots.txt, the common paths and the homepage's sitemap/RSS links. If none exist, the run ends with 0 URLs (and $0 charged) and the `OUTPUT` report shows exactly what was tried. For sites without any sitemap, use a crawler such as Website Content Crawler. (Some very large sites – e.g. Amazon, Reddit, Etsy, GitHub – deliberately do not publish their sitemaps publicly.)

**Can it handle huge sitemaps?**
Yes. Sitemaps are streamed and parsed on the fly (SAX), so 50 MB files and indexes with thousands of child sitemaps work with low memory. Use `maxSitemaps` and `maxUrls` to control the size of a run.

**Does it support `.xml.gz`?**
Yes – and also gzip files served with the wrong content type or without a `.gz` extension.

**Can I extract only blog posts / product pages?**
Use `urlIncludeRegex`, e.g. `/blog/` or `/products/`, and `urlExcludeRegex` to drop tag/author pages.

**Does it work with RSS and Atom feeds?**
Yes. Paste the feed URL directly, or let the Actor discover feeds from the homepage.

**Some sites block me – what can I do?**
Enable Apify Proxy in the input (residential proxies help for sites behind strict bot protection). Retries are automatic.

**Can I schedule it and get only new URLs?**
Schedule the Actor and compare datasets between runs (e.g. via the API, Google Sheets or Make/Zapier integrations).

### Is it legal?

Sitemaps, robots.txt and RSS feeds are files that website owners **publish specifically so machines can read them**. This Actor only reads those public files and does not log in or bypass any protection. You are responsible for how you use the URLs – respect each website's terms of service and robots.txt rules when you crawl the pages themselves.

### Related actors

- RSS Feed Reader & Feed Finder – RSS/Atom to JSON, Full Text
- Website SEO Audit *(coming soon)*
- Website Tech Stack *(coming soon)*

### Support

Questions, bugs or feature requests? Open an issue on the **Issues** tab with the website you tried.

# Actor input Schema

## `startUrls` (type: `array`):

Domains (https://example.com or example.com) or direct links to sitemap.xml, sitemap index, .xml.gz, .txt sitemaps, RSS or Atom feeds.

## `maxUrls` (type: `integer`):

Maximum number of URLs to return in total (0 = unlimited). You only pay for URLs actually returned.

## `maxUrlsPerSite` (type: `integer`):

Optional cap per start URL so one huge website does not use the whole Max URLs budget when you process several sites (0 = no per-site cap).

## `followSitemapIndex` (type: `boolean`):

Open child sitemaps listed in sitemap index files (recommended).

## `includeRobotsTxtDiscovery` (type: `boolean`):

Read the site's robots.txt and use every Sitemap: line.

## `tryCommonPaths` (type: `boolean`):

Probe /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml, /sitemap.xml.gz, /news-sitemap.xml, /sitemap_news.xml and /sitemap.txt. Missing ones are skipped silently.

## `discoverFromHomepage` (type: `boolean`):

If robots.txt lists no sitemap, look for <link rel="sitemap"> and RSS/Atom feed links on the homepage.

## `urlIncludeRegex` (type: `string`):

Only return URLs matching this regular expression (case-insensitive). Example: /blog/ or .html$

## `urlExcludeRegex` (type: `string`):

Skip URLs matching this regular expression (case-insensitive). Example: /tag/|/author/

## `sameDomainOnly` (type: `boolean`):

Drop URLs whose host differs from the start website (subdomains are kept).

## `includeMeta` (type: `boolean`):

Add lastmod, changefreq, priority, image/video/news extensions, hreflang alternates and feed titles. Turn off for a lean URL-only list.

## `dedupe` (type: `boolean`):

Return each URL only once even if it appears in several sitemaps (duplicates are never charged).

## `saveSummary` (type: `boolean`):

Store a summary with URL counts per sitemap, discovery source and errors in the key-value store under OUTPUT.

## `maxConcurrency` (type: `integer`):

How many sitemaps are downloaded in parallel.

## `maxSitemaps` (type: `integer`):

Safety cap on the number of sitemap files opened for each start URL (huge sites can have thousands of child sitemaps).

## `requestTimeoutSecs` (type: `integer`):

Timeout for the first byte and for idle connections. Large sitemaps keep streaming as long as data flows.

## `maxRetries` (type: `integer`):

Retries for network errors, timeouts, 429 and 5xx responses (exponential backoff).

## `proxyConfiguration` (type: `object`):

Most sitemaps are public and need no proxy. Enable Apify Proxy for sites that block data-center IPs.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxUrls": 500,
  "maxUrlsPerSite": 0,
  "followSitemapIndex": true,
  "includeRobotsTxtDiscovery": true,
  "tryCommonPaths": true,
  "discoverFromHomepage": true,
  "sameDomainOnly": false,
  "includeMeta": true,
  "dedupe": true,
  "saveSummary": true,
  "maxConcurrency": 8,
  "maxSitemaps": 10000,
  "requestTimeoutSecs": 30,
  "maxRetries": 3,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All scraped items in the default dataset (JSON, CSV, Excel and more via the dataset export).

## `files` (type: `string`):

Records saved in the default key-value store, e.g. the run summary.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ],
    "maxUrls": 500,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("openharvest/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com" }],
    "maxUrls": 500,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("openharvest/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxUrls": 500,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call openharvest/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,openharvest/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/VV3IR5f3tnqkohgYo/builds/p5zhpdHJd08jAiO8U/openapi.json