# Sitemap URL Extractor (`tidyrow/sitemap-url-extractor`) Actor

Extract every URL from any website's sitemaps. Finds sitemaps via robots.txt or common paths, follows sitemap indexes, reads gzip, RSS/Atom and text sitemaps, and returns lastmod, priority, hreflang, images and news. Optional HTTP status checks.

- **URL**: https://apify.com/tidyrow/sitemap-url-extractor.md
- **Developed by:** [Tidyrow Data](https://apify.com/tidyrow) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor

Returns every URL a website lists in its sitemaps, as a table you can export to CSV, Excel or JSON or read through the API.

Give it a homepage and it looks for the sitemaps itself. It reads the `Sitemap:` lines in robots.txt first, then tries the usual locations such as `/sitemap.xml`, `/sitemap_index.xml` and `/wp-sitemap.xml`. It follows nested sitemap indexes, unpacks `.xml.gz` files, and also reads RSS, Atom and plain-text sitemaps.

### How it behaves

A broken child sitemap doesn't fail the run. A 404, a timeout or a malformed XML file goes into the per-site `SUMMARY` record and the Actor moves on. When a file is damaged partway through, you still get the entries before and after the damage.

It uses plain HTTP requests and a streaming parser, with no browser, so runs are fast and the default memory is 512 MB. Large sitemap files are never loaded into memory whole.

URLs that appear in several sitemaps are saved once. URLs removed by your filters aren't saved, and you aren't charged for them.

Each row has `lastmod`, `changefreq` and `priority` when the sitemap provides them. You can also switch on hreflang alternates and image, video and Google News sitemap data.

Status checks are optional. When they're on, the Actor sends one light request per URL and records the HTTP status and any redirect target. It skips URLs that robots.txt disallows and keeps the number of parallel requests per site low.

### What people use it for

- SEO audits: pages missing from the sitemap, stale `lastmod` dates, redirects and broken URLs, hreflang coverage
- Seeding a crawl with a site's full URL list before running another Actor
- Content inventories and site migrations, with every page in one spreadsheet
- Change monitoring: schedule it with "Modified since" set to 7 days and you only get new or updated pages
- Sizing up a competitor's site and the sections it publishes

### Input

| Field | What it does |
|---|---|
| Websites or sitemap URLs | Homepages (`https://example.com`), subfolders (`https://example.com/blog/`), or direct sitemap or feed URLs. Bare domains work too. |
| Max URLs per site | Stops after this many saved URLs for one site. `0` means no limit. |
| Only URLs under the start path | For `https://example.com/blog/`, keeps only URLs under `/blog/`. On by default. |
| Include / Exclude patterns | Case-insensitive regular expressions, for example include `/products/` or exclude `\.pdf$`. |
| Modified since | Keeps only URLs with a `lastmod` on or after a date (`2024-01-31`) or within a period (`30 days`). |
| Extra fields | hreflang alternates and image, video and news sitemap data. |
| Check HTTP status | Adds `statusCode`, `finalUrl` and `statusError` to each row. |

Example input:

```json
{
    "startUrls": [{ "url": "https://crawlee.dev" }, { "url": "https://example.com/blog/" }],
    "maxUrlsPerSite": 10000,
    "excludeUrlPatterns": ["\\?", "/tag/"],
    "lastmodSince": "30 days"
}
```

### Output

One row per URL:

```json
{
    "url": "https://crawlee.dev/blog",
    "site": "https://crawlee.dev",
    "sitemapUrl": "https://crawlee.dev/sitemap.xml",
    "lastmod": "2024-04-01T03:00:00.000Z",
    "changefreq": "weekly",
    "priority": 0.5
}
```

With status checks on, each row also gets `statusCode` (such as `200` or `404`), `finalUrl` when the URL redirects, and `statusError` (such as `"timed out"` or `"blocked by robots.txt"`).

The `SUMMARY` record in the run's key-value store has one entry per site. It shows how the sitemaps were found (`robots.txt`, `common-path`, `input` or `none`), how many sitemap files were read, the counts of URLs found, saved, deduplicated and filtered out, and any sitemap errors.

### Pricing

You pay per result, with platform compute included. There's no monthly rental.

| Event | Price |
|---|---|
| URL extracted | $0.40 per 1,000 URLs |
| URL extracted with HTTP status check | $1.00 per 1,000 URLs |
| Actor start | $0.00005 per run |

If you set a maximum cost per run in the run options, the Actor stops cleanly when it reaches it.

### FAQ

**Is it OK to extract sitemaps?**
Sites publish sitemaps so that automated tools can find their pages, and this Actor reads only what's published there. Status checks skip URLs that robots.txt disallows. How you use the data is up to you.

**What if a site has no sitemap?**
The `SUMMARY` record marks it `none` and there are no URL charges for it, only the per-run start fee. If you know where the sitemap is, give the Actor that URL directly.

**Why did I get fewer URLs than I expected?**
Look at the `SUMMARY` record. It shows how many URLs were filtered out or skipped as duplicates, any sitemap errors, and whether "Max URLs per site" or "Only URLs under the start path" cut the run short.

**Does it render JavaScript or follow links?**
No, it reads sitemaps only, which is why it's quick and cheap. If you need to crawl the pages, pass its output to a crawler Actor.

# Actor input Schema

## `startUrls` (type: `array`):

Homepages (e.g. https://example.com) or direct sitemap/feed URLs. For a homepage, the Actor finds the sitemaps through robots.txt, then tries common paths like /sitemap.xml.

## `maxUrlsPerSite` (type: `integer`):

Stop after saving this many URLs for one site. 0 means no limit.

## `restrictToStartPath` (type: `boolean`):

If a start URL has a path (e.g. https://example.com/blog/), keep only URLs under that path. Has no effect for homepages or direct sitemap URLs.

## `includeUrlPatterns` (type: `array`):

Regular expressions (case-insensitive). If set, only URLs that match at least one are saved. Example: /blog/

## `excludeUrlPatterns` (type: `array`):

Regular expressions (case-insensitive). URLs that match any are skipped. Example: .(pdf|jpg)$

## `lastmodSince` (type: `string`):

Keep only URLs whose lastmod date is on or after this date. Absolute (2024-01-31) or relative (30 days). URLs without lastmod are skipped when this is set.

## `includeAlternates` (type: `boolean`):

Add the language/region alternates (\<xhtml:link rel="alternate" hreflang>) listed for each URL.

## `includeImages` (type: `boolean`):

Add image URLs, titles and captions from image sitemaps.

## `includeVideos` (type: `boolean`):

Add video titles, thumbnails and content URLs from video sitemaps.

## `includeNews` (type: `boolean`):

Add title, publication name, language and publication date from Google News sitemaps.

## `checkStatus` (type: `boolean`):

Send a lightweight request to every saved URL and record its status code and redirect target.

## `statusCheckConcurrency` (type: `integer`):

Parallel status checks per site. Keep it low to stay polite to the server.

## `respectRobotsTxt` (type: `boolean`):

Skip status checks for URLs that robots.txt disallows. They are saved with statusError "blocked by robots.txt".

## `useRobotsTxt` (type: `boolean`):

Read the Sitemap: lines in robots.txt.

## `tryCommonPaths` (type: `boolean`):

If robots.txt lists no sitemap, try /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and similar.

## `maxSitemapsPerSite` (type: `integer`):

Safety limit on how many sitemap files to read for one site.

## `maxDepth` (type: `integer`):

How many levels of nested sitemap indexes to follow.

## `maxConcurrency` (type: `integer`):

How many sites to process at the same time.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each HTTP request.

## `maxRetries` (type: `integer`):

Retries for network errors, HTTP 429 and 5xx responses.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxUrlsPerSite": 1000,
  "restrictToStartPath": true,
  "includeAlternates": false,
  "includeImages": false,
  "includeVideos": false,
  "includeNews": false,
  "checkStatus": false,
  "statusCheckConcurrency": 5,
  "respectRobotsTxt": true,
  "useRobotsTxt": true,
  "tryCommonPaths": true,
  "maxSitemapsPerSite": 5000,
  "maxDepth": 10,
  "maxConcurrency": 5,
  "requestTimeoutSecs": 30,
  "maxRetries": 3
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        }
    ],
    "maxUrlsPerSite": 1000
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidyrow/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev" }],
    "maxUrlsPerSite": 1000,
}

# Run the Actor and wait for it to finish
run = client.actor("tidyrow/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxUrlsPerSite": 1000
}' |
apify call tidyrow/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidyrow/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lFT6bn9FwySmEnQfq/builds/nG9LVxufdN7R51OPC/openapi.json