# Sitemap URL Extractor (`evolve-data/sitemap-url-extractor`) Actor

Extract every URL from any website's sitemaps: auto-discovers sitemaps, follows indexes, supports .gz, text, RSS and Atom. Filter by pattern or lastmod date, and optionally check each URL's HTTP status.

- **URL**: https://apify.com/evolve-data/sitemap-url-extractor.md
- **Developed by:** [Ahmed Zaky](https://apify.com/evolve-data) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor

Get every URL a website publishes in its sitemaps — as a clean table you can download as CSV, JSON or Excel, or feed straight into another Actor or workflow.

Paste a website address and the Actor finds the sitemaps for you. No need to know where they live.

### What it does

- **Finds sitemaps automatically** from `robots.txt` and the usual locations (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, …).
- **Follows sitemap indexes** to any depth, so large sites with hundreds of nested sitemaps work.
- **Reads every format:** XML sitemaps, gzipped `.xml.gz`, plain-text sitemaps, RSS and Atom feeds.
- **Keeps the metadata:** `lastmod`, `changefreq`, `priority`, image URLs, `hreflang` alternates and Google News titles.
- **Filters as it goes:** include/exclude URL patterns and a *Modified since* date. Schedule it daily to get only new or updated pages.
- **Optional status check:** records the HTTP status code and redirect target of each URL, so you can find broken (404) and redirected pages listed in your sitemap.
- **Never loses a whole run to one bad file.** Unreadable sitemaps are skipped and listed in the `SUMMARY` record, and everything else is saved.

### Use cases

- **SEO audits:** find 404s, redirects and orphaned URLs in your sitemap.
- **Content monitoring:** track new pages, products or articles on competitor sites with a daily schedule and *Modified since*.
- **Crawl planning:** get a clean list of URLs to feed into a content or product scraper, instead of crawling blindly.
- **AI / RAG pipelines:** enumerate every docs or blog page before indexing.
- **Site migrations:** snapshot every URL before and after a move.

### Input

| Field | Description |
|---|---|
| Start URLs | Websites or direct sitemap/feed URLs |
| Include / Exclude URL patterns | Regular expressions, e.g. `/blog/`, `/products/`, `/tag/` |
| Modified since | Keep only URLs with `lastmod` on or after this date |
| Check HTTP status | Adds `statusCode` and `finalUrl` for every URL |
| Max URLs / Max URLs per website | Cost and size limits |

### Output

```json
{
  "url": "https://example.com/blog/hello-world",
  "lastmod": "2026-09-21T10:00:00Z",
  "changefreq": "weekly",
  "priority": 0.8,
  "statusCode": 200,
  "sitemapUrl": "https://example.com/post-sitemap.xml",
  "site": "https://example.com"
}
```

A `SUMMARY` record in the key-value store lists the sitemaps found per website and any that could not be read.

### Pricing

Pay only for results: you are charged per URL saved, plus per URL when the optional status check is on. Set *Max URLs* or a maximum cost per run to stay in budget. The Actor stops cleanly at your limit and keeps everything collected so far.

### FAQ

**The site has no sitemap.** The run tells you so (it checks `robots.txt` and common locations). If you know where the sitemap lives, paste its URL directly.

**Some URLs have no `lastmod`.** The site didn't publish one. With *Modified since* set, those URLs are skipped because their age can't be confirmed.

**Is this legal?** Sitemaps are published by site owners specifically so automated tools can read them.

Found a problem or need a feature? Open an issue on the Issues tab. Issues are answered within 24 hours.

# Actor input Schema

## `startUrls` (type: `array`):

Websites (sitemaps are found automatically via robots.txt and common locations) or direct sitemap URLs. Sitemap indexes, .xml.gz, plain-text sitemaps, RSS and Atom feeds are all supported.

## `includePatterns` (type: `array`):

Only keep URLs matching at least one of these regular expressions, e.g. /blog/ or /products/.

## `excludePatterns` (type: `array`):

Drop URLs matching any of these regular expressions, e.g. /tag/ or ?page=.

## `modifiedSince` (type: `string`):

Only keep URLs whose <lastmod> is on or after this date (YYYY-MM-DD). Great for monitoring new or updated pages on a schedule. URLs without a lastmod are skipped when this is set.

## `checkStatus` (type: `boolean`):

Request every URL and record its status code and redirect target. Finds broken (404) and redirected pages listed in the sitemap. Charged per checked URL.

## `maxUrls` (type: `integer`):

Stop after saving this many URLs. 0 = no limit.

## `maxUrlsPerSite` (type: `integer`):

Stop collecting from a website after this many URLs, then move on to the next one. 0 = no limit.

## `maxSitemaps` (type: `integer`):

Safety limit on how many sitemap files to read in total.

## `proxyConfiguration` (type: `object`):

Usually not needed. Enable if a site blocks datacenter requests.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "includePatterns": [],
  "excludePatterns": [],
  "checkStatus": false,
  "maxUrls": 100,
  "maxUrlsPerSite": 0,
  "maxSitemaps": 1000,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `urls` (type: `string`):

Every URL found in the sitemaps, with lastmod, changefreq, priority and (optionally) HTTP status.

## `summary` (type: `string`):

Sitemaps found per website and any that could not be read.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ],
    "maxUrls": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("evolve-data/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com" }],
    "maxUrls": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("evolve-data/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxUrls": 100
}' |
apify call evolve-data/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,evolve-data/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/C7fFbPFta5VvCfTQ9/builds/I4wVkTvPoedUsbB0S/openapi.json