# Sitemap URL Extractor (`comradecoding/sitemap-url-extractor`) Actor

Enter a domain, get every page URL from its sitemaps with lastmod dates. Follows nested and gzipped sitemaps. Pay per URL.

- **URL**: https://apify.com/comradecoding/sitemap-url-extractor.md
- **Developed by:** [Comrade Coding](https://apify.com/comradecoding) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.50 / 1,000 sitemap urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor

Give it a list of domains. For each one it finds the sitemaps, reads them, follows sitemap index files down to the nested sitemaps, decompresses gzipped files, and returns one row per page URL with the `lastmod` date the site reports.

Use it to get a full page list for an SEO audit, to find pages changed since a date, to seed your own crawler, or to compare two sites' structure.

### How it finds sitemaps

1. Reads the domain's `robots.txt` and takes every `Sitemap:` line.
2. If robots.txt lists none, tries `/sitemap.xml`, then `/sitemap_index.xml`.
3. Each file is read as a sitemap index (its child sitemaps are queued), a URL set (its URLs become results), or a plain-text sitemap (one URL per line).
4. Gzipped files are detected by their content, not their file name, so `.xml.gz` files work whether or not the server marks them as gzip.

You can also give a sitemap URL directly, and the actor reads that file and anything it points to.

### Input

| Field | Type | Default | What it does |
| --- | --- | --- | --- |
| `startUrls` | array of strings | required, no default | Domains (`example.com`), homepage URLs, or direct sitemap URLs. A URL with `sitemap` in the path, or ending in `.xml`, `.txt`, or `.gz`, is read as a sitemap. Any other path is ignored and the domain is used. |
| `maxItems` | integer | 1000 | Total URLs to return across all domains. This is your cost cap. |
| `maxItemsPerDomain` | integer | 0 | URLs to return per domain, so one big site cannot use the whole run. 0 means no per-domain limit. |
| `maxSitemaps` | integer | 500 | Sitemap files to read per domain, index files included. |
| `urlPattern` | string | empty | Regular expression, case-insensitive. Only matching URLs are returned. Example: `/blog/`. |
| `modifiedSince` | string | empty | Only URLs with a `lastmod` on or after this date. A date (`2026-01-31`) or an age (`7 days`, `2 weeks`, `6 months`). URLs with no `lastmod` are left out when this is set. |

Example input:

```json
{
    "startUrls": ["docs.python.org", "wiki.openstreetmap.org", "https://developer.mozilla.org/sitemap.xml"],
    "maxItems": 500,
    "maxItemsPerDomain": 200,
    "modifiedSince": "30 days"
}
```

### Output

One dataset item per unique page URL. A real item from a test run (this one came from a gzipped nested sitemap):

```json
{
    "url": "https://wiki.openstreetmap.org/wiki/Main_Page",
    "lastmod": "2026-10-21T14:26:22Z",
    "changefreq": null,
    "priority": null,
    "source_url": "https://wiki.openstreetmap.org/sitemap-wiki-0.xml.gz",
    "input": "wiki.openstreetmap.org",
    "domain": "wiki.openstreetmap.org",
    "fetched_at": "2026-10-28T06:13:52.483Z"
}
```

| Field | Meaning |
| --- | --- |
| `url` | Page URL as listed in the sitemap. |
| `lastmod` | The `<lastmod>` value exactly as the site wrote it, or `null`. |
| `changefreq` | The `<changefreq>` value in lower case, or `null`. |
| `priority` | The `<priority>` value as a number, or `null`. |
| `source_url` | The sitemap file the URL was found in. |
| `input` | The `startUrls` entry this result belongs to. |
| `domain` | Host of that entry. |
| `fetched_at` | When the sitemap file was read (ISO 8601, UTC). |

Two extra records go to the run's key-value store. They are not dataset items, so they are not billed:

- `SUMMARY`: per entry, where the sitemaps were found (`robots.txt`, `fallback`, or `input`), how many sitemap files were read, how many URLs they listed, how many were returned, and why it stopped (`done`, `maxItems`, `maxItemsPerDomain`, `maxSitemaps`, `no sitemap found`, `no readable sitemap`, `robots.txt unavailable`).
- `FAILED_URLS`: every entry or sitemap file that was skipped, with the reason. For example `Disallowed by robots.txt`, `HTTP 404`, `Got an HTML page, not a sitemap`, or `No sitemap found`.

### Behavior and limits

- Respects robots.txt for every file it reads, including nested sitemaps on other hosts. If a site's robots.txt returns a server error or cannot be reached, that site is skipped.
- Waits 300 to 800 ms between requests to the same host. Sites are read one at a time.
- A URL that appears in more than one sitemap is returned once per run.
- Sitemap indexes are followed up to 5 levels deep. Loops (an index that lists itself or a parent) are read once.
- Files over 60 MB, after decompression, are skipped. The sitemap protocol caps files at 50 MB.
- Broken files are handled per file: a cut-off file returns the complete entries before the break, entries without a valid absolute `<loc>` are skipped, and the run log says how many. One bad file never stops the run.
- Some wiki sites publish index entries with the host written twice (`https://host/https://host/sitemap-0.xml.gz`). These are repaired to the inner URL.
- `lastmod` is only as accurate as the site makes it. Some sites stamp every URL with the last build time.
- Image, video, news, and hreflang extensions inside a sitemap are not returned. Only the page `<loc>` is.
- RSS and Atom feeds used as sitemaps are not supported.
- No page HTML is fetched. If a site has no sitemap, it returns nothing for that site rather than crawling links.

### Pricing

Pay per result: each returned URL is one result. You set `maxItems` (and optionally `maxItemsPerDomain`), so you know the most a run can cost before it starts. Skipped files and the `SUMMARY` and `FAILED_URLS` records are free.

A single sitemap file can list up to 50,000 URLs, and the default `maxItems` of 1000 caps a run at about $0.50. Set `maxItems` lower for a quick look.

This actor is not affiliated with any of the sites named above. They are used as examples of public sitemaps.

### Use it from an AI agent

Add this actor as a tool through Apify's MCP server: `https://mcp.apify.com?tools=comradecoding/sitemap-url-extractor`. The agent gets one JSON row per page URL found in the site's sitemaps, with lastmod when the sitemap has it.

### Integrations

Anything that can call the Apify API can run this actor: the Apify API and client libraries, and Apify's integrations for Make, n8n, Zapier and LangChain. Results come back as a dataset you can download as JSON, CSV or Excel.

### What it costs

| Run size | Cost |
|---|---|
| 1,000 URLs | about $0.50 |
| 10,000 URLs | about $5 |
| 100,000 URLs | about $50 |

Platform usage is included in the price.

### FAQ

**Does it follow nested and gzipped sitemaps?**

Yes, sitemap index files and .xml.gz files are both read.

**What if a site has no sitemap?**

It checks robots.txt and the usual sitemap paths. If none exist, the domain is listed in the failures record and nothing is charged.

**Can I get the page content too?**

Pass the URLs to URL to Markdown Converter, which returns each page as clean Markdown.

### Related actors by ComradeCoding

- [URL to Markdown Converter](https://apify.com/comradecoding/url-to-markdown)
- [PDF to Text and JSON](https://apify.com/comradecoding/pdf-to-text)

# Actor input Schema

## `startUrls` (type: `array`):

Required. A domain (example.com) or homepage URL makes the actor look up sitemaps in robots.txt, then try /sitemap.xml and /sitemap_index.xml. A direct sitemap URL (anything with "sitemap" in the path, or ending in .xml, .txt, or .gz) is read as given. Problems are listed in the FAILED_URLS record.

## `maxItems` (type: `integer`):

Stop after this many page URLs in total. You pay per result, so this caps the cost of a run.

## `maxItemsPerDomain` (type: `integer`):

Stop each domain after this many URLs, so one large site cannot use up the whole run. 0 means no per-domain limit.

## `maxSitemaps` (type: `integer`):

Stop reading a domain after this many sitemap files, counting index files and nested ones. Protects against very large or looping sitemap trees.

## `urlPattern` (type: `string`):

Only return URLs that match this regular expression (case-insensitive). Leave empty to return all. Example: /blog/ to keep blog posts, or \[.]pdf$ to keep PDF files.

## `modifiedSince` (type: `string`):

Only return URLs whose lastmod is on or after this date. Use a date (2026-01-31) or an age (7 days, 2 weeks, 6 months). URLs without a lastmod are left out when this is set. Leave empty to return all.

## Actor input object example

```json
{
  "startUrls": [
    "docs.python.org",
    "https://developer.mozilla.org/sitemap.xml"
  ],
  "maxItems": 500,
  "maxItemsPerDomain": 100,
  "maxSitemaps": 50,
  "urlPattern": "/docs/",
  "modifiedSince": "30 days"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `failures` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "docs.python.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("comradecoding/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["docs.python.org"] }

# Run the Actor and wait for it to finish
run = client.actor("comradecoding/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "docs.python.org"
  ]
}' |
apify call comradecoding/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,comradecoding/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/1mmpemH9sowXF4612/builds/01TtP6dcjUoGkoSbO/openapi.json