# Sitemap URL Extractor: sitemap.xml Parser, robots.txt & .gz (`saif-tools/sitemap-url-extractor`) Actor

Every URL a site publishes in its sitemaps, in one clean table. Finds sitemaps via robots.txt, follows sitemap indexes and unzips .gz files; one JSON row per URL with lastmod. Filter by date to see new or updated pages. $0.30 per 1,000 URLs; sites without a sitemap aren't charged.

- **URL**: https://apify.com/saif-tools/sitemap-url-extractor.md
- **Developed by:** [saif alnuaimi](https://apify.com/saif-tools) (community)
- **Categories:** SEO tools, Developer tools, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**Every URL a site publishes, in one clean table, in seconds.** Give this sitemap extractor a domain or a sitemap URL and get back every page URL listed in the site's XML sitemaps: one JSON row per URL with `lastmod`, `changefreq` and `priority`. It finds the sitemaps for you, follows sitemap indexes and unzips `.gz` files. $0.30 per 1,000 URLs; sites without a sitemap cost nothing.

- Finds sitemaps from `robots.txt`, then tries `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml` and `/sitemap.xml.gz`
- Follows nested sitemap indexes, decompresses `.gz`, and reads plain-text sitemaps and RSS/Atom feeds
- **Change monitoring:** `lastmodAfter` returns only pages added or updated since a date
- Filters: include/exclude regex, same-domain only
- Polite: respects robots.txt, waits between requests to each host, retries 429/5xx with back-off
- Lightweight: downloads only the XML files, no browser

### Quick start

1. Click **Start**.
2. Add a domain (`https://htmx.org`) or a sitemap URL (`https://example.com/sitemap.xml`) to **Websites or sitemap URLs**.
3. Optional: set **Max URLs**, a regex filter or a "modified after" date.
4. Run, then export the dataset as JSON, CSV or Excel.

```json
{
  "startUrls": ["https://htmx.org"]
}
```

### Sample output

From our test run on htmx.org (188 URLs in 0.4 s, local test). First 5 rows:

| url | lastmodIso | path | sitemapUrl |
|---|---|---|---|
| https://htmx.org/ | null | / | https://htmx.org/sitemap.xml |
| https://htmx.org/CVE-2026-3682-1/ | null | /CVE-2026-3682-1/ | https://htmx.org/sitemap.xml |
| https://htmx.org/api/ | null | /api/ | https://htmx.org/sitemap.xml |
| https://htmx.org/attributes/ | null | /attributes/ | https://htmx.org/sitemap.xml |
| https://htmx.org/attributes/hx-boost/ | null | /attributes/hx-boost/ | https://htmx.org/sitemap.xml |

`lastmodIso` is null where the site's sitemap gives no `lastmod` for that page. The per-site report saved as `OUTPUT` in the key-value store:

```json
{
  "site": "https://htmx.org",
  "sitemapsFetched": 1,
  "sitemapsFailed": 0,
  "urlsFound": 188,
  "urlsKept": 188,
  "discoveredVia": "robots.txt",
  "errors": []
}
```

On a site with a sitemap index, `sitemapsFetched` counts every child sitemap followed (in our docs.apify.com test: found via robots.txt, index followed, 1,500 URLs in 1.3 s).

#### Only pages changed since a date

```json
{
  "startUrls": ["https://htmx.org"],
  "lastmodAfter": "2026-01-01",
  "dropUrlsWithoutLastmod": true
}
```

Result: 7 of 188 URLs kept (0.3 s, local test), sorted by date:

| url | lastmodIso |
|---|---|
| https://htmx.org/essays/markdown-in-src/ | 2026-09-21T00:00:00.000Z |
| https://htmx.org/essays/working-with-ai/ | 2026-06-29T00:00:00.000Z |
| https://htmx.org/essays/universities-and-ai/ | 2026-06-11T00:00:00.000Z |
| https://htmx.org/essays/code-is-cheap/ | 2026-06-04T00:00:00.000Z |
| https://htmx.org/essays/mcp-apps-hypermedia/ | 2026-03-18T00:00:00.000Z |
| https://htmx.org/essays/yes-and/ | 2026-02-27T00:00:00.000Z |
| https://htmx.org/essays/paris-2024-olympics-htmx-network-automation/ | 2026-01-16T00:00:00.000Z |

Without `dropUrlsWithoutLastmod`, URLs that have no `lastmod` are kept too (105 rows here), because the sitemap doesn't say when they changed.

#### Output fields

| Field | Meaning |
|---|---|
| `url`, `host`, `path` | The page URL and its parts |
| `lastmod` / `lastmodIso` | As written in the sitemap / normalised to ISO 8601 UTC |
| `changefreq`, `priority` | From the sitemap, if present |
| `imageCount`, `alternateCount` | Image entries and hreflang alternates |
| `sitemapUrl`, `sitemapDepth` | Which sitemap the URL came from, and how deep in the index |

### Pricing

| Event | Price (USD) |
|---|---|
| Actor start | $0.00005 per GB of memory (default 512 MB = $0.00005) |
| URL extracted (`url-extracted`) | $0.0003 ($0.30 per 1,000 URLs) |

You pay only for URL rows delivered. Sites with no sitemap, and sitemap URLs that return 404, are reported in `OUTPUT` and not charged. Set "Max cost per run" to cap spending.

### Use cases

- **Get all URLs of a website** (as listed in its sitemaps) for an SEO audit or content inventory
- **Change monitoring:** "what pages did this site add or update this month?" with `lastmodAfter`
- Migration checklists, and finding stale pages by `lastmod`
- **Feed a crawler, scraper or RAG pipeline** with a clean URL list
- **Give an AI agent a site map** before it browses

### Input options

| Field | Default | What it does |
|---|---|---|
| `startUrls` | (required) | Domains or sitemap URLs |
| `maxUrls` | 10000 | Total URL limit (0 = no limit; hard cap 1,000,000) |
| `maxUrlsPerSite` | 0 | Per-input limit (0 = none) |
| `maxSitemaps` | 500 | Safety limit on sitemap files per site |
| `includePatterns` / `excludePatterns` | \[] | Regexes matched against each URL |
| `lastmodAfter` | (none) | ISO date; URLs with an older `lastmod` are dropped |
| `dropUrlsWithoutLastmod` | false | With a date filter, also drop URLs without `lastmod` |
| `sameDomainOnly` | true | Ignore URLs and child sitemaps on other hosts |
| `respectRobotsTxt` | true | Skip sitemap files robots.txt disallows |
| `requestDelayMs` | 300 | Delay between requests to one host |

### Use with AI agents (MCP)

```
https://mcp.apify.com/?tools=saif-tools/sitemap-url-extractor
```

Prompt: *"List all blog URLs on example.com changed since 2026-06-01."* The agent passes `startUrls`, `includePatterns: ["/blog/"]` and `lastmodAfter`. Keep `maxUrls` small for agent calls.

### FAQ

#### How do I get all URLs of a website?

Enter the domain and run. The Actor finds the sitemaps (via robots.txt or the usual locations) and returns every URL they list. It doesn't crawl pages, so URLs missing from the sitemaps aren't included. To crawl pages and check their links, use the [Broken Link Checker](https://apify.com/saif-tools/broken-link-checker).

#### What if a site has no sitemap?

The run reports "No sitemap found" for that site in `OUTPUT`, and you aren't charged for it. In our tests, example.com had no sitemap and a sitemap URL that returned 404 was logged; neither was charged.

#### Can I export to CSV or Google Sheets?

Yes. Download the dataset as CSV, Excel or JSON from the Storage tab, or send it to Google Sheets with Apify's integrations.

#### How do I use it from n8n or Make?

Use the Apify node in n8n or the Apify app in Make (Zapier works too), choose "Run Actor", pick `saif-tools/sitemap-url-extractor`, then read the dataset items in the next step. Setup guides: https://docs.apify.com/platform/integrations

#### Can it handle huge sites?

It follows sitemap indexes up to `maxSitemaps` files per site and stops at `maxUrls`. Single sitemap files larger than 60 MB uncompressed are skipped (the protocol limit is 50 MB).

#### Does it use proxies or log in?

No. It reads public files only.

### SEO toolkit

Pair it with the [Broken Link Checker](https://apify.com/saif-tools/broken-link-checker): the sitemap list shows what you tell search engines exists; the link checker shows what's actually broken on your pages.

# Actor input Schema

## `startUrls` (type: `array`):

Domains (e.g. https://example.com) or direct sitemap URLs (e.g. https://example.com/sitemap.xml). For domains, sitemaps are discovered from robots.txt, then common locations (/sitemap.xml, /sitemap_index.xml).

## `maxUrls` (type: `integer`):

Stop after this many page URLs across all inputs. 0 = no limit (hard cap 1,000,000).

## `maxUrlsPerSite` (type: `integer`):

Per-input limit so one huge site does not use the whole budget. 0 = no per-site limit.

## `maxSitemaps` (type: `integer`):

Safety limit on how many sitemap files (including nested index children) are fetched per site.

## `includePatterns` (type: `array`):

Only keep URLs matching at least one of these regular expressions, e.g. /blog/ or .html$. Empty = keep all.

## `excludePatterns` (type: `array`):

Drop URLs matching any of these regular expressions.

## `lastmodAfter` (type: `string`):

ISO date like 2026-01-01. URLs without lastmod are kept unless 'Drop URLs without lastmod' is on.

## `dropUrlsWithoutLastmod` (type: `boolean`):

Only applies when a lastmod date filter is set.

## `sameDomainOnly` (type: `boolean`):

Ignore URLs (and child sitemaps) on other hosts than the input site.

## `respectRobotsTxt` (type: `boolean`):

Skip sitemap files whose path is disallowed for our user agent in robots.txt.

## `requestDelayMs` (type: `integer`):

Politeness delay between sitemap downloads on the same host.

## Actor input object example

```json
{
  "startUrls": [
    "https://htmx.org"
  ],
  "maxUrls": 10000,
  "maxUrlsPerSite": 0,
  "maxSitemaps": 500,
  "includePatterns": [],
  "excludePatterns": [],
  "dropUrlsWithoutLastmod": false,
  "sameDomainOnly": true,
  "respectRobotsTxt": true,
  "requestDelayMs": 300
}
```

# Actor output Schema

## `results` (type: `string`):

One row per URL found in the sitemaps, with lastmod, changefreq and priority.

## `summary` (type: `string`):

Per-site summary: sitemaps found, URL counts and any per-site errors.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://htmx.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("saif-tools/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://htmx.org"] }

# Run the Actor and wait for it to finish
run = client.actor("saif-tools/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://htmx.org"
  ]
}' |
apify call saif-tools/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,saif-tools/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/mfgmXwBPohNyk4axb/builds/YK3WMSnhPkwt15Djs/openapi.json