# Sitemap URL Extractor: All Pages from sitemap.xml (`itayozer9/sitemap-url-extractor`) Actor

Get every page URL of a website from its sitemaps: robots.txt discovery, sitemap.xml, index files, .gz and text sitemaps. Returns lastmod, changefreq, priority, images and news fields, with path and date filters and per-section counts.

- **URL**: https://apify.com/itayozer9/sitemap-url-extractor.md
- **Developed by:** [Itay Ozer](https://apify.com/itayozer9) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor: All Pages from sitemap.xml

Give it a website or a sitemap address. Get back every page URL the site lists in its sitemaps, one clean row per URL, with the last-modified date, change frequency and priority. Useful for SEO audits, content inventories, migration checks, change monitoring and as the URL list for a crawler or a RAG pipeline.

It finds the sitemaps for you: first the `Sitemap:` lines in the site's `robots.txt`, then the usual locations (`/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml`). Sitemap index files are followed down to every child sitemap, and gzip (`.xml.gz`) and plain-text sitemaps are read too.

It only reads files that websites publish for machines, over plain HTTP, and it obeys `robots.txt` for every file. No browser, no proxy, so it is fast (50,000 URLs in about 4 seconds in our tests) and cheap: **$0.30 per 1,000 URLs**, and duplicates are free.

### What you get

One row per unique page URL:

- `url`: the page address (`<loc>`), made absolute
- `lastmod`, `changefreq`, `priority`: exactly as the sitemap states them (`null` when the sitemap leaves them out)
- `section`: the first folder of the path, such as `/blog` or `/products`, so you can group and count pages
- `images`: image URLs from image sitemaps (up to 50 per page)
- `newsTitle`, `newsPublishedAt`: from Google News sitemaps
- `sitemapUrl`: the sitemap file the URL was found in
- `inputUrl`: the website or sitemap you sent

Plus a **SUMMARY** record (in the run's key-value store, linked on the Output tab) with, for each website: the sitemaps read, URLs found, returned, duplicates and filtered out, the URL count per section, and every sitemap that could not be read with the reason (missing, blocked by robots.txt, not a sitemap, ...).

### Input

| Field | Required | Example | Notes |
|---|---|---|---|
| `urls` | yes | `["https://www.sitemaps.org"]` | Websites or sitemap addresses, up to 1,000 per run. `example.com` without https:// works too |
| `maxUrls` | no | `10000` | Most URLs per website (or per sitemap you send). Default 10,000, up to 500,000 |
| `urlMustContain` | no | `["/blog/"]` | Keep only URLs containing one of these texts (not case sensitive) |
| `urlMustNotContain` | no | `["/tag/", "?page="]` | Drop URLs containing any of these texts |
| `lastmodFrom` | no | `"2026-10-01"` or `"7 days"` | Keep only URLs modified on or after this date. URLs with no lastmod are left out when set |

A website address (`https://example.com`) starts discovery from its robots.txt. An address that ends in `.xml`, `.xml.gz` or `.txt`, or has `sitemap` in its path, is read directly.

```json
{
    "urls": ["https://www.sitemaps.org", "https://developer.mozilla.org/sitemaps/ko/sitemap.xml.gz"],
    "maxUrls": 10000,
    "urlMustNotContain": ["/tag/"],
    "lastmodFrom": "30 days"
}
```

Tip for schedules: run it daily with `lastmodFrom` set to `"1 day"` to get only the pages a site added or changed since yesterday.

### Output example

A real row (from a run on nodejs.org):

```json
{
    "url": "https://nodejs.org/en/blog/release/v010.28",
    "lastmod": "2026-10-30T16:17:53.645Z",
    "changefreq": "always",
    "priority": null,
    "section": "/en",
    "images": [],
    "newsTitle": null,
    "newsPublishedAt": null,
    "sitemapUrl": "https://nodejs.org/sitemap.xml",
    "inputUrl": "https://nodejs.org"
}
```

And part of the SUMMARY record for docs.apify.com from the same run (lists shortened):

```json
{
    "input": "https://docs.apify.com",
    "sitemapsRead": ["https://docs.apify.com/sitemap.xml", "https://docs.apify.com/sitemap_base.xml", "..."],
    "urlsFound": 4153,
    "urlsReturned": 4145,
    "duplicates": 8,
    "filteredOut": 0,
    "sections": { "/": 16, "/academy": 201, "/api": 1920, "/legal": 19 },
    "stoppedAtLimit": false,
    "problems": []
}
```

### Pricing

Pay per event:

| Event | Price | When |
|---|---|---|
| URL extracted | $0.0003 | Every unique URL in your dataset |
| Actor start | $0.00005 | Once per run |

Example: a site with 12,000 URLs, run with `maxUrls` 20,000, costs 12,000 x $0.0003 + $0.00005 = $3.60005.

Free: a URL found in several sitemaps (charged once), URLs your filters drop, and sitemaps that cannot be read. The actor checks your run's maximum charge before it starts and stops at the number of URLs that charge covers, so a run never goes over the limit you set.

### Limits

- Reads sitemaps, not pages. A site with no sitemap (or one that blocks bots from it) returns no URLs, and the SUMMARY record says why. To discover pages by following links, use a crawler.
- Up to 2,000 sitemap files per run and 500,000 URLs per run. One sitemap file is read up to 60 MB after unzipping (the protocol's limit is 50 MB).
- RSS and Atom feeds listed as sitemaps are not read.
- Sites that refuse bots (HTTP 403 or a challenge page) are reported, not bypassed. There is no proxy rotation.
- `lastmod`, `changefreq` and `priority` are the site's own claims. Some sites set every lastmod to the day the sitemap was built, which makes the date filter less useful on those sites.
- Up to 4 sitemap files are read at a time.

### FAQ

**Does it respect robots.txt?**
Yes, always, and it cannot be turned off. It identifies itself as `SitemapUrlExtractorBot` with a link to this page, reads each site's robots.txt once per run, and skips any sitemap file the site disallows, following RFC 9309 (the internet standard for robots.txt).

**Why did I get fewer URLs than `maxUrls`?**
The site lists fewer pages, your filters dropped some, the same URL appeared in several sitemaps (it is returned once), or your run's maximum charge covered fewer URLs. The SUMMARY record has the counts for each site.

**Can I send several websites in one run?**
Yes, up to 1,000. `maxUrls` applies to each website separately, so one huge site cannot use up the run.

**Can I call it from an AI agent, MCP client or the API?**
Yes. The input is a list of addresses plus optional filters, every output field has a fixed type, and it runs with limited permissions, so it needs no access to your other Apify data.

**Is there personal data in the results?**
The rows are page addresses and dates that the website itself publishes for search engines. The actor adds nothing of its own. You are responsible for how you use the URLs of the sites you choose.

**Something broke or a sitemap was read badly. What do I do?**
Open an issue on the Issues tab with your run link and the sitemap address. Issues are answered within a few days.

# Actor input Schema

## `urls` (type: `array`):

One per line. A website (https://example.com) finds its sitemaps through robots.txt, then /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml. A sitemap address (ending in .xml, .xml.gz or .txt, or with "sitemap" in the path) is read directly. Sitemap index files are followed to every child sitemap. Up to 1,000 per run.

## `maxUrls` (type: `integer`):

Most page URLs to return for each website or sitemap you send (1 to 500,000; a run returns at most 500,000 in all). You pay only for URLs returned. A URL found in several sitemaps is returned and charged once.

## `urlMustContain` (type: `array`):

Optional. Keep only URLs that contain at least one of these texts (not case sensitive), for example /blog/ or /products/.

## `urlMustNotContain` (type: `array`):

Optional. Drop URLs that contain any of these texts (not case sensitive), for example /tag/ or ?page=.

## `lastmodFrom` (type: `string`):

Optional. Keep only URLs whose sitemap lastmod is on or after this date: 2026-10-01, or a number of days back such as 7 days. URLs without a lastmod are left out when this is set.

## Actor input object example

```json
{
  "urls": [
    "https://www.sitemaps.org",
    "https://developer.mozilla.org/sitemaps/ko/sitemap.xml.gz"
  ],
  "maxUrls": 10000,
  "urlMustContain": [
    "/blog/"
  ],
  "urlMustNotContain": [
    "/tag/"
  ],
  "lastmodFrom": "30 days"
}
```

# Actor output Schema

## `results` (type: `string`):

One item per unique page URL: lastmod, changefreq, priority, section, images, news fields and the sitemap it came from.

## `summary` (type: `string`):

Per website: sitemaps read, URLs found, returned, duplicates and filtered out, URL counts per section, and every sitemap that could not be read with the reason.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.sitemaps.org"
    ],
    "maxUrls": 10000
};

// Run the Actor and wait for it to finish
const run = await client.actor("itayozer9/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://www.sitemaps.org"],
    "maxUrls": 10000,
}

# Run the Actor and wait for it to finish
run = client.actor("itayozer9/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.sitemaps.org"
  ],
  "maxUrls": 10000
}' |
apify call itayozer9/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,itayozer9/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/sondxkQK5NZw2nnWy/builds/8bHbXfcOFpgRMuqVJ/openapi.json