# Sitemap Extractor: All URLs from sitemap.xml, robots.txt & .gz (`makerio/sitemap-extractor`) Actor

Sitemap extractor and URL finder: get every page URL of a website with lastmod, changefreq and priority. Finds sitemaps via robots.txt and common locations, reads nested indexes, .gz, text sitemaps and RSS/Atom feeds, images, videos, news and hreflang. Status check and new-URL monitoring.

- **URL**: https://apify.com/makerio/sitemap-extractor.md
- **Developed by:** [Makerio](https://apify.com/makerio) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.10 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor

Get every URL from a website's sitemaps: give it a site or a sitemap URL and get one clean row per page with `lastmod`, `changefreq`, `priority`, images, videos, news and hreflang alternates. It finds the sitemaps for you, reads nested sitemap indexes, `.gz` files, plain-text sitemaps and RSS/Atom feeds, and keeps going when a sitemap is broken. **$0.10 per 1,000 URLs.**

### What you can use it for

- **Crawl seeding:** a complete list of a site's pages to feed into your own scraper or crawler.
- **SEO audits:** check what a site tells search engines: missing `lastmod`, URLs blocked by robots.txt, 404s and redirects in the sitemap, hreflang pairs.
- **Competitor monitoring:** schedule it with **Only new URLs** and get the pages a competitor published since the last run.
- **Content inventories and migrations:** every URL, with dates, before you move a site.
- **News and media:** article titles and publication dates from news sitemaps, image and video URLs from media sitemaps.

### How it works

1. **Finding the sitemaps.** For a website, the Actor reads `robots.txt` and follows its `Sitemap:` lines. If there are none, it tries the usual locations: `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz`, `/sitemap.txt` and a few more, and stops at the first one that works. If you give a sitemap URL that turns out to be wrong, it falls back to the same search.
2. **Reading them.** Sitemap indexes are followed to any depth (up to 10 levels), and a sitemap that links back to itself is read once. Files are streamed, so a 50 MB sitemap never has to fit in memory. Gzip is recognized by its content, not by the file name or headers, so `.gz` files served as plain text and plain files named `.gz` both work.
3. **Forgiving parsing.** Missing closing tags, namespace prefixes (`ns0:url`), upper-case tags, CDATA and stray characters do not stop the run. An HTML page served at `/sitemap.xml` (a "soft 404") is reported as "not a sitemap" instead of being read as one.
4. **One row per URL.** A URL listed in several sitemaps is saved once. When a URL is in both a sitemap and a feed, the sitemap's data wins.

Measured on a site with about a million URLs in 23 sitemap files: all of them read in 15 seconds with about 100 MB of memory.

### Input

| Field | What it does |
|---|---|
| **Websites or sitemap URLs** | A website, a sitemap or sitemap index (`.xml`, `.xml.gz`, `.txt`), a `robots.txt` file or an RSS/Atom feed. Bare domains such as `example.com` work too. |
| **Max URLs** | Stop after this many URLs in total. 0 means no limit. |
| **Only URLs matching / Skip URLs matching** | Patterns such as `*/blog/*` or `*.pdf`, or a regular expression in slashes. |
| **Only URLs modified since** | A date (`2026-10-01`) or a period (`7 days`, `24 hours`, `3 months`). Child sitemaps whose own `lastmod` is older are skipped without being downloaded, which makes large news sites fast. |
| **Only new URLs since the last run** | Save only URLs that were not in the sitemaps last time. The summary also counts and lists removed URLs. |
| **Check the HTTP status of every URL** | Adds the status code, redirect target and response time of each URL. |
| **Images, videos, news and hreflang** | On by default. Turn off for leaner rows. |

Example input:

```json
{
    "startUrls": [
        { "url": "https://www.example.com" },
        { "url": "https://shop.example.org/sitemap_products_1.xml" }
    ],
    "excludeUrlPatterns": ["*/tag/*"],
    "modifiedSince": "30 days"
}
```

### Output

One row per URL:

```json
{
    "url": "https://www.example.com/blog/how-we-ship",
    "lastmod": "2026-11-12T09:30:00+00:00",
    "changefreq": "weekly",
    "priority": 0.8,
    "title": null,
    "images": [
        { "url": "https://www.example.com/img/ship.jpg", "title": null, "caption": "Our warehouse" }
    ],
    "videos": [],
    "news": null,
    "alternates": [
        { "hreflang": "de", "url": "https://www.example.com/de/blog/how-we-ship" }
    ],
    "sitemapUrl": "https://www.example.com/post-sitemap.xml",
    "startUrl": "https://www.example.com",
    "blockedByRobotsTxt": false
}
```

- `lastmod` is exactly as the sitemap gives it. `title` is filled for feed items, and `news` holds the title, publication and date from news sitemaps.
- `blockedByRobotsTxt` is `true` when the site's robots.txt disallows the URL for Googlebot. A sitemap should not list such URLs.
- With **Check the HTTP status of every URL**, each row also gets `statusCode`, `finalStatusCode`, `finalUrl` (where redirects end), `redirects`, `responseTimeMs` and `statusError` (for example `TIMEOUT`, `DNS_FAILURE` or `BLOCKED_BY_ROBOTS_TXT`).

The dataset has views for **URLs**, **Status check**, **Images, videos, news** and **Hreflang**. Export to JSON, CSV, Excel or HTML, or get the URLs alone with the `fields=url` option of the dataset export.

The **OUTPUT** record in the key-value store has a summary per start URL:

- how the sitemaps were found (robots.txt, a common location or the start URL) and what robots.txt lists;
- every sitemap file with its HTTP status, type (`urlset`, `sitemapindex`, `rss`, `atom`, `text`), whether it was gzipped, its size, its URL count and any error (`HTTP_404`, `NOT_A_SITEMAP`, `NO_URLS`, `TIMEOUT`, `GZIP_ERROR`);
- URLs found, saved, duplicated, filtered out and blocked by robots.txt;
- status code counts, and new and removed URLs when monitoring.

A start URL without any sitemap does not fail the run: the summary says what was tried and why nothing was found.

### Pricing

| Event | Price |
|---|---|
| URL saved | $0.0001 ($0.10 per 1,000) |
| Status check, when turned on | $0.0002 per URL checked ($0.20 per 1,000) |

Finding and reading the sitemaps, the summary, and URLs that your filters skip are free. With **Only new URLs**, you pay only for the new ones.

| Run | URLs | Cost |
|---|---|---|
| A blog | 800 | $0.08 |
| An online store with status checks | 5,000 | $1.50 |
| A large site | 100,000 | $10 |
| A daily new-page check that finds 12 new URLs | 12 | $0.0012 |

You can set a maximum cost per run in the run options. The Actor stops at that limit and keeps what it has saved.

### Monitoring new pages

1. Turn on **Only new URLs since the last run** and save the input as a task.
2. Add a daily or weekly schedule.
3. Each run saves only the URLs that appeared since the run before. Send them to Slack, email, Google Sheets or a webhook with Apify integrations.

The first run saves every URL as a baseline. The lists of seen URLs are kept in the key-value store `sitemap-extractor-monitor` of your account, one record per start URL. Give runs different **Monitor name** values to keep separate lists for the same site.

### Questions

**The site has no sitemap.** The summary lists what was tried. This Actor only reads sitemaps and does not crawl pages. To find pages by following links, use a crawler.

**Why are there fewer URLs than the site has pages?** A sitemap lists what the site owner put in it, which is often not every page. Check `filteredOut` and `duplicates` in the summary too.

**Does it work with WordPress, Shopify, Wix and Squarespace?** Yes. Their sitemaps are found through robots.txt or the standard locations (`/wp-sitemap.xml`, Yoast and Rank Math `/sitemap_index.xml`, Shopify's `/sitemap.xml` with product, collection and blog sitemaps).

**Does it use proxies?** No. Sitemaps are public files meant for bots, so the Actor requests them directly with its own user agent and retries on `429` and `5xx` answers, honouring `Retry-After`. Sites that block all bots will show the error in the summary.

**Is a big sitemap slow or expensive?** Sitemaps are streamed and parsed as they download. You pay per URL saved; use **Max URLs**, patterns or **Only URLs modified since** to take only what you need.

**Can I use it from my code or an AI agent?** Yes, through the Apify API, the JavaScript and Python clients, or the Apify MCP server.

# Actor input Schema

## `startUrls` (type: `array`):

A website (https://www.example.com), a sitemap or sitemap index (https://www.example.com/sitemap.xml, .xml.gz, .txt), a robots.txt file or an RSS/Atom feed. For a website, the sitemaps are found through robots.txt and the usual locations (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and more).

## `maxUrls` (type: `integer`):

Stop after this many URLs in total. 0 means no limit. You pay only for URLs saved.

## `includeUrlPatterns` (type: `array`):

Keep only URLs that match one of these patterns. \* matches anything: */blog/*, \*.html. Wrap a regular expression in slashes: //products/\[a-z-]+$/

## `excludeUrlPatterns` (type: `array`):

Skip URLs that match one of these patterns, e.g. */tag/* or */page/*.

## `modifiedSince` (type: `string`):

Keep only URLs whose lastmod is on or after this date. A date such as 2026-10-01 or a period such as "7 days", "24 hours" or "3 months". Child sitemaps whose own lastmod in the index is older are skipped too, which makes large news sites fast.

## `keepUrlsWithoutLastmod` (type: `boolean`):

With "Only URLs modified since" set, also keep URLs that have no lastmod date at all.

## `onlyNewUrls` (type: `boolean`):

Remember the URLs of each start URL between runs and save only URLs that were not there last time. Schedule the Actor daily or weekly to track new pages on your or a competitor's site. The summary also counts removed URLs. The first run saves everything.

## `monitorName` (type: `string`):

Optional. Runs with different names keep separate lists of seen URLs for the same site.

## `checkStatus` (type: `boolean`):

Request each URL (HEAD, then GET if needed) and add its status code, redirect target and response time. Finds 404s, redirects and server errors listed in the sitemap. Charged as an extra event per URL checked.

## `maxConcurrency` (type: `integer`):

How many URLs are checked at the same time. Keep it low for small sites.

## `includeImagesVideosNews` (type: `boolean`):

Add the image, video and news sitemap data and the hreflang alternates of each URL to its row.

## `respectRobotsTxt` (type: `boolean`):

Do not request URLs that the site's robots.txt disallows. Sitemaps themselves are always read.

## `maxSitemapsPerSite` (type: `integer`):

Safety limit for very large sitemap indexes.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxUrls": 100,
  "keepUrlsWithoutLastmod": false,
  "onlyNewUrls": false,
  "checkStatus": false,
  "maxConcurrency": 5,
  "includeImagesVideosNews": true,
  "respectRobotsTxt": true,
  "maxSitemapsPerSite": 1000
}
```

# Actor output Schema

## `urls` (type: `string`):

One row per page URL found in the sitemaps: lastmod, changefreq, priority, images, videos, news, hreflang alternates, the sitemap it came from and, if enabled, its HTTP status.

## `summary` (type: `string`):

Per start URL: how the sitemaps were found, robots.txt, every sitemap file with its status, type and URL count, errors, duplicates, filters and new or removed URLs.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ],
    "maxUrls": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("makerio/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://apify.com" }],
    "maxUrls": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("makerio/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxUrls": 100
}' |
apify call makerio/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,makerio/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/B7mRPwoMe484fz0xv/builds/fa5BsKlcNeGgx15FQ/openapi.json
