# Sitemap URL Extractor – All Page URLs + Lastmod (`glidepath/sitemap-extractor`) Actor

Sitemap extractor: get every page URL of a site from robots.txt and sitemap.xml, with sitemap indexes and .gz files followed. Input: websites or sitemap URLs. Output: URLs + lastmod, changefreq, priority, HTTP status. $0.30/1k URLs.

- **URL**: https://apify.com/glidepath/sitemap-extractor.md
- **Developed by:** [Glidepath](https://apify.com/glidepath) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.09 / 1,000 sitemap urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor – All Page URLs + Lastmod

**Get every page URL of a website from its sitemaps,** with the last-modified date, change frequency and priority the site publishes, and optionally each URL's HTTP status. Give it a website (`https://example.com`) or a sitemap URL; it finds the sitemaps through robots.txt or `/sitemap.xml`, follows sitemap indexes and `.gz` files, and returns one clean, deduplicated row per page.

Plain HTTP, no browser, so even sites with tens of thousands of pages are read in seconds to minutes.

### Who uses it

- **SEO specialists:** audit what a site submits to search engines, find 404s and redirects listed in the sitemap, and compare sitemap coverage with your crawl.
- **Content and competitor research:** list every blog post, product or docs page of a competitor, and see what was added or updated recently with **Only pages modified on or after**.
- **Developers and data teams:** get the URL list to feed a scraper, a link checker, a migration or an AI/RAG pipeline, without crawling the whole site.
- **Monitoring:** schedule it daily with a date filter to get only new or updated pages.

### How to use it

1. Put one or more websites or sitemap URLs in **Websites or sitemap URLs** (up to 100).
2. Optionally narrow the output: **Only URLs matching** (e.g. `/blog/`), **Skip URLs matching**, **Only pages modified on or after**, and the URL limits. Turn on **Check each URL's HTTP status** to find broken links and redirects.
3. Run it and download the URLs as JSON, CSV or Excel, or pass them to another Actor or tool.

### Input example

```json
{
  "urls": ["https://docs.apify.com", "https://example.com/post-sitemap.xml"],
  "includePatterns": ["/blog/", "/academy/"],
  "excludePatterns": ["/tag/"],
  "lastmodAfter": "2026-09-01",
  "maxUrls": 5000,
  "maxUrlsPerSite": 2000,
  "checkStatus": false
}
```

How sitemaps are found for a website: first the `Sitemap:` lines of its robots.txt, otherwise `/sitemap.xml`, then `/sitemap_index.xml`. If a site keeps its sitemap somewhere else, enter that sitemap URL directly.

### Output example

```json
{
  "url": "https://docs.apify.com/academy",
  "site": "docs.apify.com",
  "sitemapUrl": "https://docs.apify.com/sitemap_base.xml",
  "lastmod": "2026-09-30T10:00:00+00:00",
  "changefreq": "weekly",
  "priority": 0.5,
  "imageCount": 0,
  "httpStatus": 200,
  "finalUrl": null,
  "statusNote": null,
  "extractedAt": "2026-10-07T19:30:00Z"
}
```

| Field | Type | Description |
|---|---|---|
| `url` | string | Page URL listed in the sitemap |
| `site` | string | Host of the website or sitemap you entered |
| `sitemapUrl` | string | The sitemap file that lists this URL |
| `lastmod` | string / null | Last modification date as published in the sitemap (W3C datetime) |
| `changefreq` | string / null | `always`, `hourly`, `daily`, `weekly`, `monthly`, `yearly` or `never` |
| `priority` | number / null | 0.0 to 1.0, as published |
| `imageCount` | integer | Images listed for this URL in an image sitemap |
| `httpStatus` | integer / null | HTTP status (only with **Check each URL's HTTP status**) |
| `finalUrl` | string / null | Where the URL redirects to (only with the status check; null when no redirect) |
| `statusNote` | string / null | Why a URL was not checked or the check failed |
| `extractedAt` | string | ISO 8601 UTC |

Sitemaps that could not be read are listed in the run's key-value store as `ERRORS` with the reason (`NO_SITEMAP`, `NOT_FOUND`, `NOT_A_SITEMAP`, `ROBOTS_DISALLOWED`, `ROBOTS_UNREACHABLE`, `BLOCKED`, `FETCH_FAILED`, `TOO_MANY_SITEMAPS`, `TOO_DEEP`, `REFUSED`), and a per-site summary as `SUMMARY` (sitemaps read, URLs found, URLs saved).

### Use it with

- **Schedules:** save the input as a Task with a **modified on or after** date and run it daily or weekly.
- **Other Actors:** pass the URLs to a scraper or to a change monitor as its start URLs.
- **Make, Zapier, n8n, Google Sheets, Slack:** through the Integrations tab.
- **API and code:** the Apify API, the JavaScript and Python clients, or CSV/JSON/Excel downloads.
- **AI agents:** the Apify MCP server exposes this Actor as a tool; the input is a list of websites.

### Pricing

Pay per URL: **$0.30 per 1,000 URLs** on the Apify Free plan; on paid plans **$0.27** (Starter), **$0.18** (Scale) and **$0.09** (Business and higher) (the price for your plan is shown on the Pricing tab). The examples below use the Free-plan price. Only saved URLs are charged; URLs removed by your filters, duplicates and sitemaps that fail are free. The status check is included in the price.

- A 1,000-page blog → about **$0.30**.
- A 20,000-page shop → about **$6.00**.
- 50 competitor sites, new pages only, checked weekly (about 2,000 URLs/month) → about **$0.60/month**.

A small start fee of $0.00005 per run applies. Set a **maximum cost per run** in the run options and the Actor stops cleanly when it is reached; **Max URLs per run** does the same by count.

### Limits and notes

- **Only what the site publishes in its sitemaps.** Pages that are not in any sitemap are not found (this Actor does not crawl links). Sitemaps follow the sitemaps.org protocol: XML `urlset` and `sitemapindex`, gzip, plain-text lists, RSS 2.0 and Atom.
- Up to 100,000 URLs per run, 500 sitemap files and 5 index levels per site, 50 MB per sitemap (the protocol's own limit).
- **robots.txt is respected** (RFC 9309, including `*` and `$` rules): sitemaps and status checks that it disallows are skipped. If robots.txt itself can't be read (server error), the site is skipped to be safe.
- **No bot-check bypass, no proxies, no logins.** A Cloudflare/DataDome-style challenge, a 401/403 or a persistent 429 on a sitemap is reported as `BLOCKED` and not charged. Private or internal addresses are refused.
- **Politeness:** at most two requests per second per website, five websites in parallel.
- `lastmod` is shown exactly as the site publishes it; some sites set it to the build time of the whole site, so it is only as good as the site's sitemap.

### FAQ

**Is it legal to extract sitemaps?** Sitemaps are files that websites publish specifically so that crawlers can find their pages, and this Actor reads only those files (plus, if you ask, the HTTP status of the listed pages) where robots.txt allows it, at a polite rate, without logging in or getting around any protection. It outputs URLs and dates, no personal data. What you do with the pages afterwards depends on each site's terms; if in doubt, ask a lawyer.

**Why did I get `NO_SITEMAP`?** The site lists no sitemap in robots.txt and has none at `/sitemap.xml` or `/sitemap_index.xml`. Look for it in the site's footer or in Google Search Console and enter the sitemap URL directly.

**Why are there fewer URLs than pages on the site?** The site's sitemaps don't list them all, or your filters or limits removed them. The `SUMMARY` record shows how many URLs each site's sitemaps contained and how many were saved.

**Can it find broken links?** Yes, for URLs in the sitemap: turn on **Check each URL's HTTP status** and filter the output for `httpStatus` 4xx/5xx or a non-empty `finalUrl` (redirects).

**Does it work on WordPress, Shopify, Wix and Yoast sitemaps?** Yes: they are standard sitemap indexes (often `sitemap_index.xml` or `sitemap.xml`) and are followed automatically.

### Changelog

See [CHANGELOG.md](CHANGELOG.md).

### Support

Found a problem or need a feature? Open an issue on the Actor's Issues tab. We read every one.

# Changelog

This Actor's version history is a separate document: https://apify.com/glidepath/sitemap-extractor/changelog.md

# Actor input Schema

## `urls` (type: `array`):

One per line. A website (https://example.com) is looked up through its robots.txt 'Sitemap:' lines, then /sitemap.xml and /sitemap_index.xml. A sitemap URL (…/sitemap.xml, …/post-sitemap.xml.gz, a sitemap index, a text, RSS or Atom sitemap) is read directly. Up to 100 per run.

## `maxUrls` (type: `integer`):

Stop after saving this many URLs in total (each saved URL is charged). Sitemaps are not read further once it is reached.

## `maxUrlsPerSite` (type: `integer`):

Cap for each website, so one large site doesn't use up the whole run.

## `includePatterns` (type: `array`):

Keep only URLs that match at least one of these regular expressions, e.g. '/blog/' or '/products/'. Up to 20. Filtered-out URLs are not charged.

## `excludePatterns` (type: `array`):

Drop URLs that match any of these regular expressions, e.g. '/tag/' or '?page='. Up to 20.

## `lastmodAfter` (type: `string`):

Keep only URLs whose sitemap 'lastmod' is on or after this date (YYYY-MM-DD, UTC). URLs without a lastmod are skipped when this is set.

## `checkStatus` (type: `boolean`):

Send a HEAD request to every saved URL (GET without the body if HEAD isn't supported) and add httpStatus and finalUrl, to find broken links and redirects in a sitemap. Slower: at most two requests per second per website. URLs that robots.txt disallows are listed but not checked.

## Actor input object example

```json
{
  "urls": [
    "https://docs.apify.com"
  ],
  "maxUrls": 1000,
  "maxUrlsPerSite": 0,
  "checkStatus": false
}
```

# Actor output Schema

## `urls` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `errors` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("glidepath/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://docs.apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("glidepath/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.apify.com"
  ]
}' |
apify call glidepath/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,glidepath/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/MWpEACz0yBhkR7nPJ/builds/EteiWBXRfqUQbUWA4/openapi.json