# Sitemap URL Extractor: All Website URLs from Sitemaps (`pavel.tashev/sitemap-url-extractor`) Actor

Sitemap URL extractor: get all URLs of a website from its XML sitemaps, with status code checking and change mode reporting new, removed and updated pages since the last run. Never charges for a failed or empty result.

- **URL**: https://apify.com/pavel.tashev/sitemap-url-extractor.md
- **Developed by:** [Pavel Tashev](https://apify.com/pavel.tashev) (community)
- **Categories:** SEO tools, Developer tools, MCP servers
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Sitemap URL Extractor do?

**Sitemap URL Extractor gets every URL a website lists in its sitemaps, with last-modified date, change frequency, priority, images, videos, news data and language versions (hreflang).** Enter a website such as `apify.com` and it finds the sitemaps automatically, through robots.txt and the usual locations (WordPress, Yoast, `.xml.gz`, text sitemaps). You can also paste a direct sitemap, sitemap index or RSS/Atom feed URL.

It is built to **finish reliably**: broken XML is repaired where possible, a website without a sitemap never fails the whole run, and every problem is explained in plain language. Because it runs on the Apify platform, you get API access, scheduling, webhooks, integrations (Make, Zapier, n8n, Google Sheets) and run monitoring.

### You are never charged for a failed or empty result

Every result says what it is, and only real results cost anything.

| `status` | Meaning | Charged |
|---|---|---|
| `ok` | A real result | Yes |
| `empty` | Worked, but there was nothing to report | No |
| `failed` | Could not be done — the `error` field says why | No |
| `truncated` / `partial` | Cut short by a limit you set; what was delivered is kept | Only the delivered part |
| `skipped` | Not attempted, for example after the maximum cost per run was reached | No |

If every input fails, the run itself fails with an explanation instead of reporting success with no data.

### Why use Sitemap URL Extractor?

- **SEO audits and content inventories:** list every page, see what changed recently, and spot old content.
- **Crawl planning:** feed an exact URL list into your scraper or RAG pipeline instead of crawling the whole site.
- **Competitor monitoring:** schedule a daily run in *changes since last run* mode to get only new, updated and removed products, posts, job pages or landing pages.
- **Migration checks:** export all URLs before and after a redesign and compare them.
- **Broken link checks:** turn on status checks to get the HTTP status and final URL after redirects.

### How to extract all URLs from a website's sitemap

1. Click **Try for free**.
2. In **Websites or sitemap URLs**, enter one or more websites (for example `apify.com`) or sitemap links.
3. Optionally set **Maximum URLs per website**, **Include/Exclude URLs matching** or **Modified after**.
4. Click **Start**. Most sites finish in seconds.
5. Open the **Output** tab. **Overview** shows URL, last modified, change frequency and priority. **Details** adds images, language versions and the sitemap each URL came from.

### Input

| Field | Description | Default |
|---|---|---|
| **Websites or sitemap URLs** | Websites, domains, or direct links to a sitemap, sitemap index, `.xml.gz`, `.txt` sitemap or RSS/Atom feed. Up to 50 per run. | – |
| **Maximum URLs per website** | Stop collecting from a website after this many URLs (up to 200,000) | 10,000 |
| **Report changes since last run** | Off, *New and modified pages*, or *New, modified and removed pages* | off |
| **First run in change mode** | Return all URLs as `initial`, or save the baseline only (free) | return all |
| **Removal safety threshold (%)** | Don't report removals if a website suddenly lists fewer URLs than this share of last run | 50 |
| **Include URLs matching** | Glob (`*/blog/*`), plain text (`/products/`) or regular expression between slashes (`/\/2026\//`) | all |
| **Exclude URLs matching** | Same syntax, for example `*/tag/*` | none |
| **Modified after** | Keep only URLs with a sitemap lastmod on or after this date | any |
| **Only URLs on the same website** | Drop links to other domains (subdomains are kept) | on |
| **Remove duplicate URLs** | Save each URL once across all sitemaps | on |
| **Include images, videos and news data** | Add sitemap extension data | on |
| **Check HTTP status of each URL** | Request each URL and save status code and final URL | off |
| **Parallel status checks** | How many URLs are checked at once | 5 |
| **Maximum sitemap index depth** | Levels of nested sitemap indexes to follow | 5 |

Example input:

```json
{
  "startUrls": ["apify.com", "https://wordpress.org/news/"],
  "maxUrls": 1000,
  "excludePatterns": ["*/tag/*"],
  "lastmodAfter": "2026-01-01"
}
```

### Output

Each URL is one item in the dataset. Real examples from a run on 15 September 2026:

```json
{
  "url": "https://wordpress.org/embed-image/",
  "site": "https://wordpress.org",
  "sitemapUrl": "https://wordpress.org/image-sitemap-1.xml",
  "lastmod": "2022-10-04T14:46:18+00:00",
  "changefreq": null,
  "priority": null,
  "alternates": [],
  "images": [
    { "url": "https://wordpress.org/files/2022/08/embed-image.png", "title": null, "caption": null }
  ],
  "videos": [],
  "news": null,
  "error": null,
  "scrapedAt": "2026-09-15T11:20:24.820134+00:00"
}
```

With status checks on, each item also has `"statusCode": 200`, `"finalUrl"` and `"statusError"`.

A website whose sitemap can't be read gets one item with an `error` explaining why (for example *"No sitemap found in robots.txt or at the usual locations"* or *"HTTP 403 Forbidden: the site refused automated access"*) and `suggestions`. These items are never charged. A per-website summary of every sitemap found, how many URLs each had and any errors is saved in the key-value store as `SITEMAPS_REPORT`.

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

### Monitor a website for new and removed pages

Turn on **Report changes since last run** and put the Actor on a **schedule** (for example daily) to watch a website without paying for the same URLs again and again:

1. Enter the websites to watch, for example a competitor's shop, blog or careers site. Use *Include URLs matching* (`*/products/*`, `*/jobs/*`) to watch only one section.
2. Choose **New and modified pages** or **New, modified and removed pages**.
3. Choose what the first run returns: every URL marked `initial`, or nothing (**Save baseline only**, free).
4. Save the configuration as a task and add a schedule. Connect a webhook, Slack, email or Google Sheets integration to be notified when a run finds changes.

Every item then has a `changeType`:

| `changeType` | Meaning |
|---|---|
| `new` | The URL was not in the sitemap last time |
| `modified` | The URL's `lastmod` date changed (`previousLastmod` holds the old date) |
| `removed` | The URL was in the sitemap last time but is gone now (only in *New, modified and removed pages* mode) |
| `initial` | First run for this website, returned when *Return all URLs* is selected |

Example `new` item:

```json
{
  "url": "https://example.com/products/new-running-shoe",
  "site": "https://example.com",
  "sitemapUrl": "https://example.com/sitemap-products.xml",
  "lastmod": "2026-09-15T00:00:00+00:00",
  "changeType": "new",
  "previousLastmod": null,
  "scrapedAt": "2026-09-16T06:00:04+00:00"
}
```

**How it works.** After each run, the list of URLs for each website (URL and lastmod only) is saved in a key-value store named **`sitemap-url-extractor-state`** in your own Apify account, and the next run compares against it. One list is kept per start URL plus include/exclude patterns and the same-website setting, so changing those starts a new baseline. *Modified after* only filters what is returned and does not reset the list. To start over, delete the store in **Storage → Key-value stores**.

**Built-in safeguards against false alarms:**

- If a website lists far fewer URLs than last time (below the **Removal safety threshold**), the run assumes a sitemap failed to load: nothing is reported as removed and the saved list is kept.
- If *Maximum URLs per website* is reached, a website's sitemaps hit one of the Actor's own ceilings (500 sitemap files or 512 MB per website), the run runs out of time, or some sitemaps can't be read, new and modified pages are still reported, but removals are not, and the saved list is only extended.
- If the run stops at your maximum cost, the saved list is not updated, so no change is lost; it is reported on the next run.
- `SITEMAPS_REPORT` shows per website: previous and current URL counts, new, modified and removed counts, whether this was the baseline, and a note explaining any safeguard that was applied.

**You pay only for changes:** in change mode only the URLs returned (new, modified, removed, or initial on a first run) are charged. A scheduled run on a website that didn't change returns nothing and costs only the small Actor start fee.

### Output fields

| Field | Meaning |
|---|---|
| `url` | Page URL from the sitemap |
| `site` | Website it belongs to |
| `sitemapUrl` | Sitemap the URL was found in |
| `lastmod` | Last modification date (ISO 8601, UTC), when the sitemap has it |
| `changefreq`, `priority` | Crawl hints from the sitemap |
| `alternates` | Language versions: `[{ "hreflang": "de", "url": "..." }]` |
| `images`, `videos`, `news` | Image, video and Google News sitemap data |
| `statusCode`, `finalUrl`, `statusError` | HTTP check results (only when status checks are on; removed URLs are not checked) |
| `changeType`, `previousLastmod` | `new`, `modified`, `removed` or `initial` in change mode, otherwise `null` |
| `error`, `suggestions` | Set only on the item for a website whose sitemap couldn't be read |

### How much does it cost to extract sitemap URLs?

This Actor uses **pay-per-event pricing**:

- **URL extracted:** charged for each URL saved to your dataset. Duplicates, filtered URLs and websites without a readable sitemap cost nothing. In change mode, only the changed URLs returned are charged.
- **URL status checked:** charged only when *Check HTTP status* is on, for each URL actually checked.

See the **Pricing** tab for current prices per 1,000 URLs. Platform compute is included. You can set a maximum cost per run and the Actor stops cleanly when it is reached.

### Tips

- **Big sites:** set *Maximum URLs per website* and use include patterns (for example `*/products/*`) to get only what you need.
- **Only new content:** use *Report changes since last run* with a daily schedule. Unlike *Modified after*, it also catches new pages that have no lastmod date.
- **Status checks are slower:** they request every URL. Keep *Parallel status checks* low for small websites.
- **No sitemap found?** Look for a sitemap link in the site's robots.txt or footer and enter it directly, or use the site's RSS feed.

### Related tools

Audit the URLs you extracted:

- [Broken Link Checker](https://apify.com/pavel.tashev/broken-link-checker): crawl a website and find broken links, images and redirect chains.
- [SEO Audit & Metadata Checker](https://apify.com/pavel.tashev/seo-metadata-audit): check titles, meta descriptions, canonicals and other on-page SEO for pages or whole sites.
- [Website Tech Stack Detector](https://apify.com/pavel.tashev/website-tech-stack-detector): detect the CMS, frameworks, analytics and hosting of any website.

### FAQ and support

**Does it respect robots.txt?** Yes. Sitemaps disallowed for crawlers in robots.txt are skipped and reported.

**What happens when a site won't let us read its robots.txt?** Some sites put their `/robots.txt` behind bot protection and answer 403 or time out. We then can't tell what the site allows, so we follow one rule everywhere: **the address you entered is still read, and addresses we would have worked out ourselves are not.** A site with no robots.txt at all (404) is treated as allowing everything. For this Actor that means a sitemap URL you paste is always read, while the usual locations we try for you (`/sitemap.xml`, `/sitemap_index.xml`, …) are not tried at such a site.

**Why did a site fail with HTTP 403?** Some websites block automated requests from cloud servers. The run still finishes for your other websites, and the reason is in the `error` field and `SITEMAPS_REPORT`.

**Does it crawl pages that are not in the sitemap?** No. It reads sitemaps only, which is fast and light on the website. Pages missing from the sitemap won't appear.

**Missing a format or a field?** Open an issue in the **Issues** tab. Custom extraction pipelines are available on request.

Only extract and use data from websites in line with their terms of use.

# Actor input Schema

## `startUrls` (type: `array`):

Websites or domains (for example apify.com; sitemaps are found automatically through robots.txt and the usual locations) or direct links to a sitemap, sitemap index, .xml.gz, .txt sitemap or RSS/Atom feed. Up to 50 websites per run; extra entries are ignored.

## `maxUrls` (type: `integer`):

Stop collecting from a website after this many URLs (after filters and duplicates are removed). Sitemap files themselves are never charged; a website is read up to 500 sitemap files per run.

## `changeMode` (type: `string`):

For scheduled monitoring. "Off" returns every URL. "New and modified pages" returns only URLs that were not in the sitemap last time, plus URLs whose lastmod date changed. "New, modified and removed pages" also returns URLs that disappeared. The previous URL list is kept per website in a key-value store named sitemap-url-extractor-state in your Apify account. You pay only for the changes returned.

## `firstRunOutput` (type: `string`):

What to return the first time a website is monitored, when there is nothing to compare with yet. "Return all URLs" saves the baseline and returns every URL with changeType "initial" (charged). "Save baseline only" returns nothing and costs nothing; changes are reported from the next run.

## `removalSafetyPercent` (type: `integer`):

If a website returns fewer URLs than this percentage of last run's count, the run assumes a sitemap failed to load: no pages are reported as removed and the saved URL list is kept. Set 0 to turn the check off.

## `includePatterns` (type: `array`):

Keep only URLs matching at least one pattern. Use a glob (*/blog/*), plain text (/products/) or a regular expression between slashes (//20(25|26)//). Leave empty to keep all.

## `excludePatterns` (type: `array`):

Drop URLs matching any pattern (same pattern syntax as above), for example */tag/* or /?page=/.

## `lastmodAfter` (type: `string`):

Keep only URLs whose sitemap lastmod date is on or after this date (URLs without a lastmod date are dropped). Format: 2026-01-31.

## `onlyDomain` (type: `boolean`):

Drop URLs that point to other domains. Subdomains of the website are kept.

## `deduplicate` (type: `boolean`):

Save each URL once, even when it appears in several sitemaps. Duplicates are not charged.

## `includeExtensions` (type: `boolean`):

Add image, video and Google News details from sitemap extensions when the sitemap has them.

## `checkStatus` (type: `boolean`):

Request every URL (HEAD, falling back to GET) and save its status code and final URL after redirects. Slower, and charged as a separate event per URL checked.

## `statusConcurrency` (type: `integer`):

How many URLs are checked at the same time when status checks are on. Keep it low to be polite to the website.

## `maxDepth` (type: `integer`):

How many levels of nested sitemap indexes to follow.

## Actor input object example

```json
{
  "startUrls": [
    "https://apify.com",
    "https://wordpress.org/news/"
  ],
  "maxUrls": 100,
  "changeMode": "off",
  "firstRunOutput": "all",
  "removalSafetyPercent": 50,
  "onlyDomain": true,
  "deduplicate": true,
  "includeExtensions": true,
  "checkStatus": false,
  "statusConcurrency": 5,
  "maxDepth": 5
}
```

# Actor output Schema

## `urls` (type: `string`):

No description

## `sitemapsReport` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://apify.com",
        "https://wordpress.org/news/"
    ],
    "maxUrls": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("pavel.tashev/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        "https://apify.com",
        "https://wordpress.org/news/",
    ],
    "maxUrls": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("pavel.tashev/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://apify.com",
    "https://wordpress.org/news/"
  ],
  "maxUrls": 100
}' |
apify call pavel.tashev/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pavel.tashev/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fzHxVhX6uDrZlyIlk/builds/pOBJrWtbJbgrQ0HZu/openapi.json