# Sitemap URL Extractor: all URLs from XML sitemaps (`edgef-4/sitemap-url-extractor`) Actor

Every URL from a site's XML, text, RSS or Atom sitemaps, with lastmod and source file, for one site or a whole client list. Handles gzip, nested indexes and cut files. Delta mode returns only new or changed URLs. $0.40 per 1,000 URLs.

- **URL**: https://apify.com/edgef-4/sitemap-url-extractor.md
- **Developed by:** [Edge](https://apify.com/edgef-4) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / 1,000 url records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**For SEO and web agencies that track many client and prospect sites.** Sitemap URL Extractor lists every URL a site publishes in its sitemaps, with the sitemap file it came from and its lastmod date, for one site or a whole client list in one run.

**The job in one line:** give it domains, get a clean URL inventory per site, and on later runs only the URLs that are new or changed.

**What one run costs (pay per event):**

| Event | Price |
|---|---|
| URL record (one delivered URL) | $0.0004 |
| Actor start (once per run, Apify's standard fee) | $0.00005 |

1,000 URL records cost $0.40. Site summaries and error records are free. A real run on 17 Oct 2026 (wordpress.org, `maxUrls` 50) took 7.2 seconds, delivered 50 URL records plus the site summary and cost a buyer $0.02005.

### Quick start

```json
{
    "sites": ["wordpress.org"],
    "maxUrls": 50
}
```

For a client portfolio, list every domain under `sites`, raise `maxUrls` (default 1,000, up to 1,000,000, counted across all sites) and switch on `deltaMode` for scheduled runs.

### Output example

REAL records from a run of the input above, read on 8 Oct 2026 at 01:53 UTC (one of its 50 `url` records and its `site_summary`, `sitemaps_listed` shortened to 3 of 9 here). The site changes, so a later run can give other URLs and dates.

```json
[
  {
    "id": "be5e3f0b503a8a36269f",
    "record_type": "url",
    "status": "ok",
    "site": "wordpress.org",
    "url": "https://wordpress.org/about/privacy/cookies/",
    "lastmod": "2026-09-01T12:37:54Z",
    "changefreq": null,
    "priority": null,
    "sitemap_url": "https://wordpress.org/sitemap-1.xml",
    "depth": 1,
    "images_count": 0,
    "videos_count": 0,
    "news_count": 0,
    "sitemap_truncated": false,
    "change": null,
    "source_url": "https://wordpress.org/sitemap-1.xml",
    "fetched_at": "2026-10-08T01:53:12Z"
  },
  {
    "id": "c30ba320d4fd9550c3c0",
    "record_type": "site_summary",
    "status": "ok",
    "error_type": null,
    "site": "wordpress.org",
    "source_url": "https://wordpress.org/robots.txt",
    "fetched_at": "2026-10-08T01:53:10Z",
    "discovery": "robots.txt",
    "sitemaps_listed": [
      "https://wordpress.org/sitemap.xml",
      "https://wordpress.org/news-sitemap.xml",
      "https://wordpress.org/themes/sitemap.xml"
    ],
    "files_read": 2,
    "files_failed": 0,
    "urls_found": 50,
    "urls_delivered": 50,
    "duplicates": 0,
    "filtered_out": 0,
    "other_host_skipped": 0,
    "unchanged_skipped": 0,
    "loops_skipped": 0,
    "truncated": false,
    "truncated_files": [],
    "errors_by_type": {},
    "errors": [],
    "stopped_reason": "maxUrls reached"
  }
]
```

In delta mode `change` is `new` or `lastmod_changed`. `id` comes from the site and the URL, so it is the same on every run.

### Input

| Field | Default | Meaning |
|---|---|---|
| `sites` | none | Domains (`example.com`) or site addresses. Sitemaps are found through robots.txt, then `/sitemap.xml`, `/sitemap_index.xml` and `/wp-sitemap.xml`. |
| `sitemapUrls` | none | Sitemap or index files to read directly. |
| `include` / `exclude` | none | Regular expressions to keep or drop URLs (for example only `/blog/`). |
| `lastmodAfter` | none | Keep only URLs with a lastmod on or after this date (YYYY-MM-DD, UTC). |
| `maxUrls` | 1000 | Stop after this many URL records in total, across all sites. |
| `sameHostOnly` | true | Keep only URLs on the site's own host or its www form. |
| `deltaMode` | false | Deliver only URLs that are new or whose lastmod changed since the last run with the same sites and filters. |

Give at least one entry in `sites` or `sitemapUrls`.

### Three agency use cases

1. **Site migration and redesign checks.** Pull the full URL list of a client site before a migration and again after it. Compare the two lists to build the redirect map and to catch pages that dropped out of the sitemap.
2. **Competitor content watch for clients.** Schedule a weekly run over each client's competitors with `deltaMode` on. Each run returns only the pages that are new or whose lastmod changed, so the strategist sees new landing pages and posts without reading whole sitemaps.
3. **Feed for your own audit tools.** Use the URL list as the input for a link checker, a crawler or an indexing check across the whole portfolio. `site_summary` tells you per site why a sitemap gave nothing (`not_found`, `blocked`, `error`, `skipped`).

### Limits and what it does not do

- Reads only robots.txt and sitemap files, with GET requests. It never opens the pages a sitemap lists, so it does not check status codes, titles or indexing.
- Obeys robots.txt: a sitemap file that robots.txt disallows is reported, not read. Some sites refuse automated agents; those come back as `blocked` with the HTTP code.
- At most 2 requests per second per host, 30 seconds per request, 50 MB per file, 500 sitemap files per site, 5 index levels.
- `maxUrls` counts across sites: sites after the limit is reached are not read and get a `site_summary` with status `skipped`.
- A sitemap file that ends early gives the URLs before the cut, flagged with `sitemap_truncated` and `truncated`.
- Delta state is kept in a named key-value store in your account, per site and filter set.

### FAQ

**Does it follow sitemap index files?** Yes, up to 5 levels. An index that points back at itself or at one already read is skipped and counted under `loops_skipped`.

**Why did a site give no URLs?** Its `site_summary` says why: `not_found` (no sitemap at robots.txt or the usual addresses), `blocked`, `skipped` (`maxUrls` reached before its turn) or `error` with the error types.

**Does `truncated: false` mean I have every URL?** No. It means the files read were complete. Check `stopped_reason` too: `maxUrls reached` means the run stopped at your limit.

**Are duplicates charged?** No. A URL listed twice is delivered and charged once per run and counted under `duplicates`.

**How do I cap my spend?** Set `maxUrls` and a maximum cost per run in the run options. The run stops cleanly when the maximum charge is reached.

Keywords: sitemap URL extractor, sitemap crawler for agencies, XML sitemap to CSV, site migration URL list, new page monitor, competitor content tracking, sitemap index parser

# Changelog

This Actor's version history is a separate document: https://apify.com/edgef-4/sitemap-url-extractor/changelog.md

# Actor input Schema

## `sites` (type: `array`):

Domains (example.com) or site addresses (https://example.com). Sitemaps are found through robots.txt, then /sitemap.xml, /sitemap_index.xml and /wp-sitemap.xml.

## `sitemapUrls` (type: `array`):

Full addresses of sitemap or sitemap index files to read directly, in addition to or instead of sites.

## `include` (type: `array`):

Regular expressions. When given, a URL is kept only if it matches at least one.

## `exclude` (type: `array`):

Regular expressions. A URL that matches any of them is left out.

## `lastmodAfter` (type: `string`):

Keep only URLs whose lastmod is on or after this date (YYYY-MM-DD, UTC). URLs without a lastmod are left out when this is set.

## `maxUrls` (type: `integer`):

Stop after delivering this many URL records in total.

## `sameHostOnly` (type: `boolean`):

Keep only URLs on the site's own host or its www form, as the sitemap protocol requires.

## `deltaMode` (type: `boolean`):

Deliver only URLs that are new or whose lastmod changed since the last run with the same sites and filters.

## Actor input object example

```json
{
  "sites": [
    "wordpress.org"
  ],
  "sitemapUrls": [],
  "include": [],
  "exclude": [],
  "maxUrls": 50,
  "sameHostOnly": true,
  "deltaMode": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "wordpress.org"
    ],
    "maxUrls": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("edgef-4/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sites": ["wordpress.org"],
    "maxUrls": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("edgef-4/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "wordpress.org"
  ],
  "maxUrls": 50
}' |
apify call edgef-4/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,edgef-4/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/j9jTWjBHJ5RqTxzfM/builds/4iKO7TVIeuPWc3zB5/openapi.json