# Sitemap URL Extractor: All Pages of Any Website (`jtpalms/sitemap-url-extractor`) Actor

Get every URL of a website from its XML sitemaps. Auto-discovers sitemaps from robots.txt, follows sitemap indexes and .gz files, and returns lastmod, changefreq, priority and image counts. Filter by URL pattern or last-modified date. USD 0.50 per 1,000 URLs.

- **URL**: https://apify.com/jtpalms/sitemap-url-extractor.md
- **Developed by:** [JT Palms](https://apify.com/jtpalms) (community)
- **Categories:** SEO tools, Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap URL Extractor: get every page of any website

Give it a domain and get back every URL the site lists in its XML sitemaps, with last-modified date, change frequency, priority and image count. It finds the sitemaps for you (robots.txt first, then the usual locations), follows sitemap indexes to any depth, and reads `.xml.gz` and plain-text sitemaps.

**USD 0.50 per 1,000 URLs.** Sites without a sitemap are reported free.

### What people use it for

- **SEO audits.** List every indexable page and spot stale content by `lastmod`.
- **Crawl seeding.** Feed a complete URL list into a scraper, a broken-link checker or an AI/RAG ingestion pipeline, instead of crawling link by link.
- **Content monitoring.** Schedule it daily with **Modified since** to catch a competitor's new and updated pages, products or articles.
- **Site migrations.** Snapshot all URLs before and after a redesign to build redirect maps.
- **Catalog sizing.** Count products, posts or pages on any site in seconds.

### Sample output

```json
{
  "site": "ghost.org",
  "url": "https://ghost.org/about/",
  "lastmod": "2026-08-06T10:10:45-05:00",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "sitemapUrl": "https://ghost.org/sitemap-pages.xml"
}
```

Real runs: vercel.com gave 7,383 URLs in under a second. A news sitemap index on bbc.com gave 1,688 URLs from 4 sitemaps.

### How to use it

1. Add one or more domains (`example.com`) or sitemap URLs (`https://example.com/sitemap.xml`) to **Websites or sitemap URLs**.
2. Optionally narrow the results:
   - **Only URLs containing**: `/blog/` or `/products/`.
   - **Skip URLs containing**: `/tag/`, `?page=`.
   - **Modified since**: `2026-09-01`.
3. Click **Start**, then export as CSV, JSON or Excel, or read the dataset through the Apify API.

For monitoring, create a schedule (daily or weekly), set **Modified since** to a recent date and connect the results to Google Sheets, Slack or a webhook.

### What it handles

- Sitemap indexes, nested to any depth, with each sitemap file read once.
- `.xml.gz` compressed sitemaps and plain-text sitemaps (one URL per line).
- Image sitemaps (`imageCount`, and image URLs if you turn on **Include image URLs**).
- `hreflang` alternates, returned as `alternates`.
- News sitemaps (`newsTitle`).
- Duplicate URLs across sitemaps, counted once.

### Pricing

| What | Price |
|---|---|
| URL extracted | USD 0.0005 (USD 0.50 per 1,000) |
| Site with no sitemap, or unreachable | Free |

Set a maximum cost per run in the run options and the actor stops cleanly when it gets there. **Max URLs per site** caps the spend on very large sites.

### Limits

- **The actor identifies itself.** Every request carries the user agent `Mozilla/5.0 (compatible; sitemap-url-extractor/1.0; +https://apify.com/JTPalms/sitemap-url-extractor)`, with plain request headers. It never pretends to be a browser. Some sites block bots; a blocked request is reported as an error, not hidden.
- It only returns URLs the site lists in its sitemaps. Pages missing from the sitemap are not found; use a crawler for those.
- Up to 500 sitemap files per site by default (raise it under **Advanced**).
- Some sites block bots and answer HTTP 403 (BBC's news sitemaps do); those errors are listed in the run summary. A proxy under **Advanced** only helps when a site blocks data-center addresses, not bots.

### Related actors

- [URL Status Checker: Bulk HTTP Status Code & Redirect Checker](https://apify.com/JTPalms/url-status-checker): Bulk URL status and HTTP status code checker.
- [Broken Link Checker: Find Dead Links on Any Website](https://apify.com/JTPalms/broken-link-checker): Crawl a website and find every broken link, image, script and stylesheet: 404s, 5xx, DNS and TLS errors, redirect loops and redirected...
- [SEO Audit & AI Search Readiness Checker (GPTBot, llms.txt)](https://apify.com/JTPalms/seo-ai-readiness-audit): Crawl a site and audit every page for SEO basics (title, meta, H1, canonical, hreflang, noindex, alt text, JSON-LD, status, speed) plus...
- [RSS Feed Reader & Scraper: Atom, JSON Feed & New-Item Monitor](https://apify.com/JTPalms/rss-feed-monitor): RSS feed reader and scraper.

### FAQ

**How does it find the sitemap?** It reads `Sitemap:` lines in `robots.txt`. If there are none, it tries `/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-index.xml`, `/wp-sitemap.xml` and `/sitemap.txt`.

**Is it legal?** Sitemaps are files that websites publish so machines can list their pages. The actor reads only those public files and `robots.txt`. It does not log in and collects no personal data.

**Something missing or wrong?** Open an issue with the site.

# Actor input Schema

## `startUrls` (type: `array`):

A domain (example.com) finds the sitemaps automatically from robots.txt and the usual locations. A sitemap URL (https://example.com/sitemap.xml) starts from that sitemap.

## `maxUrlsPerSite` (type: `integer`):

Stop after this many URLs per site. 0 means no limit. Large sites can list hundreds of thousands of URLs, so start with a cap and raise it once you have seen the results.

## `includeUrlPatterns` (type: `array`):

Keep only URLs that contain any of these texts, for example /blog/. Wrap in slashes for a regular expression, for example //products/\[^/]+$/.

## `excludeUrlPatterns` (type: `array`):

Drop URLs that contain any of these texts, for example /tag/ or ?page=.

## `lastmodSince` (type: `string`):

Only URLs whose sitemap lastmod is on or after this date (YYYY-MM-DD). URLs without a lastmod are skipped when this is set. Handy with a daily schedule to catch new and updated pages.

## `includeImages` (type: `boolean`):

Add up to 20 image URLs per page from image sitemaps.

## `maxSitemapsPerSite` (type: `integer`):

Stop after reading this many sitemap files on one site.

## `maxConcurrency` (type: `integer`):

How many items to process at the same time.

## `timeoutSecs` (type: `integer`):

How long to wait for one request before giving up.

## `proxyConfiguration` (type: `object`):

Optional. Only needed if a site blocks requests from data centers.

## Actor input object example

```json
{
  "startUrls": [
    "vercel.com"
  ],
  "maxUrlsPerSite": 500,
  "includeImages": false,
  "maxSitemapsPerSite": 500,
  "maxConcurrency": 3,
  "timeoutSecs": 30
}
```

# Actor output Schema

## `results` (type: `string`):

All output rows in the default dataset (JSON, CSV, Excel via the format parameter).

## `summary` (type: `string`):

Counts and per-input status for this run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "vercel.com"
    ],
    "maxUrlsPerSite": 500
};

// Run the Actor and wait for it to finish
const run = await client.actor("jtpalms/sitemap-url-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["vercel.com"],
    "maxUrlsPerSite": 500,
}

# Run the Actor and wait for it to finish
run = client.actor("jtpalms/sitemap-url-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "vercel.com"
  ],
  "maxUrlsPerSite": 500
}' |
apify call jtpalms/sitemap-url-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,jtpalms/sitemap-url-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/yVO5VUFIirylSm7Kc/builds/GbjO4xkvSS4w6eMac/openapi.json