# Sitemap Extractor — All URLs, lastmod, Images & hreflang (`saliga/sitemap-extractor`) Actor

Get every URL a website lists in its XML sitemaps: found through robots.txt, indexes and .gz followed, with lastmod, images, hreflang and news. Filter by date or pattern, check HTTP status.

- **URL**: https://apify.com/saliga/sitemap-extractor.md
- **Developed by:** [Salih Küçükkayış](https://apify.com/saliga) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.25 / 1,000 url from a sitemaps

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

Get every URL a website lists in its XML sitemaps. Give a website or a sitemap link: sitemaps are found through **robots.txt** and the usual locations, **sitemap indexes**, **.xml.gz** files and text sitemaps are followed, and each URL comes back with its **lastmod**, changefreq, priority, **images**, **hreflang alternates** and **news** data. Filter by date ("changed since") or by pattern, and optionally **check the HTTP status** of every URL to find broken links and redirects.

Use it to feed a crawler or [Bulk HTTP Requests](https://apify.com/saliga/bulk-http-requests), to audit a site's SEO coverage, to watch a competitor's new pages, or to get a clean URL list for RAG ingestion.

**$0.25 per 1,000 URLs. $1 per 1,000 status checks** (optional). Sites without a readable sitemap are free. No browser and no proxy needed, so runs are fast and cheap.

### Quick start

Press **Start** with the example input:

```json
{
  "startUrls": ["https://docs.apify.com/sitemap.xml"],
  "maxUrlsPerSite": 200
}
```

One row per URL (trimmed):

```json
{
  "url": "https://docs.apify.com/api",
  "lastmod": null,
  "changefreq": "weekly",
  "priority": 0.5,
  "sitemapUrl": "https://docs.apify.com/sitemap_base.xml",
  "site": "https://docs.apify.com"
}
```

With *Check HTTP status* on, each row also gets `status` (for example 200 or 404) and `finalUrl` when the URL redirects.

### What it does

- **Finds the sitemaps**: `Sitemap:` lines in robots.txt; if there are none, `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`, `/sitemap.xml.gz` and `/sitemap-index.xml`. Or give the sitemap link yourself.
- **Follows everything**: sitemap indexes (nested up to 6 levels, loops ignored), gzip files, text sitemaps, child sitemaps on another host (for example a CDN).
- **Extras**: `images` (image sitemap), `alternates` (hreflang links), `newsTitle` and `newsDate` (news sitemap), `videoTitles`.
- **Filters**: *Changed since* keeps URLs with a recent lastmod and skips whole child sitemaps whose lastmod is older, which saves time on big sites. Regular-expression include and exclude patterns.
- **Status check** (optional): HEAD request, and any error confirmed with a GET that does not download the page (many servers answer HEAD wrongly). Redirects followed, final URL reported. A page that does not answer within 15 seconds is reported with `checkError`.
- **Summary per site** in the key-value store record `SUMMARY`: sitemaps read and their errors, URLs found, filtered out and duplicates.
- **Duplicates removed**: a URL listed in several sitemaps is returned (and charged) once per site.

### Polite and safe

- robots.txt is read once per site and honoured for every sitemap and every status check. A sitemap or page that robots.txt disallows is not requested.
- One request at a time per host, with a short pause between requests.
- XML is parsed safely (no entity expansion or external entities); gzip is unpacked with a size cap.
- Private and internal addresses (localhost, 10.x, 192.168.x, cloud metadata) are blocked, including every redirect.

### Input

| Field | Default | What it does |
|---|---|---|
| `startUrls` | | Websites or sitemap links |
| `maxUrlsPerSite` | 10,000 | Cap per site |
| `lastmodSince` | | Only URLs changed on or after this date: `2026-10-01`, `7 days`, `2 weeks`, `yesterday` |
| `keepUndated` | false | With `lastmodSince`, keep URLs that have no lastmod |
| `includePatterns`, `excludePatterns` | | Regular expressions on the URL |
| `includeExtras` | true | Images, hreflang, news, video titles |
| `checkStatus` | false | HTTP status of each URL |
| `maxStatusChecksPerSite` | 1,000 | Cap on status checks per site |
| `maxSitemapsPerSite` | 1,000 | Cap on sitemap files read per site |

### What it costs

| Run | Price |
|---|---|
| A blog with 2,000 URLs | $0.50 |
| 20 sites, 500 URLs each | $2.50 |
| Pages changed in the last week on a 1-million-URL news site (`lastmodSince`) | only the new URLs are charged |
| 1,000 URLs with status checks | $1.25 |

You only pay for URLs you get. The run stops cleanly at your spending limit.

### Good to know

- Only what the sitemaps list is returned. Pages a site leaves out of its sitemaps are not found; this is not a crawler.
- `lastmod` is whatever the site writes; some sites put today's date on every URL.
- Status checks are polite (about 4 requests a second per site), so checking 10,000 URLs of one site takes around 45 minutes. Checks on different sites run in parallel. The log shows progress, and a run that is about to hit its timeout stops checking and returns everything it has (checks not done are not charged).
- A site that answers its sitemap URL with an HTML page or JSON is reported as "Not a sitemap".

### FAQ

**Can I find every page that changed since my last run?** Yes: schedule the Actor daily with *Changed since* = `1 day` (or weekly with `7 days`). The date moves with every run.

**Can I use the URLs in another Actor?** Yes. Pass the dataset to [Bulk HTTP Requests](https://apify.com/saliga/bulk-http-requests), [Website Screenshot + Change Detection](https://apify.com/saliga/website-screenshot-change-detection) or any crawler that takes a URL list.

**Can an AI agent use it?** Yes, through the Apify MCP server: the agent gives a website and gets its URL list back.

### More from this developer

- [Bulk HTTP Requests](https://apify.com/saliga/bulk-http-requests): call any API once for every row of a CSV, JSON list or Apify dataset.
- [Website Screenshot + Change Detection](https://apify.com/saliga/website-screenshot-change-detection): screenshots of many URLs on desktop and mobile, with visual change detection between runs.
- [Document to Markdown](https://apify.com/saliga/document-to-markdown): PDF, Word, PowerPoint, Excel and scans to clean Markdown for LLMs and RAG.
- [Keyword Search Volume](https://apify.com/saliga/keyword-search-volume): Google search volume, CPC and trends for a keyword list, in up to 20 countries in one run.
- [Audio to Text](https://apify.com/saliga/audio-to-text): audio, video and podcast feeds to timestamped text and subtitles with Whisper.
- [Video to Scenes](https://apify.com/saliga/video-to-scenes): scene cuts, keyframes, a contact sheet and the transcript of each scene.
- [Career Site Jobs](https://apify.com/saliga/career-site-jobs): open jobs from Greenhouse, Lever, Ashby, Workable, Recruitee and Workday career sites.
- [Image to Text OCR](https://apify.com/saliga/image-to-text-ocr): text from images and scanned PDFs in 40+ languages, with confidence and word boxes, $1.50 per 1,000 images.
- [Tech Stack Detector](https://apify.com/saliga/tech-stack-detector): which CMS, ecommerce platform, analytics and hosting a website uses, $1.50 per 1,000 websites.

Code examples (curl, Python, Node.js, n8n) for this and the other Actors: [github.com/msalihk/apify-actor-examples](https://github.com/msalihk/apify-actor-examples).

Questions or a missing feature: open an issue on the Actor page.

# Actor input Schema

## `startUrls` (type: `array`):

A website (example.com) or a sitemap link (https://example.com/sitemap.xml, .xml.gz, sitemap index or text sitemap). For a website, the sitemaps listed in robots.txt are used; if there are none, the usual locations (/sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml) are tried.

## `maxUrlsPerSite` (type: `integer`):

Stop collecting a site's URLs after this many. Caps cost on very large sites.

## `lastmodSince` (type: `string`):

Only URLs whose lastmod is on or after this date, or within the last days, weeks or months (for example "7 days" in a daily or weekly schedule). Child sitemaps of an index whose lastmod is older are not even downloaded. Leave empty for all URLs.

## `keepUndated` (type: `boolean`):

With "Changed since", also return URLs that have no lastmod.

## `includePatterns` (type: `array`):

Keep only URLs that match at least one of these regular expressions (case-insensitive), for example /blog/ or .pdf$.

## `excludePatterns` (type: `array`):

Drop URLs that match any of these regular expressions, for example /tag/ or ?page=.

## `includeExtras` (type: `boolean`):

Add image links, hreflang alternates, news titles and dates, and video titles when the sitemap has them.

## `checkStatus` (type: `boolean`):

Request each returned URL (HEAD, or GET when HEAD is refused) and add its status and the final URL after redirects. URLs that robots.txt disallows are not requested and are marked blockedByRobots.

## `maxStatusChecksPerSite` (type: `integer`):

Check at most this many URLs per site (the first ones in sitemap order).

## `maxSitemapsPerSite` (type: `integer`):

Stop after reading this many sitemap files for one site (indexes included).

## Actor input object example

```json
{
  "startUrls": [
    "https://docs.apify.com/sitemap.xml"
  ],
  "maxUrlsPerSite": 10000,
  "keepUndated": false,
  "includePatterns": [],
  "excludePatterns": [],
  "includeExtras": true,
  "checkStatus": false,
  "maxStatusChecksPerSite": 1000,
  "maxSitemapsPerSite": 1000
}
```

# Actor output Schema

## `results` (type: `string`):

One row per URL: lastmod, HTTP status when checked, and the sitemap it came from.

## `fullResults` (type: `string`):

Every field, including images, hreflang alternates, news titles and redirect targets.

## `summary` (type: `string`):

Sitemaps read, errors, URLs found and filtered, per site.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://docs.apify.com/sitemap.xml"
    ],
    "includePatterns": [],
    "excludePatterns": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("saliga/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://docs.apify.com/sitemap.xml"],
    "includePatterns": [],
    "excludePatterns": [],
}

# Run the Actor and wait for it to finish
run = client.actor("saliga/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://docs.apify.com/sitemap.xml"
  ],
  "includePatterns": [],
  "excludePatterns": []
}' |
apify call saliga/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,saliga/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/iU4JnYrI7ve0oTeQd/builds/Ih2OHIq3I4bbGfOnj/openapi.json