# Sitemap Extractor - Sitemap URL Scraper for XML & Gzip (`ahmed.fouaad82/sitemap-extractor`) Actor

Sitemap extractor: get deduplicated URLs from XML and gzip sitemaps, nested sitemap indexes and RSS, with lastmod and hreflang. $0.50/1,000 URLs.

- **URL**: https://apify.com/ahmed.fouaad82/sitemap-extractor.md
- **Developed by:** [Ahmed Fouad](https://apify.com/ahmed.fouaad82) (community)
- **Categories:** SEO tools, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.50 / 1,000 sitemap urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Sitemap Extractor - Sitemap URL Scraper for XML & Gzip

**Extract sitemap URLs and metadata** from XML, gzip, nested sitemap indexes, RSS/Atom and text lists for SEO specialists, migration teams and content-monitoring developers. Get deduplicated URLs, lastmod dates and available hreflang/media metadata; this does not crawl the listed pages.

**First 25 saved unique URLs of every run are free; after that, $0.50 per 1,000 saved unique URLs.** Filtered URLs, duplicates and failed sitemap documents do not create result events. The Pricing tab is authoritative. Apify plan allowances and any applicable platform/storage charges are separate.

**Example output** (illustrative abbreviated record, not a captured result):

```json
{"loc":"https://example.com/page","lastmod":"2026-10-01","sitemapSource":"https://example.com/sitemap.xml","alternates":[],"images":[],"videos":[],"news":[]}
```

### What does Sitemap URL Extractor do?

**Extract URLs from XML sitemaps**, nested sitemap indexes, compressed `.xml.gz` files, RSS/Atom feeds and plain-text URL lists. Supply domains or sitemap URLs and get a deduplicated inventory with modification dates, hreflang alternates and image/video/news metadata. This is a lightweight HTTP sitemap extraction API, not a website crawler: it never visits the listed pages or checks whether they are indexed by search engines. It follows the [Sitemaps protocol](https://www.sitemaps.org/).

### Why use this sitemap extractor?

- Build SEO audits, migration inventories and content-monitoring workflows.
- Discover sitemaps through robots.txt and `/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml`.
- Follow nested indexes, decompress gzip, preserve alternate languages and rich sitemap extensions.
- Filter URLs using safe RE2 include/exclude regexes and an inclusive last-modified date.
- Export results, schedule runs, connect integrations and call it through the Apify API.
- Stop at an exact saved-URL cap or your configured result-charge budget.

### How to extract sitemap URLs

1. Open the Input tab and enter a domain, site root, or direct sitemap URL. The default is a small public sitemap.
2. Set **maxUrls**, optionally add filters, then start the Actor.
3. Download the dataset or use the API tab for programmatic integration.
4. Check SUMMARY for skipped documents, robots restrictions and the stop reason.

### What data can it extract?

| Field | Type | Meaning |
|---|---|---|
| loc | string | Unique normalized URL; fragments removed |
| lastmod | string/null | Publisher lastmod, Atom updated or RSS pubDate |
| changefreq | string/null | Publisher-supplied change frequency |
| priority | number/null | Publisher priority, not actual search ranking |
| sitemapSource | string | Requested sitemap supplying the first occurrence |
| alternates | array | hreflang and href pairs |
| images, videos, news | arrays | Source extension metadata with namespace prefixes removed |

All nine output fields appear in the dataset schema. URLs are deduplicated within a run; first occurrence wins, so metadata is not merged across duplicate entries. Alternates and media remain nested and do not create separately billed rows. Missing extension groups are empty arrays.

### How much does sitemap extraction cost?

**First 25 saved unique URLs of every run are free; after that, $0.50 per 1,000 results.** One result = one unique, filter-matching URL saved to the dataset, including its metadata.

This Actor is public with pay-per-event pricing. The Pricing tab shows the current event price. No extra custom start fee is implemented. Apify account limits and applicable platform charges depend on your plan. Failed documents, duplicates and filtered URLs do not generate result events.

### Input and filters

See the Input tab for full configuration options. Example:

```json
{
  "startUrls": [{"url":"https://www.sitemaps.org/sitemap.xml"}],
  "maxUrls": 100,
  "maxSitemaps": 100,
  "includeRegex": "sitemaps\\.org",
  "excludeRegex": "[?]preview=",
  "lastmodSince": "",
  "delayMillis": 1000,
  "maxResponseMb": 10
}
```

A bare domain is treated as HTTPS. Root URLs trigger discovery; other URLs are parsed directly as a sitemap or feed. Include/exclude expressions match anywhere in loc and exclusion wins. RE2 deliberately rejects lookarounds and backreferences. `lastmodSince` excludes missing or invalid modification dates. Defaults cap results at 100 and sitemap documents at 100; increase these for larger inventories.

### Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. A simplified format example (not a captured result):

```json
[
  {"loc":"https://example.com/a","lastmod":"2026-10-01","changefreq":"weekly","priority":0.8,"sitemapSource":"https://example.com/sitemap.xml","alternates":[{"hreflang":"fr","href":"https://example.com/fr/a"}],"images":[],"videos":[],"news":[]},
  {"loc":"https://example.com/b","lastmod":null,"changefreq":null,"priority":null,"sitemapSource":"https://example.com/sitemap.xml","alternates":[],"images":[],"videos":[],"news":[]}
]
```

SUMMARY contains total saved rows, processed document count, stop reason and per-document errors. STATE holds deduplication keys and pending work for graceful migration/restart.

### Limits, safety and troubleshooting

Only public HTTP(S) endpoints on standard ports are fetched. Private/reserved destinations are blocked, DNS answers are pinned, and redirects are checked again. robots.txt is checked before each origin's sitemap requests. Missing robots (404/410) allows access; denied, challenged or unavailable robots fails closed. Requests are sequential, with at least one second per host and robots Crawl-delay honored up to 60 seconds. No proxy, login, browser, CAPTCHA solving or access-control bypass is used.

Both transferred and inflated documents are capped at 10 MB by default (configurable up to 20 MB). Index depth is capped at 20. Oversized or malformed XML, DTDs and entity declarations are rejected; unsupported encodings and unusual custom namespaces are not guaranteed. Image/video/news use standard extension prefixes. The Actor does not fetch media, follow feed pagination, crawl HTML links or guarantee that a sitemap lists every page. Large inventories should use direct child sitemaps and sensible caps. Crash recovery is best-effort; dataset write and billing are not transactional.

An empty successful run may mean filters excluded everything. All-document access/parsing failure fails the run and records errors. Partial runs retain valid saved rows. Use only sites you are authorized to access, comply with terms and robots rules, and avoid collecting sensitive personal data in URL query strings. Public URLs can still contain regulated data; you are responsible for a lawful purpose. Report reproducible issues through the Issues tab, with the input URL and SUMMARY error, never credentials.

# Actor input Schema

## `startUrls` (type: `array`):

1-50 domains, site roots, or direct sitemap/feed URLs. Strings or objects with url. Roots discover robots.txt and common paths.

## `maxUrls` (type: `integer`):

Stop after this many saved unique URLs. First 25 results of every run are free.

## `maxSitemaps` (type: `integer`):

Maximum discovered sitemap documents, including missing common paths.

## `includeRegex` (type: `string`):

RE2 pattern matched anywhere in loc. Empty includes all. No lookarounds or backreferences.

## `excludeRegex` (type: `string`):

RE2 pattern excluding matching loc values; exclusion wins.

## `lastmodSince` (type: `string`):

Inclusive ISO date, e.g. 2026-01-01. Missing or invalid lastmod rows are excluded when active.

## `delayMillis` (type: `integer`):

Minimum ms between requests per host. robots Crawl-delay honored up to 60 seconds.

## `maxResponseMb` (type: `integer`):

Safety limit for both transferred bytes and uncompressed XML; larger files are skipped.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.sitemaps.org/sitemap.xml"
    }
  ],
  "maxUrls": 10,
  "maxSitemaps": 100,
  "includeRegex": "",
  "excludeRegex": "",
  "lastmodSince": "",
  "delayMillis": 1000,
  "maxResponseMb": 10
}
```

# Actor output Schema

## `dataset` (type: `string`):

Saved URLs and metadata.

## `summary` (type: `string`):

Counts, stop reason and failures.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.sitemaps.org/sitemap.xml"
        }
    ],
    "maxUrls": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("ahmed.fouaad82/sitemap-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://www.sitemaps.org/sitemap.xml" }],
    "maxUrls": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("ahmed.fouaad82/sitemap-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.sitemaps.org/sitemap.xml"
    }
  ],
  "maxUrls": 10
}' |
apify call ahmed.fouaad82/sitemap-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ahmed.fouaad82/sitemap-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3aQI2d9OHi7H6eXQE/builds/NvoyhpnKDCqWYsGL6/openapi.json