# Sitemap Extractor Done Right (`inn_corp/sitemap-extractor-pro`) Actor

Parse XML sitemaps into clean URL records. Gzip support, nested sitemap-index recursion with a sane depth cap, lastmod/pattern filtering. Reads only files a site published for exactly this purpose.

- **URL**: https://apify.com/inn\_corp/sitemap-extractor-pro.md
- **Developed by:** [Inn Corp](https://apify.com/inn_corp) (community)
- **Categories:** SEO tools, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap Extractor Done Right

Give it a sitemap URL, get back one clean record per URL: `loc`, `lastmod`,
`changefreq`, `priority`. Handles gzip, follows sitemap index files into
their children with a depth cap so a nested index tree can never run away,
and filters by URL pattern or by last-modified date. Built because the
current store leader for this sits at 3.2 stars on real demand, which is a
gap this Actor is built to close.

### What it does

- Fetches each `sitemapUrls` entry with a plain HTTP GET.
- Detects gzip automatically by sniffing the response's magic bytes, so a
  literal `.xml.gz` file and a server that transfer-encodes a plain `.xml`
  body both just work, no configuration needed.
- Parses XML namespace-agnostically: whatever namespace URI (or none) a
  sitemap declares, `<url>`, `<loc>`, `<lastmod>` etc. are matched by local
  tag name, not by exact namespace string. Extra vendor tags (image, video,
  news extensions) are present in plenty of real sitemaps and are ignored
  cleanly rather than tripping the parser.
- When a file is a sitemap index (`<sitemapindex>`), follows its child
  `<sitemap><loc>` entries and processes each one the same way, recursively,
  up to `maxDepth` levels.
- Filters by `urlPattern` (regex, or a plain substring if the regex does not
  compile) and by `respectLastmodAfter` (a date floor on `<lastmod>`).
- Dedupes URLs across the entire run: the same URL discovered twice, whether
  from two different sitemap files or two different `sitemapUrls` entries,
  is written and charged once.
- Writes a `summary` record per `sitemapUrls` entry no matter what happened:
  clean run, partial run, or failure. `status` is one of `ok`, `error`
  (malformed or non-sitemap XML), `too-many-nested` (nesting exceeded
  `maxDepth` before any URL could be found), `too-many-sitemaps` (the entry
  has more sitemap files than `maxSitemapFiles` allows, before any URL could
  be found), or `download-failed` (the file itself could not be fetched).

### What it deliberately does not do

- **No content fetching.** This Actor reads only the sitemap XML the site
  published. It never fetches the pages the sitemap lists.
- **No runaway recursion or fan-out.** Two independent guards, both enforced
  in code, not left to trust: `maxDepth` stops a sitemap index that points to
  indexes that point to indexes, and `maxSitemapFiles` stops a site with many
  shallow sitemap files (per-category or per-day sitemaps are common) from
  racking up one fetch and one charge per file even when `maxUrlsPerSitemap`
  is small, since that cap only counts URL records, not files visited. A
  cycle guard also stops an index that points back at itself. Any of the
  three stopping a branch shows up in the summary instead of the run hanging.
- **No invented fields.** `lastmod`, `changefreq`, and `priority` come from
  the sitemap or are `null`. A `respectLastmodAfter` filter excludes URLs
  with no `<lastmod>` at all, since there is nothing to verify them against;
  that is documented here, not a silent surprise.

### Sitemaps and machine access

A sitemap is a file a site owner publishes at a predictable, public URL
specifically so it can be parsed by software, per the
[sitemaps.org protocol](https://www.sitemaps.org/protocol.html) that Google,
Bing, and every other search engine already read the same way. This Actor
processes only sitemap URLs you supply; it is not choosing what to crawl and
it is not reading page content, only the index file the site itself
published for exactly this purpose.

### Output example

Real records from `https://squareup.com/sitemap.xml`, a live sitemap index
whose children include both gzipped leaf sitemaps and further nested
sitemap indexes:

A `url` record, from a gzipped child (`sitemap.xml.gz`, decompressed
automatically):

```json
{
  "recordType": "url",
  "sourceSitemapUrl": "https://squareup.com/jp/ja/sitemap.xml.gz",
  "url": "https://squareup.com/jp/ja/townsquare/omino",
  "lastmod": "2026-08-25T03:00:40.493Z",
  "changefreq": "weekly",
  "priority": "0.5",
  "discoveredAt": "2026-08-25T03:29:39+00:00"
}
```

The matching `summary` record for the top-level entry:

```json
{
  "recordType": "summary",
  "url": "https://squareup.com/sitemap.xml",
  "status": "ok",
  "urlsFound": 18223,
  "sitemapsProcessed": 17,
  "error": null
}
```

### Input

| Field | Meaning |
| --- | --- |
| `sitemapUrls` | Sitemap files to parse: plain `.xml`, gzipped `.xml.gz`, or a sitemap index. Required. |
| `maxUrlsPerSitemap` | Total URL cap per `sitemapUrls` entry, shared across every child sitemap discovered under it. Default 5000, max 100000. |
| `followSitemapIndexes` | Follow child sitemaps listed by a sitemap index. Default on; off returns zero URLs for an index-only entry. |
| `urlPattern` | Keep only matching URLs. Tried as a regex first, falls back to a plain substring match if it does not compile. |
| `maxDepth` | Levels of nested sitemap indexes to follow. The entry itself is depth 1. Default 3. |
| `maxSitemapFiles` | Total sitemap FILES (not URL records) one entry may fetch, across its whole nested tree. Independent of `maxUrlsPerSitemap`, which only counts URL records. Default 500, max 5000. |
| `respectLastmodAfter` | `YYYY-MM-DD`. Keeps only URLs whose `<lastmod>` is on or after this date; URLs with no `<lastmod>` are excluded once this is set. |

### Typical uses

- Pull every URL a site has published, for a crawl budget or a content
  audit, without writing an XML parser.
- Feed a downstream scraper a clean, deduped, pattern-filtered URL list
  instead of every URL on the site.
- Track what changed recently with `respectLastmodAfter` on a schedule.
- Sanity-check a site's own sitemap: `too-many-nested` and `error` statuses
  surface a badly configured sitemap tree immediately.

### Fair pricing

Pay per URL record returned and per sitemap file successfully fetched and
parsed (index files and leaf files both count), once pay-per-event pricing
is enabled. A sitemap that fails to download or does not parse as valid
sitemap XML costs nothing. No subscription.

# Actor input Schema

## `sitemapUrls` (type: `array`):

Sitemap files to parse: plain .xml, gzipped .xml.gz, or a sitemap index (<sitemapindex>) that points to child sitemaps. Each URL you give gets its own summary record.

## `maxUrlsPerSitemap` (type: `integer`):

Total cap on URL records for one sitemapUrls entry, shared across every child sitemap discovered under it (not reset per file). Processing for that entry stops once the cap is hit.

## `followSitemapIndexes` (type: `boolean`):

When a file is a sitemap index (<sitemapindex>), fetch and process the child sitemaps it lists. Off returns only whatever the top-level file itself contains directly (an index alone yields zero URLs).

## `urlPattern` (type: `string`):

Keep only URLs matching this. Tried as a regex first (re.search); if it is not valid regex, used as a plain substring match instead. Leave empty for no filter.

## `maxDepth` (type: `integer`):

How many levels of nested sitemap indexes to follow before giving up on that branch. The sitemapUrls entry itself is depth 1. Prevents runaway recursion on indexes that point to indexes that point to indexes.

## `maxSitemapFiles` (type: `integer`):

Total cap on how many sitemap FILES (not URL records) one sitemapUrls entry is allowed to fetch, shared across the whole nested tree under it. A site with many shallow sitemap files (per-category or per-day sitemaps are common) can otherwise rack up one fetch and one sitemap-processed charge per file even when maxUrlsPerSitemap is small, since that cap only counts URL records, not files visited. This is the independent breadth guard for that case.

## `respectLastmodAfter` (type: `string`):

YYYY-MM-DD. Keeps only URLs whose <lastmod> is on or after this date. A URL entry with no <lastmod> is excluded once this filter is set, since it cannot be verified. Leave empty for no filter.

## Actor input object example

```json
{
  "sitemapUrls": [
    "https://squareup.com/sitemap.xml"
  ],
  "maxUrlsPerSitemap": 5000,
  "followSitemapIndexes": true,
  "urlPattern": "/blog/",
  "maxDepth": 3,
  "maxSitemapFiles": 500,
  "respectLastmodAfter": "2026-01-01"
}
```

# Actor output Schema

## `records` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sitemapUrls": [
        "https://squareup.com/sitemap.xml"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("inn_corp/sitemap-extractor-pro").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sitemapUrls": ["https://squareup.com/sitemap.xml"] }

# Run the Actor and wait for it to finish
run = client.actor("inn_corp/sitemap-extractor-pro").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sitemapUrls": [
    "https://squareup.com/sitemap.xml"
  ]
}' |
apify call inn_corp/sitemap-extractor-pro --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,inn_corp/sitemap-extractor-pro"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/d1FH7CuAvVU0bDKDw/builds/Y6X1ikl2TyyJfmhvE/openapi.json