# Sitemap URL Intelligence (`axel_brasil/sitemap-url-intelligence`) Actor

Extract clean URL inventories from sitemap.xml files for SEO audits, site migrations, competitor research, and AI/RAG ingestion.

- **URL**: https://apify.com/axel\_brasil/sitemap-url-intelligence.md
- **Developed by:** [Lucas Bonardo](https://apify.com/axel_brasil) (community)
- **Categories:** SEO tools, Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.10 / 1,000 urls

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Sitemap URL Intelligence

Turn any website's `sitemap.xml` into a clean, structured URL inventory — in
seconds, without setting up a crawler. Point the Actor at a homepage or a
sitemap URL and get back one tidy row per page, enriched with SEO and content
metrics, ready to export as **JSON, CSV or Excel**.

Perfect for **SEO audits, site migrations, competitor content research, and
AI/RAG ingestion**, where you need a complete, deduplicated list of a site's
URLs fast.

### Why use it

- **No crawling required.** Sitemaps are the site's own map — reading them is
  faster, cheaper and gentler than crawling every page.
- **Handles real-world sitemaps.** Follows `sitemapindex` files recursively,
  deduplicates URLs across overlapping sitemaps, and survives broken or missing
  sitemaps without crashing the run.
- **Instant enrichment.** Every URL is classified (homepage / html / image /
  document / file) with path depth, extension and query-param counts — the raw
  material for an SEO or content audit.
- **Export anywhere.** Results land in an Apify dataset you can download as CSV,
  JSON or Excel, or pull via the API into your own pipeline.

### What it does

1. Accepts website homepages **or** direct sitemap URLs. For a homepage it
   automatically tries `/sitemap.xml`.
2. Detects **sitemap index** files and recurses into every child sitemap.
3. Extracts each `<url>` entry with its `lastmod`, `changefreq` and `priority`.
4. Optionally enriches each URL with derived metrics (see below).
5. Deduplicates URLs across all sitemaps and writes one row per page.
6. Records unreachable or malformed sitemaps as error rows so nothing is lost.

### Use cases

| Buyer | How they use it |
| --- | --- |
| **SEO consultant** | Export the full URL set to check indexation coverage, spot orphan sections, and compare `lastmod` freshness. |
| **Migration engineer** | Snapshot every live URL before a replatform, then diff old vs. new to build redirect maps. |
| **Content marketer** | Pull a competitor's blog/product URLs to size their content library and find gaps. |
| **AI / RAG builder** | Get a deduplicated URL list to seed a document ingestion or embedding pipeline. |
| **Agency / analyst** | Deliver a client-ready URL inventory spreadsheet without standing up a scraper. |

### Input

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `startUrls` | array | — | Homepages or sitemap XML URLs (required). |
| `maxSitemaps` | integer | `50` | Cap on sitemap files fetched, incl. nested ones. |
| `maxUrls` | integer | `10000` | Stop after this many unique URLs. |
| `includeUrlMetrics` | boolean | `true` | Add SEO/content columns to each row. |
| `requestTimeoutSecs` | integer | `30` | Per-request timeout (2 retries on failure). |
| `proxyConfiguration` | object | disabled | Optional Apify Proxy for blocked targets. |

#### Example input

```json
{
  "startUrls": [{ "url": "https://www.apify.com/sitemap.xml" }],
  "maxUrls": 1000,
  "includeUrlMetrics": true
}
```

You can also pass a bare homepage and let the Actor find the sitemap:

```json
{ "startUrls": [{ "url": "https://example.com" }] }
```

### Output

One dataset item per URL. With `includeUrlMetrics` enabled a row looks like:

```json
{
  "url": "https://example.com/blog/how-to-audit-seo",
  "sourceSitemap": "https://example.com/sitemap.xml",
  "lastmod": "2026-05-12",
  "changefreq": "weekly",
  "priority": 0.8,
  "hostname": "example.com",
  "path": "/blog/how-to-audit-seo",
  "pathDepth": 2,
  "extension": null,
  "urlType": "html",
  "queryParamCount": 0,
  "scrapedAt": "2026-07-04T10:00:00.000Z"
}
```

#### Output fields

| Field | Description |
| --- | --- |
| `url` | The page URL from the sitemap (relative locs resolved to absolute). |
| `sourceSitemap` | The sitemap file this URL came from. |
| `lastmod` / `changefreq` / `priority` | Standard sitemap hints (`null` if absent). |
| `hostname` / `path` | Parsed host and path. |
| `pathDepth` | Number of path segments (a rough section-depth signal). |
| `extension` | File extension, or `null` for extension-less pages. |
| `urlType` | `homepage`, `html`, `image`, `document`, `file`, or `invalid`. |
| `queryParamCount` | Number of query-string parameters. |
| `scrapedAt` | ISO timestamp of when the row was produced. |

Rows for sitemaps that could not be fetched or parsed contain `sourceSitemap`,
`error` and `scrapedAt` instead — so failures are visible, not silent.

A run **summary** is also written to the default key-value store under the key
`SUMMARY`:

```json
{
  "sitemapsFetched": 4,
  "sitemapErrors": 0,
  "urlsFound": 1287,
  "urlTypes": { "html": 1180, "image": 90, "document": 17 },
  "generatedAt": "2026-07-04T10:00:05.000Z"
}
```

### Limitations

- Reads what the sitemap declares — pages **missing from the sitemap** won't
  appear. This is not a crawler.
- **Gzipped sitemaps** (`.xml.gz`) are fetched but not decompressed; point the
  Actor at the plain `.xml` variant when available.
- `lastmod`, `changefreq` and `priority` are self-reported by the site and may
  be stale or absent.
- Very large sites are bounded by `maxUrls` / `maxSitemaps`; raise them for a
  full export.

### Pricing positioning

Runs are cheap and fast because there is no page rendering or deep crawling —
just a handful of XML fetches. A typical mid-size site (a few thousand URLs)
completes in well under a minute on the smallest memory setting, making this one
of the lowest-cost ways to get a complete URL inventory on the Apify platform.

### Tips

- Feed the output straight into the **Email & Social Lead Finder** Actor to turn
  a URL inventory into contact data.
- Schedule the Actor and diff successive runs to monitor when a site publishes or
  removes pages.

***

**Keywords:** sitemap parser, sitemap.xml extractor, URL inventory, SEO audit,
site migration, XML sitemap crawler, sitemap index, URL list export, content
research, RAG ingestion, website URL scraper, SEO data.

# Actor input Schema

## `startUrls` (type: `array`):

Website homepages (e.g. https://example.com) or direct sitemap XML URLs (e.g. https://example.com/sitemap.xml). For homepages the Actor automatically tries /sitemap.xml. Sitemap index files are followed recursively.

## `maxSitemaps` (type: `integer`):

Safety cap on how many individual sitemap files to fetch, including nested sitemaps discovered inside a sitemap index. Large sites split their sitemap into many files.

## `maxUrls` (type: `integer`):

Stop after collecting this many unique page URLs across all sitemaps. Use it to keep runs fast and cheap while sampling large sites.

## `includeUrlMetrics` (type: `boolean`):

Add derived SEO/content columns to every row: hostname, path, path depth, file extension, URL type (homepage / html / image / document / file) and query-parameter count. Turn off for a minimal URL-only export.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for each sitemap request before giving up. Failed fetches are retried twice and, if still failing, recorded as an error row instead of stopping the run.

## `proxyConfiguration` (type: `object`):

Optional proxy. Sitemaps are usually publicly reachable without a proxy; enable Apify Proxy only if a target blocks datacenter IPs or you need a specific country.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.apify.com/sitemap.xml"
    }
  ],
  "maxSitemaps": 50,
  "maxUrls": 10000,
  "includeUrlMetrics": true,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `urls` (type: `string`):

Sitemap URLs with SEO/content metrics.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.apify.com/sitemap.xml"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("axel_brasil/sitemap-url-intelligence").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://www.apify.com/sitemap.xml" }] }

# Run the Actor and wait for it to finish
run = client.actor("axel_brasil/sitemap-url-intelligence").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.apify.com/sitemap.xml"
    }
  ]
}' |
apify call axel_brasil/sitemap-url-intelligence --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,axel_brasil/sitemap-url-intelligence"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/a2tNJO27XPdgjR7zK/builds/IyAImycYURIee9t9f/openapi.json