# SITEMAP INTELLIGENCE — every URL a site publishes, sorted (`egeusta/sitemap-intelligence`) Actor

Extract every URL from XML and gzipped sitemaps, classify products, categories, blog posts, and pages, and get per-site totals with lastmod metadata—ideal for SEO audits, migrations, and catalog intelligence.

- **URL**: https://apify.com/egeusta/sitemap-intelligence.md
- **Developed by:** [Ege](https://apify.com/egeusta) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 url extracteds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

Turn a website's XML sitemaps into a structured URL inventory with **content classification**, **last-modified metadata**, and **per-site totals**. Sitemap Intelligence discovers sitemap files from `robots.txt` and conventional locations, follows sitemap indexes, reads compressed `.xml.gz` files, and returns a sortable Apify dataset.

Give it a homepage such as `https://example.com`. It does not crawl every page: it reads the site's own sitemap declarations, so large catalogs can be mapped quickly with lightweight HTTP requests.

### What Sitemap Intelligence does

- Discovers sitemap files from `robots.txt`, `/sitemap.xml`, and related conventional paths.
- Follows sitemap indexes into child sitemap files.
- Supports regular XML and gzipped XML sitemaps.
- Classifies each URL as `product`, `category`, `blog`, `page`, or `other`.
- Preserves the sitemap's `lastmod` value and the source sitemap URL.
- Adds a free summary row for each site with totals by content type.
- Runs with limited permissions and respects the run's maximum charge.

Classification trusts sitemap context first. For example, a URL inside `sitemap_products_1.xml` is treated as a product even when its path does not contain `/products/`. URL-path patterns are only the fallback, and uncertain records remain `other` instead of being forced into a misleading category.

### How to use

1. Add one or more site or store URLs in **Sites**. A homepage URL is enough.
2. Set **Maximum URLs** to cap the number of paid URL records.
3. Optionally choose content types such as **Product** or **Category**. Filtered-out URLs are not charged.
4. Adjust **Sitemaps to read per site** only when a large site splits its catalog across many files.
5. Click **Start**.
6. Open the **Output** tab to switch between the URL table and the per-site summary. Export the dataset as JSON, CSV, Excel, XML, RSS, or HTML.

Example input:

```json
{
  "startUrls": [{ "url": "https://www.gymshark.com" }],
  "maxUrls": 5000,
  "includeTypes": ["product"],
  "maxSitemapsPerSite": 15
}
```

### Input

| Field | What it does |
|---|---|
| `startUrls` | Site or store URLs. The homepage is enough. |
| `maxUrls` | Hard cap across the entire run. |
| `includeTypes` | Optional filter for products, categories, blog posts, pages, or other URLs. |
| `maxSitemapsPerSite` | Maximum sitemap files to read for each site. |
| `crawlerIdentity` | Optional custom User-Agent for transparent, scheduled crawling. |

The cheapest way to target one kind of content is to set `includeTypes`. Filtering happens while the sitemap is read, so the run spends its result budget only on records you requested.

### Output

The default dataset contains two row types.

A URL row:

```json
{
  "rowType": "url",
  "url": "https://www.gymshark.com/products/legacy-fitted-t-shirt",
  "type": "product",
  "lastmod": "2026-07-18",
  "domain": "www.gymshark.com",
  "fromSitemap": "https://www.gymshark.com/sitemap_products_1.xml"
}
```

A free per-site summary row:

```json
{
  "rowType": "summary",
  "domain": "www.gymshark.com",
  "total": 8964,
  "product": 8402,
  "category": 221,
  "blog": 338,
  "page": 3,
  "other": 0,
  "sitemapsRead": 12,
  "sitemapsSkipped": 0,
  "source": "robots-sitemap"
}
```

The Output tab provides separate **URLs** and **Per-site summary** views. The same dataset is available through the Apify API and integrations, and can be scheduled for recurring sitemap inventories.

### Pricing

Sitemap Intelligence uses transparent pay-per-event pricing with platform usage included:

| Event | Price |
|---|---:|
| Actor start at the default memory | $0.01 |
| Each classified and delivered URL | $0.0015 |
| Per-site summary row | Free |

Typical costs at the default memory are about **$0.16 for 100 URLs**, **$1.51 for 1,000 URLs**, and **$15.01 for 10,000 URLs**. The exact total is controlled by your maximum charge and `maxUrls`. URLs that are filtered out, cannot be delivered, or are never produced are not charged.

### Reliability and responsible use

Requests to one host are spaced out, response bodies are capped, timeouts are bounded, and an HTTP `429` receives one backed-off retry. Different sites can be processed concurrently, but a run does not fan out aggressively against one host.

This Actor reports what a site publishes in its sitemap; it is not a live-page availability audit. A listed URL may later return an error, and an unlisted page may still exist. If a site exposes no readable sitemap, the log says so instead of presenting an empty dataset as proof that the site has no pages.

Use the Actor only on public sitemap data and follow the target site's terms, robots policy, and applicable laws.

### Automation and integrations

Use Apify schedules to take weekly or monthly URL inventories. Connect run results to webhooks and integrations, or retrieve them with the Apify API. Common workflows include SEO migration planning, catalog-size monitoring, content-type analysis, stale-content reviews, and feeding known URLs into downstream crawlers.

### FAQ

#### Does it crawl every web page?

No. It reads sitemap files and does not download every listed page. This keeps runs fast and lightweight.

#### Why is a URL classified as other?

The sitemap context and URL shape did not provide enough evidence for a reliable category. Returning `other` is safer than guessing.

#### Are summary rows charged?

No. The per-site summary counts URL rows already delivered and is written without the paid `url-extracted` event.

#### Can I process several sites in one run?

Yes. Add multiple entries to `startUrls`; `maxUrls` remains the hard cap across the whole run.

### Support

If a sitemap format is not handled correctly or you have a feature request, open an issue in the Actor's **Issues** tab and include the public sitemap URL plus the relevant run ID.

# Actor input Schema

## `startUrls` (type: `array`):

Site or store URLs. The homepage is enough — sitemaps are found from robots.txt and conventional sitemap locations.

## `maxUrls` (type: `integer`):

Hard cap for the whole run, across every site.

## `includeTypes` (type: `array`):

Leave empty to keep everything. Filtered-out URLs are never charged, so narrowing this is the cheapest way to get only what you need.

## `maxSitemapsPerSite` (type: `integer`):

A large site can split its catalog across dozens of sitemap files. Raise this to read deeper; each one is an extra request.

## `crawlerIdentity` (type: `string`):

How the crawler introduces itself. Leave blank for the default. A real contact address is the courteous choice for large or scheduled runs.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxUrls": 1000,
  "includeTypes": [],
  "maxSitemapsPerSite": 15
}
```

# Actor output Schema

## `results` (type: `string`):

The default dataset contains one classified row per published URL plus one free summary row per site. Use the dataset view selector to switch between URL details and summaries.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("egeusta/sitemap-intelligence").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://apify.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("egeusta/sitemap-intelligence").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ]
}' |
apify call egeusta/sitemap-intelligence --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,egeusta/sitemap-intelligence"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/smlBa8lVEgDyjt0gV/builds/GJeQ4srNn8IBcJn2X/openapi.json