# Sitemap Extractor + robots.txt & llms.txt AI Crawler Checker (`ai_solution/sitemap-robots-llms-extractor`) Actor

Extract every URL from a website's sitemaps (nested, gzipped) plus robots.txt rules, AI-crawler policy and llms.txt - one clean JSON dataset for SEO audits, RAG ingestion and AI agents.

- **URL**: https://apify.com/ai_solution/sitemap-robots-llms-extractor.md
- **Developed by:** [MB AISolution](https://apify.com/ai_solution) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor + robots.txt & llms.txt AI Crawler Checker

Get **every URL from a website's sitemaps** (nested indexes and `.gz` files included) together with its **robots.txt rules, AI-crawler policy (GPTBot, ClaudeBot, PerplexityBot ...) and llms.txt** - as one clean, stable JSON dataset. No page scraping, no browser, no proxies: only the public files that websites publish for crawlers. Built to be called by humans, workflows and AI agents.

### What it does

For each website you provide, the Actor:

1. Fetches `robots.txt`, parses all rules and the declared `Sitemap:` lines.
2. Discovers and crawls all sitemaps - sitemap index files, `.xml.gz`, plain-text sitemaps - and outputs one record per URL (`lastmod`, `changefreq`, `priority`, image count, source sitemap).
3. If no sitemap is declared, probes the common locations (`/sitemap.xml`, `/sitemap_index.xml`, `/wp-sitemap.xml` ...), honouring robots.txt.
4. Reads `/llms.txt` and `/llms-full.txt` (the emerging standard that tells AI models what a site is about) and parses title, summary, sections and links.
5. Reports, for 14 well-known AI crawlers, whether the site **blocks or allows** them.

### Use cases

- **RAG / LLM ingestion**: get the list of all pages (filtered by regex or `lastModifiedAfter`) before you fetch and embed them.
- **SEO and site audits**: sitemap coverage, stale `lastmod` dates, missing sitemaps or robots.txt, accidental `Disallow: /`.
- **AI-visibility / GEO audits**: which sites publish `llms.txt`, which block GPTBot or ClaudeBot.
- **Competitor and content monitoring**: schedule it daily with `lastModifiedAfter` to detect new pages.
- **AI agents**: a one-call tool that answers "what pages does this site have and may I crawl them?"

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | string\[] | required | Domains or URLs. Only the site root is used. |
| `includeSitemapUrls` | boolean | `true` | Output `sitemap_url` records. |
| `includeSiteSummary` | boolean | `true` | Output one `site_summary` record per site (robots + llms.txt + stats). |
| `maxUrlsPerSite` | integer | `1000` | Cap on URLs per site (max 5,000,000). |
| `maxItems` | integer | `1000` | Hard cap on billed results for the whole run. |
| `maxSitemapsPerSite` | integer | `50` | Cap on sitemap files fetched per site. |
| `urlIncludePattern` / `urlExcludePattern` | regex string | - | Filter URLs, e.g. `/blog/`. |
| `lastModifiedAfter` | ISO date | - | Only URLs with `lastmod` on or after this date. |
| `llmsTxtMaxChars` | integer | `100000` | Truncate stored llms.txt text. |
| `requestTimeoutSecs` | integer | `20` | Per-request timeout. |

Example:

```json
{
  "urls": ["example.com", "https://www.anthropic.com"],
  "maxUrlsPerSite": 500,
  "urlIncludePattern": "/blog/",
  "lastModifiedAfter": "2025-01-01"
}
```

### Output

Every record has `recordType`, `url` and `scrapedAt`. Three record types share one dataset (use the dataset views *Sitemap URLs*, *Site summaries*, *Errors*).

`sitemap_url`:

```json
{
  "recordType": "sitemap_url",
  "url": "https://www.sitemaps.org/protocol.html",
  "domain": "www.sitemaps.org",
  "lastmod": "2022-12-15",
  "changefreq": null,
  "priority": null,
  "imageCount": 0,
  "sourceSitemap": "https://www.sitemaps.org/sitemap.xml",
  "scrapedAt": "2026-10-01T19:15:02.353Z"
}
```

`site_summary` (abridged):

```json
{
  "recordType": "site_summary",
  "url": "https://llmstxt.org",
  "domain": "llmstxt.org",
  "robots": { "found": true, "blocksEntireSiteForAll": false, "declaredSitemaps": ["..."],
              "aiCrawlers": { "GPTBot": "default_allowed", "ClaudeBot": "blocked" } },
  "llmsTxt": { "found": true, "title": "llms.txt", "summary": "...", "linkCount": 3, "sections": [ ... ] },
  "llmsFullTxt": { "found": false },
  "sitemaps": { "fetched": 1, "failed": 0, "urlsExtracted": 2, "truncated": true },
  "scrapedAt": "2026-10-01T19:15:02.353Z"
}
```

AI-crawler values: `blocked`, `allowed` (explicit rules for that bot), `default_allowed`, `default_blocked` (inherited from `User-agent: *`).

`error` records (free): invalid input, unreachable sites, blocked targets - one bad site never stops the run.

### How much does it cost?

Pay per event: **0.001 USD per result** (about 1.00 USD per 1,000) plus a tiny start fee of 0.00005 USD per run (per GB of memory, this Actor uses 1 GB at most). One `sitemap_url` record or one `site_summary` record = 1 result. `error` records are free. Platform usage is included in the event price.

| Scenario | Results | Approx. cost |
|---|---|---|
| Audit 100 sites, summary only (`includeSitemapUrls: false`) | 100 | 0.10 USD |
| 5 sites with the default `maxUrlsPerSite` of 1,000 (+ 5 summaries) | up to 5,005 | about 5.01 USD |
| One large site, 10,000 URLs + 1 summary | 10,001 | about 10.00 USD |
| One run with default settings (`maxItems` 1,000) | max 1,000 | max about 1.00 USD |

**Cost control:** `maxItems` is a hard cap on billed results for the whole run (default 1,000), and the Actor also stops when your run's *maximum cost per run* setting is reached. Use `maxUrlsPerSite`, `urlIncludePattern` and `lastModifiedAfter` to pay only for the URLs you need.

### Why this Actor (vs. alternatives)

The Store has single-purpose tools: sitemap URL extractors, robots.txt checkers and llms.txt/AI-crawler auditors. This Actor puts the three in **one run and one dataset**:

| | This Actor | Sitemap-only extractors (e.g. `apify/sitemap-extractor`) | AI-crawler / llms.txt audit Actors |
|---|---|---|---|
| All URLs from nested / gzipped sitemaps | yes | yes | not their focus |
| robots.txt rules + AI-crawler block/allow for 14 bots | yes | no | yes |
| llms.txt / llms-full.txt parsed | yes | no | yes |
| HTTP status check of every URL | **no** | yes (apify/sitemap-extractor) | no |
| Page content / scores / fix suggestions | no | no | varies |

Pick this one when you need the URL list **and** the crawler/AI policy of the same sites in a single call. If you only need cheap bulk URL lists with status codes, a sitemap-only Actor can be the better fit. (Comparison based on the public Store listings, 2026-10-02.)

### Use it from n8n, Make, code and AI agents

**n8n / Make**: use the Apify node/module "Run Actor" (or "Run Actor and get dataset items") with the input above and read the dataset items.

**API (curl)**:

```bash
curl -X POST "https://api.apify.com/v2/acts/ai_solution~sitemap-robots-llms-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls":["example.com"],"maxUrlsPerSite":100}'
```

**MCP / AI agents**: add this Actor in the Apify MCP server (`https://mcp.apify.com`, tool `ai_solution/sitemap-robots-llms-extractor`). The input schema is fully described so an agent can call it without extra instructions.

### Scheduling and webhooks

1. In Apify Console open *Schedules*, create a schedule (for example `0 6 * * 1` = every Monday 06:00 UTC) and add this Actor with your input.
2. In the Actor's *Integrations* tab (or the schedule's run) add a **webhook** for the event `ACTOR.RUN.SUCCEEDED` pointing to your endpoint or an n8n/Make *Webhook* node. The default payload contains `resource.defaultDatasetId`.
3. Fetch the results: `GET https://api.apify.com/v2/datasets/<defaultDatasetId>/items?token=...`

To detect new pages, pass a fresh `lastModifiedAfter` date on each run (for example with an n8n expression and the *Run Actor* node, or via the API).

### FAQ and limitations

- **Does it scrape pages?** No. It only reads robots.txt, sitemaps and llms.txt.
- **Is robots.txt respected?** Yes: the Actor identifies itself with its own User-Agent, throttles requests per host (about 3/s max), and does not probe guessed sitemap locations that robots.txt disallows. Sitemaps explicitly declared in robots.txt are fetched, as search engines do.
- **JavaScript-generated or login-protected sitemaps** are not supported. Sites behind bot protection (Cloudflare challenge etc.) may return errors - the Actor does not try to bypass them.
- Private/localhost addresses are refused. Very large sitemaps are capped at 60 MB per file.
- `lastmod` values are returned as published by the site; they are often inaccurate.

### Changelog

- **1.0.4** - Lower price: 0.001 USD per result (was 0.0015 USD). No other changes.
- **1.0.3** - Clearer README (cost examples, comparison, scheduling), new Store title and description. No changes to input, output fields, behaviour or pricing.
- **1.0.2** - Output schema (dataset views in the Output tab).
- **1.0.0** - First release.

### Support / Issues

Found a bug or need a field added? Open a ticket in this Actor's **Issues** tab in Apify Console.

### Legal

Only publicly available files that sites publish for crawlers are fetched. No personal data is collected. You are responsible for using the output in line with the target sites' terms and applicable law.

# Actor input Schema

## `urls` (type: `array`):

Domains or URLs of the websites to analyse. A path is ignored - the site root is used. Examples: example.com, https://www.example.org.

## `includeSitemapUrls` (type: `boolean`):

Output one record (recordType = sitemap_url) per URL found in the site's sitemaps, with lastmod, changefreq and priority.

## `includeSiteSummary` (type: `boolean`):

Output one summary record (recordType = site_summary) per site: robots.txt rules, AI-crawler policy (GPTBot, ClaudeBot, ...), declared sitemaps, llms.txt title/summary/links and sitemap statistics.

## `maxUrlsPerSite` (type: `integer`):

Stop extracting sitemap URLs for a site after this many URLs. Large sites can have millions of URLs - keep this low unless you need them all. Each URL is one billed result.

## `maxItems` (type: `integer`):

Hard cap on billed results (URL records + summary records) for the whole run, across all sites. Use it to control cost.

## `maxSitemapsPerSite` (type: `integer`):

Maximum number of sitemap files (including nested index entries) fetched per site.

## `urlIncludePattern` (type: `string`):

Optional JavaScript regular expression. Only URLs that match are output. Example: /blog/ or .pdf$

## `urlExcludePattern` (type: `string`):

Optional JavaScript regular expression. URLs that match are skipped. Example: /tag/|/category/

## `lastModifiedAfter` (type: `string`):

Optional ISO date, e.g. 2025-01-31. Only URLs whose sitemap lastmod is on or after this date are output (URLs without lastmod are skipped).

## `llmsTxtMaxChars` (type: `integer`):

The full llms.txt text is included in the summary record, truncated to this many characters.

## `requestTimeoutSecs` (type: `integer`):

Timeout for each HTTP request.

## Actor input object example

```json
{
  "urls": [
    "example.com",
    "https://www.anthropic.com"
  ],
  "includeSitemapUrls": true,
  "includeSiteSummary": true,
  "maxUrlsPerSite": 1000,
  "maxItems": 1000,
  "maxSitemapsPerSite": 50,
  "urlIncludePattern": "/blog/",
  "urlExcludePattern": "/tag/|/category/",
  "lastModifiedAfter": "2025-01-31",
  "llmsTxtMaxChars": 100000,
  "requestTimeoutSecs": 20
}
```

# Actor output Schema

## `results` (type: `string`):

All records in the default dataset.

## `urls` (type: `string`):

One row per sitemap URL

## `sites` (type: `string`):

robots.txt, AI-crawler policy and llms.txt per site

## `errors` (type: `string`):

Errors view of the default dataset.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.sitemaps.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("ai_solution/sitemap-robots-llms-extractor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://www.sitemaps.org"] }

# Run the Actor and wait for it to finish
run = client.actor("ai_solution/sitemap-robots-llms-extractor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.sitemaps.org"
  ]
}' |
apify call ai_solution/sitemap-robots-llms-extractor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ai_solution/sitemap-robots-llms-extractor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/v65ogaDlRxRlW97CW/builds/06TDgdyeduE8xp3uz/openapi.json