# Sitemap Extractor + robots.txt & llms.txt Checker for AI (`tinplatelabs/site-discovery`) Actor

Extract every URL from a website's sitemaps (indexes, gzip, nested) and check robots.txt and llms.txt: which AI crawlers (GPTBot, ClaudeBot, Google-Extended) may read the site, with the deciding rule. For SEO audits, crawler seeding and AI agents. $0.002/site.

- **URL**: https://apify.com/tinplatelabs/site-discovery.md
- **Developed by:** [tinplatelabs](https://apify.com/tinplatelabs) (community)
- **Categories:** SEO tools, Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Sitemap Extractor + robots.txt and llms.txt Checker for AI Crawlers

**Give it domains and get every URL from their sitemaps (indexes, gzip, nested), the robots.txt verdict for each
AI and search crawler (GPTBot, ClaudeBot, Google-Extended, PerplexityBot...) with the rule that decided, and the
site's llms.txt, as one JSON record per site. $0.002 per site.**

For SEO and AI-visibility audits, crawler and scraper builders who need a URL list without crawling, and AI agents
(Claude, ChatGPT, Cursor and others through the [Apify MCP server](https://mcp.apify.com)) that must check whether
they may read a site and where its docs are.

### Use cases

- **Sitemap URL extraction:** a site's full URL inventory with `lastmod`, filtered by path prefix or regex.
- **AI crawler access audit:** which AI bots a site blocks or allows, and the exact robots.txt line that decides.
- **llms.txt check:** whether a site publishes llms.txt / llms-full.txt, parsed into sections and links.
- **Crawler seeding and change monitoring:** feed the URL list to a scraper; compare `lastmod` between runs.
- **"May I fetch this URL?":** check mode tests any URLs against robots.txt for the user agents you name.

### Quick start

This input (the Actor's example) analyzes two sites in about 5 seconds for about $0.006:

```json
{"sites": ["nytimes.com", "developers.cloudflare.com"], "maxUrls": 1000}
```

One dataset item per site (abridged, real output from 2026-10-24):

```json
{
  "input": "nytimes.com",
  "site": "https://nytimes.com",
  "status": "ok",
  "crawlerAccess": {
    "GPTBot": {"allowed": false, "group": "GPTBot", "rule": "Disallow: /", "named": true, "reason": "longest matching rule is 'Disallow: /'"},
    "Googlebot": {"allowed": true, "group": "*, Googlebot", "rule": null, "named": false, "reason": "no rule matches"}
  },
  "sitemaps": {"source": "robots.txt", "count": 8, "complete": false, "incompleteReason": "maxUrls (1,000) reached", "returnedUrls": 1000},
  "urls": [{"loc": "https://www.nytimes.com/athletic/...", "lastmod": "2026-10-24T16:20:30Z"}, "..."],
  "llmsTxt": {"present": false, "httpStatus": 404, "note": "Not published (HTTP 404)."}
}
```

For `developers.cloudflare.com` the same record shows every crawler allowed and `llmsTxt.present: true` with the
file's title, sections and links.

### Pricing in plain words

You pay **$0.002 per site** analyzed plus **$0.001 per 1,000 URLs** returned. An AI-access and llms.txt audit of
100 sites with `includeUrls: false` costs $0.20; a site with 25,000 sitemap URLs costs $0.027. Sites that don't
exist or never answer cost nothing, and `maxUrls` caps the cost per site. Full table under
[Pricing](#pricing-pay-per-event).

### Why this one

- **It does not give up.** Sitemaps in the wild are messy. This Actor reads gzip files whatever their
  Content-Type says, double-gzipped files, BOMs, missing or wrong XML namespaces, unescaped `&` in URLs, files cut
  off mid-way, plain-text sitemaps, RSS and Atom feeds, redirects across hosts, and HTML "not found" pages served
  with status 200. Transient errors (429, 5xx, timeouts) are retried with backoff and `Retry-After`.
- **Partial beats failed.** One broken sitemap gets its own error entry with the reason and a fix; the rest of the
  site is still returned. Time and size limits return what was read so far, marked `complete: false` with the reason.
- **Huge sites are fine.** Sitemaps are parsed as a stream (no 50 MB file held in memory), indexes are expanded
  recursively without loops, and URLs are deduplicated across all sitemaps.
- **robots.txt done properly.** RFC 9309 matching (exact product-token groups, longest match wins, `Allow` wins ties,
  `*` and `$` wildcards), lenient parsing of real-world mistakes (each reported as a warning), and the RFC rules for
  missing (4xx = allow all) and broken (5xx = disallow all) robots.txt.
- **Agent-shaped output.** Every verdict says which group applied and which rule decided, so an agent can explain it.
- **Polite and honest.** Only files that sites publish for machines are fetched: `/robots.txt`, sitemaps,
  `/llms.txt`, `/llms-full.txt`. No page is ever crawled. Requests identify as
  `TinplateSiteDiscovery/1.0 (+https://tinplatelabs.github.io)`, are paced per host (Crawl-delay honoured up to 3 s),
  and sitemaps and llms files are skipped where robots.txt disallows them (sitemaps that robots.txt itself declares
  count as an invitation).

### Input

| Field | Type | Default | What it does |
|---|---|---|---|
| `sites` | array of strings | required in discover mode | Domains or URLs on the site. HTTPS first, then HTTP. Max 1,000. |
| `mode` | `discover` | `check` | `discover` | Full site records, or robots.txt verdicts for `urlsToCheck`. |
| `urlsToCheck` | array of strings | | Check mode: full URLs to test. Max 10,000. |
| `userAgents` | array of strings | AI + search crawlers | Product tokens to evaluate: `GPTBot`, `ClaudeBot`, `Google-Extended`, `*`... |
| `includeUrls` | boolean | `true` | Return the URL list. `false` = counts only (no URL fees). |
| `maxUrls` | integer 1–200,000 | `10000` | Matching URLs returned per site at most. |
| `urlPrefixes` | array of strings | | Keep URLs starting with these (`/blog/` matches the path, a full URL matches the URL). |
| `urlRegex` / `excludeUrlRegex` | string | | Keep / drop URLs matching a Python regex. |
| `maxSitemaps` | integer 1–2,000 | `200` | Sitemap files read per site. |
| `maxConcurrency` | integer 1–20 | `5` | Sites analyzed in parallel. |
| `siteTimeoutSecs` | integer 30–1,800 | `300` | After this, URLs found so far are returned (`complete: false`). |

```json
{
  "sites": ["python.org", "developers.cloudflare.com"],
  "userAgents": ["GPTBot", "ClaudeBot", "Google-Extended", "*"],
  "maxUrls": 5000,
  "urlPrefixes": ["/workers/"]
}
```

Check mode:

```json
{"mode": "check", "urlsToCheck": ["https://example.com/blog/post", "https://example.com/admin/"], "userAgents": ["GPTBot", "Googlebot"]}
```

### Output

One dataset item per site (shortened):

```json
{
  "input": "example.com",
  "site": "https://example.com",
  "status": "ok",
  "robotsTxt": {
    "url": "https://example.com/robots.txt", "httpStatus": 200, "status": "ok", "sizeBytes": 412,
    "groups": [
      {"userAgents": ["GPTBot", "CCBot"], "allow": [], "disallow": ["/"], "crawlDelay": null},
      {"userAgents": ["*"], "allow": ["/admin/public/"], "disallow": ["/admin/"], "crawlDelay": 2.0}
    ],
    "sitemaps": ["https://example.com/sitemap_index.xml"],
    "otherDirectives": {}, "warnings": []
  },
  "crawlerAccess": {
    "GPTBot": {"allowed": false, "group": "GPTBot, CCBot", "rule": "Disallow: /", "crawlDelay": null, "named": true, "reason": "longest matching rule is 'Disallow: /'"},
    "ClaudeBot": {"allowed": true, "group": "*", "rule": null, "crawlDelay": 2.0, "named": false, "reason": "no rule matches"}
  },
  "sitemaps": {
    "source": "robots.txt", "count": 3, "failed": 1, "complete": true, "incompleteReason": null,
    "urlsScanned": 1207, "uniqueUrls": 1180, "duplicateUrls": 27, "invalidEntries": 0,
    "matchedUrls": 1180, "returnedUrls": 1180,
    "items": [
      {"url": "https://example.com/sitemap_index.xml", "type": "sitemapindex", "httpStatus": 200, "gzip": false, "urls": 0, "sitemaps": 2, "depth": 0, "error": null},
      {"url": "https://example.com/post-sitemap.xml.gz", "type": "urlset", "httpStatus": 200, "gzip": true, "urls": 1207, "sitemaps": 0, "depth": 1, "error": null},
      {"url": "https://example.com/old-sitemap.xml", "type": null, "httpStatus": 404, "urls": 0, "sitemaps": 0, "depth": 1,
       "error": {"code": "HTTP_ERROR", "message": "HTTP 404. The sitemap URL listed for this site does not work; the site owner needs to fix or remove it."}}
    ]
  },
  "urls": [
    {"loc": "https://example.com/blog/hello", "lastmod": "2026-09-30", "changefreq": "weekly", "priority": 0.8}
  ],
  "llmsTxt": {
    "url": "https://example.com/llms.txt", "present": true, "httpStatus": 200, "sizeBytes": 1834,
    "title": "Example", "summary": "Docs for the Example API.", "details": null, "validFormat": true, "linkCount": 12,
    "sections": [{"title": "Docs", "optional": false, "notes": null,
                  "links": [{"title": "Quick start", "url": "https://example.com/start.md", "description": "install and first call"}]}],
    "content": "# Example\n\n> Docs for the Example API.\n..."
  },
  "llmsFullTxt": {"url": "https://example.com/llms-full.txt", "present": false, "httpStatus": 404, "note": "Not published (HTTP 404)."},
  "warnings": []
}
```

Check mode returns one item per URL:

```json
{"input": "https://example.com/admin/", "url": "https://example.com/admin/", "status": "ok",
 "robotsTxtUrl": "https://example.com/robots.txt", "robotsTxtStatus": "ok",
 "results": {"GPTBot": {"allowed": false, "group": "GPTBot, CCBot", "rule": "Disallow: /", "named": true, "crawlDelay": null, "reason": "longest matching rule is 'Disallow: /'"}}}
```

A site that cannot be analyzed gives an error item, and the run carries on:

```json
{"input": "no-such-domain.example", "status": "error",
 "error": {"code": "SITE_NOT_FOUND", "message": "The host name 'no-such-domain.example' does not resolve in DNS (...). Check the spelling of the domain; the site may not exist or may have expired."}}
```

- `robotsTxt.status`: `ok`, `not_found` (4xx: everything allowed), `unreachable` (5xx or no answer: everything
  disallowed, per RFC 9309), `html` (an HTML page was served: no rules).
- `sitemaps.source`: `robots.txt` (declared), `common-locations` (found at `/sitemap.xml`, `/sitemap_index.xml` or
  `/wp-sitemap.xml`) or `none`.
- Site error codes: `INVALID_SITE`, `INVALID_URL`, `SITE_NOT_FOUND`, `SITE_UNREACHABLE`, `PRIVATE_ADDRESS`, `TIMEOUT`,
  `SPENDING_LIMIT_REACHED`, `INTERNAL_ERROR`. Sitemap error codes: `HTTP_ERROR`, `NOT_A_SITEMAP`, `PARSE_ERROR`,
  `TOO_LARGE`, `FETCH_FAILED`, `BLOCKED_BY_ROBOTS`. Every message says what to do next.
- If a site's URL list is too big for one dataset item (~8 MB), it is saved in the run's key-value store and the
  item gives `urlsKeyValueStoreKey` and `urlsUrl`.

### Pricing (pay per event)

| Event | Price | When |
|---|---|---|
| `site-analyzed` | **$0.002** | Each site analyzed (robots.txt, AI access, sitemaps, llms.txt). In check mode, each site whose robots.txt was read. |
| `sitemap-urls-1000` | **$0.001** | Each started block of 1,000 URLs returned. |
| Actor start | set by Apify | Per run |

Examples: an AI-access and llms.txt audit of 100 sites with `includeUrls: false` costs $0.20. A site with 25,000
URLs in its sitemaps costs $0.002 + $0.025 = $0.027. Sites that do not exist or never answer are not charged.
When the run's maximum total charge is reached, URL lists are cut to what it covers and the remaining sites are
marked `SPENDING_LIMIT_REACHED`.

### Limits

- 1,000 sites (or 10,000 URLs to check) per run; up to 200,000 returned URLs per site; up to 1,000,000 URLs counted
  per site with `includeUrls: false`.
- Up to 2,000 sitemap files per site, nested up to 6 levels; 100 MB per uncompressed sitemap; 120 s per sitemap download.
- Only public addresses are fetched. Redirects: up to 5 per request.
- User-agent groups are matched by exact product token (as RFC 9309 says); Google's fallback from tokens such as
  `Googlebot-Image` to `Googlebot` is not applied.
- Image, video and news sitemap extensions are not returned (their page URLs are).
- llms.txt content is returned up to 100,000 characters; llms-full.txt is summarized (size, title, headings), not returned.

### Use from an AI agent (MCP)

Add the Apify MCP server to your agent and enable this Actor. Typical calls: `{"sites": ["example.com"],
"includeUrls": false}` to see if AI crawlers are allowed and whether an llms.txt exists, or `{"sites": ["example.com"], "urlPrefixes": ["/docs/"]}` to get the docs URLs to read next.

### Open-source licenses

All dependencies are permissively licensed. CI fails on any copyleft dependency.

| Component | License | Role |
|---|---|---|
| [httpx](https://github.com/encode/httpx) 0.28.1, httpcore, h11, anyio, idna | BSD-3-Clause / MIT | HTTP |
| [lxml](https://lxml.de) 6.1.3 (libxml2) | BSD-3-Clause / MIT | Streaming, error-tolerant XML parsing |
| apify 4.1.0, apify-client, crawlee and their dependencies (pydantic, cryptography, websockets, ...) | Apache-2.0 / MIT / BSD | Apify SDK |
| certifi | MPL-2.0 (used unmodified) | CA certificates |

The robots.txt parser and matcher are our own (RFC 9309). Test-only: pytest (MIT), Protego (BSD-3-Clause) as a
reference to compare robots.txt verdicts against.

***

Made by Tinplate Labs. This Actor is maintained by an AI; there is no support desk, so every error message is
written to tell you exactly what to change.

# Actor input Schema

## `sites` (type: `array`):

Sites to analyze, one per line (every sitemap URL, robots.txt access for AI crawlers, llms.txt): a domain (<code>example.com</code>) or any URL on the site (<code>https://www.example.com/blog/</code>). Without a scheme, HTTPS is tried first, then HTTP. Duplicates are analyzed once. Max 1,000 per run. Used when mode is <b>discover</b>.

## `mode` (type: `string`):

<b>discover</b>: one full record per site (robots.txt, AI-crawler access, sitemaps and URLs, llms.txt). <b>check</b>: one record per URL in <i>URLs to check</i> saying whether each user agent may fetch it according to robots.txt, and which rule decided.

## `urlsToCheck` (type: `array`):

Full URLs to test against their site's robots.txt, e.g. <code>https://example.com/blog/post-1</code>. Each site's robots.txt is fetched once. Max 10,000 per run. Used when mode is <b>check</b>.

## `userAgents` (type: `array`):

Robots.txt product tokens to evaluate (case-insensitive), e.g. <code>GPTBot</code>, <code>ClaudeBot</code>, <code>Google-Extended</code>, <code>\*</code>. Leave empty for the default list of AI and search crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, Googlebot, Bingbot, PerplexityBot, Perplexity-User, CCBot, Applebot-Extended, Bytespider, meta-externalagent, Amazonbot, cohere-ai and \*.

## `includeUrls` (type: `boolean`):

Return the URLs found in the sitemaps (with lastmod, changefreq, priority). Turn off to get counts only (up to 1,000,000 URLs per site), which costs no URL fees.

## `maxUrls` (type: `integer`):

Stop after this many matching URLs per site; the record is then marked incomplete. URLs are billed per 1,000 returned, so this also caps the cost per site.

## `urlPrefixes` (type: `array`):

Keep only URLs that start with one of these. A value starting with <code>/</code> is matched against the URL path (<code>/blog/</code>); a full URL is matched against the whole URL (<code>https://shop.example.com/</code>).

## `urlRegex` (type: `string`):

Keep only URLs where this Python regular expression matches anywhere, e.g. <code>/(docs|guides)/</code>.

## `excludeUrlRegex` (type: `string`):

Drop URLs where this Python regular expression matches anywhere, e.g. <code>.(pdf|jpg)$|/tag/</code>.

## `maxSitemaps` (type: `integer`):

Read at most this many sitemap files per site (indexes count too). Very large sites can have thousands; the record says how many were left unread.

## `maxConcurrency` (type: `integer`):

How many sites are analyzed at the same time. Requests to any one host are always paced (Crawl-delay honoured up to 3 s).

## `siteTimeoutSecs` (type: `integer`):

When reading sitemaps takes longer than this, the URLs found so far are returned and the record is marked incomplete (still a success).

## Actor input object example

```json
{
  "sites": [
    "nytimes.com",
    "developers.cloudflare.com"
  ],
  "mode": "discover",
  "includeUrls": true,
  "maxUrls": 10000,
  "maxSitemaps": 200,
  "maxConcurrency": 5,
  "siteTimeoutSecs": 300
}
```

# Actor output Schema

## `results` (type: `string`):

All site records, URL verdicts and per-item error records as JSON.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "nytimes.com",
        "developers.cloudflare.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tinplatelabs/site-discovery").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sites": [
        "nytimes.com",
        "developers.cloudflare.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("tinplatelabs/site-discovery").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "nytimes.com",
    "developers.cloudflare.com"
  ]
}' |
apify call tinplatelabs/site-discovery --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tinplatelabs/site-discovery"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/9mhG5SJzrYmDDY5BW/builds/aaQjYG3wP2BGqIsNg/openapi.json