# Robots.txt, Sitemap & AI Crawler Access Analyzer (`isnard/robots-sitemap-ai-analyzer`) Actor

Check which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and more) and search engines a site's robots.txt allows or blocks, whether it has llms.txt, and how many URLs its sitemaps list. Plain-English recommendations per site.

- **URL**: https://apify.com/isnard/robots-sitemap-ai-analyzer.md
- **Developed by:** [Matias Isnard](https://apify.com/isnard) (community)
- **Categories:** SEO tools, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 site analyzeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Robots.txt, Sitemap & AI Crawler Access Analyzer

Give it a list of domains. For each site you get:

- which AI crawlers its robots.txt allows or blocks at `/` (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 10 more), and at any extra paths you list
- the same for Googlebot and Bingbot
- the parsed robots.txt: groups, Allow and Disallow rules, Crawl-delay, Sitemap lines, and lines crawlers cannot understand
- whether `/llms.txt` exists
- the sitemaps: files read, number of URLs, oldest and newest `<lastmod>`, invalid entries, URLs on other hosts
- a short list of findings and suggestions in plain English

Useful for SEO audits, for publishers who want to check that their AI opt-out does what they think, and for anyone who collects data and wants to know what a site permits.

### Input

```json
{
    "domains": ["nytimes.com", "github.com", "wordpress.org"],
    "paths": ["/blog/", "/search?q=test"],
    "maxSitemapFiles": 10,
    "maxSitemapUrls": 50000,
    "timeoutSecs": 15
}
```

- `domains` (required): domains or URLs. Only the hostname is used. robots.txt belongs to one host, so `www.example.com` and `example.com` are different sites, but redirects are followed (nytimes.com is answered by www.nytimes.com). Duplicates are checked once, invalid lines are skipped with a warning.
- `paths` (optional): extra paths to check for every crawler, besides `/`. A query string is allowed.
- `maxSitemapFiles` (default 10): sitemap files downloaded per site, index files included.
- `maxSitemapUrls` (default 50,000): stop counting sitemap URLs at this number.
- `timeoutSecs` (default 15): limit for each request.

### What the Actor requests

Per site, in this order and one at a time:

1. `https://<host>/robots.txt` (plain http only if https cannot connect)
2. `/llms.txt`
3. the sitemaps listed in robots.txt, or `/sitemap.xml` when robots.txt lists none, then the child sitemaps of index files, up to `maxSitemapFiles`

With the defaults that is at most 12 requests per site. No HTML pages are fetched. When robots.txt gets no answer at all (DNS error, connection refused), the other requests are skipped. Requests identify themselves with the User-Agent `Mozilla/5.0 (compatible; RobotsSitemapAIAnalyzer/1.0; +https://apify.com/isnard/robots-sitemap-ai-analyzer)`.

### How robots.txt is read

The rules follow RFC 9309, which is also what Google and Bing document:

- A crawler uses every group that names its user-agent token (case-insensitive, `/version` ignored), merged. Only when no group names it does it use the `User-agent: *` group. So a `GPTBot` group replaces the `*` rules for GPTBot completely.
- The most specific rule wins: the longest matching path pattern. When an Allow and a Disallow are equally long, Allow wins.
- `*` matches any characters and a `$` at the end anchors the end of the URL.
- An empty `Disallow:` blocks nothing.
- A robots.txt answering 4xx (usually 404) means no restrictions: everything is `allowed`.
- A robots.txt answering 5xx, or not answering, gives `unknown`. Google treats that case as "do not crawl" for a while.

Each crawler entry looks like this:

```json
{
    "bot": "GPTBot",
    "operator": "OpenAI",
    "purpose": "training",
    "matchedGroup": "GPTBot",
    "root": "blocked",
    "rootRule": "disallow: /",
    "disallowRuleCount": 1,
    "paths": { "/blog/": "blocked", "/search?q=test": "blocked" }
}
```

- `matchedGroup`: the token of the group that applies, `*` when the `*` group is used, null when there is no group for it at all (everything allowed).
- `root`: `allowed`, `blocked` or `unknown` for the path `/`. A site can be `allowed` at `/` and still block most sections; `disallowRuleCount` and `paths` show that.
- `rootRule`: the rule that decided `/`, null when no rule matched.

### Crawlers checked

| Token | Operator | Purpose |
| --- | --- | --- |
| GPTBot | OpenAI | training |
| OAI-SearchBot | OpenAI | search (ChatGPT search results) |
| ChatGPT-User | OpenAI | user (fetches a page when a user asks) |
| ClaudeBot | Anthropic | training |
| Claude-SearchBot | Anthropic | search |
| Claude-User | Anthropic | user |
| Claude-Web | Anthropic | legacy, no longer used |
| anthropic-ai | Anthropic | legacy, no longer used |
| Google-Extended | Google | training (Gemini). A control token only, it does not crawl and does not affect Google Search |
| PerplexityBot | Perplexity | search |
| CCBot | Common Crawl | training (the dataset is widely used to train models) |
| Bytespider | ByteDance | training |
| Applebot-Extended | Apple | training. A control token only, Applebot does the crawling |
| Meta-ExternalAgent | Meta | training |
| Amazonbot | Amazon | training and product use |
| Googlebot | Google | search engine |
| Bingbot | Microsoft | search engine |

Sources for the tokens, checked in October 2026:

- OpenAI: https://developers.openai.com/api/docs/bots
- Anthropic: https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler (lists ClaudeBot, Claude-User and Claude-SearchBot; anthropic-ai and Claude-Web are not listed there any more)
- Google: https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers
- Perplexity: https://docs.perplexity.ai/guides/bots
- Common Crawl: https://commoncrawl.org/ccbot
- Apple: https://support.apple.com/en-us/119829
- Meta: https://developers.facebook.com/docs/sharing/webmasters/web-crawlers/
- Amazon: https://developer.amazon.com/amazonbot
- Bing: https://www.bing.com/webmasters/help/?topicid=8c184ec0
- Bytespider: ByteDance publishes no documentation. The token `Bytespider` is the one seen in server logs and used in robots.txt files.

robots.txt is a request, not a lock. This Actor tells you what the file says, not what each crawler really does. OpenAI and Perplexity say their user-triggered fetchers (ChatGPT-User, Perplexity-User) may not follow robots.txt, and Bytespider has been reported to ignore it.

### Sitemaps

- XML `<urlset>`, XML `<sitemapindex>`, plain text (one URL per line) and gzip files are read. Gzip is detected by content, so `.xml.gz` files and servers that send gzip without saying so both work.
- Index files are followed breadth first: first all sitemaps listed in robots.txt, then their children. The walk stops at `maxSitemapFiles` files or `maxSitemapUrls` URLs, and `truncated` is set to `true`. A large site such as nytimes.com has thousands of sitemap files, so its count is always a lower bound.
- Invalid entries: `<loc>` missing, relative, not http(s), longer than 2,048 characters, or a `<lastmod>` that is not a W3C date.
- URLs on other hosts are compared without a leading `www.`. Search engines ignore those unless both hosts are verified by the same owner.
- `lastmodMin` and `lastmodMax` cover the page entries, not the dates in index files.
- Each file is read up to 50 MB uncompressed, the limit of the sitemap protocol.

### Output

One dataset item per site. Shortened example (real output for github.com, full examples in `ejemplos/salida.json` in the source):

```json
{
    "input": "github.com",
    "host": "github.com",
    "checkedAt": "2026-10-29T09:41:02+00:00",
    "summary": {
        "robotsTxt": "found",
        "aiBlocked": ["Bytespider"],
        "aiAllowed": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "..."],
        "aiTrainingBlockedCount": 1,
        "aiTrainingTotal": 8,
        "aiNamedInRobots": ["GPTBot", "OAI-SearchBot", "ClaudeBot", "anthropic-ai", "PerplexityBot", "Bytespider"],
        "googlebot": "allowed",
        "bingbot": "allowed",
        "llmsTxtFound": true,
        "sitemapFound": true,
        "sitemapListedInRobots": true,
        "sitemapUrls": 376,
        "sitemapUrlsTruncated": false,
        "reachable": true
    },
    "recommendations": [
        "robots.txt blocks some AI training crawlers by name (Bytespider) but these are not named and still allowed: Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent, Amazonbot. If the goal is to opt out of AI training, add groups for them too.",
        "The sitemap URLs have no <lastmod>. Adding it helps search engines recrawl changed pages first."
    ],
    "aiCrawlers": ["one entry per crawler, see above"],
    "searchCrawlers": ["Googlebot and Bingbot"],
    "robots": { "url": "https://github.com/robots.txt", "status": 200, "state": "found", "groups": ["..."], "sitemaps": ["..."], "invalidLineCount": 0 },
    "llmsTxt": { "url": "https://github.com/llms.txt", "found": true, "status": 200, "sizeBytes": 28658 },
    "sitemaps": { "source": "robots.txt", "filesFetched": 1, "totalUrls": 376, "truncated": false, "invalidEntries": 0, "otherHostUrls": 0, "lastmodMin": null, "lastmodMax": null, "files": ["..."] },
    "errors": []
}
```

`summary.robotsTxt` is `found`, `missing` (4xx) or `unreachable`. `llmsTxt.found` is `true` only for a 200 answer that is not an HTML page, because many sites answer every unknown path with their homepage.

A site that does not answer still gets an item, with `reachable: false` and the reason in `errors`. The Actor does not fail because of one bad site.

### Pricing

Pay per event: `site-analyzed`, once per site, only when robots.txt or a sitemap answered. A domain that does not exist or does not answer is not charged.

### Limitations

- Paths are compared as given. Percent-encoded and plain characters (`/%7Ea` and `/~a`) are not treated as equal.
- User-agent matching is by exact token. Google's fallback from a specific crawler to a general one (Googlebot-News to Googlebot) is not modeled, because only the main tokens are checked.
- Meta tags (`noai`, `noimageai`), `X-Robots-Tag` headers and TDM reservation files are not read, only robots.txt.
- llms.txt is only checked for presence, not validated.
- Sites behind bot protection may answer robots.txt with 403. That is reported as `missing`, which is what crawlers would assume too. A 429 (rate limited) is `unreachable`, as Google treats it.
- Requests come from Apify's servers. A site that serves a different robots.txt by country or by user-agent is seen as Apify sees it.

### Run locally

```bash
python -m venv .venv
.venv/Scripts/pip install -r requirements.txt   # on Linux/macOS: .venv/bin/pip
## put input in storage/key_value_stores/default/INPUT.json
.venv/Scripts/python -m my_actor
python -m unittest discover -s tests -t .
```

Author: Matias Isnard.

# Actor input Schema

## `domains` (type: `array`):

Domains or URLs, one per line. Only the hostname is used (https://www.example.com/page becomes www.example.com). robots.txt belongs to one host, so www.example.com and example.com are different sites, but redirects are followed. Duplicates are checked once.

## `paths` (type: `array`):

Optional. Besides /, report for every crawler whether these paths are allowed, for example /blog/ or /search?q=x. Same paths for all sites.

## `maxSitemapFiles` (type: `integer`):

How many sitemap files to download per site, sitemap index files included. Child sitemaps of an index are followed until this limit.

## `maxSitemapUrls` (type: `integer`):

Stop counting sitemap URLs at this number. The result says when the count was cut.

## `timeoutSecs` (type: `integer`):

Maximum time for each request.

## Actor input object example

```json
{
  "domains": [
    "nytimes.com",
    "wordpress.org"
  ],
  "maxSitemapFiles": 10,
  "maxSitemapUrls": 50000,
  "timeoutSecs": 15
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "nytimes.com",
        "wordpress.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("isnard/robots-sitemap-ai-analyzer").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "nytimes.com",
        "wordpress.org",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("isnard/robots-sitemap-ai-analyzer").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "nytimes.com",
    "wordpress.org"
  ]
}' |
apify call isnard/robots-sitemap-ai-analyzer --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,isnard/robots-sitemap-ai-analyzer"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/nV5BGV1JQTj7KqzXY/builds/WFHanAn70fxcSQUXK/openapi.json