# AI Crawler Access Checker: GPTBot, ClaudeBot, robots.txt (`atalaya-ai/ai-crawler-access-checker`) Actor

Bulk-check which AI crawlers (GPTBot, ClaudeBot, PerplexityBot...) each domain allows or blocks in robots.txt. Pay per domain, EN/ES, no API keys.

- **URL**: https://apify.com/atalaya-ai/ai-crawler-access-checker.md
- **Developed by:** [Atalaya AI](https://apify.com/atalaya-ai) (community)
- **Categories:** SEO tools, Agents, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 domain checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler Access Checker: GPTBot, ClaudeBot and robots.txt, in bulk

Paste a list of domains and get, for each one, which AI crawlers (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended and 15 more) its `robots.txt` allows or blocks, **the exact rule and group that decided it**, whether it has `llms.txt` / `llms-full.txt`, and a one-line verdict with the top fix, in English or Spanish. One cheap, pay-per-domain check you can run on 10, 500 or 5,000 domains. No browser, no API keys.

Built for people who check many sites: SEO and GEO agencies auditing client portfolios, in-house SEOs benchmarking competitors, brands and publishers checking which AI bots they let in, and AI agents that need a factual answer before giving advice.

> Need the full picture of one site (page metadata, schema, scored fixes)? Use the companion Actor [`atalaya-ai/ai-search-readiness-auditor`](https://apify.com/atalaya-ai/ai-search-readiness-auditor). This one only answers "who can crawl it?", fast and for many domains.

### What does it check for each domain?

- **robots.txt fetch status:** found, missing (404), HTML served instead, access denied (401/403), rate limited (429), server error (5xx) or unreachable. Status codes are interpreted per [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309).
- **Allowed or blocked for every AI crawler**, for `/` and for any extra paths you list, with the **matched rule** (for example `Disallow: /`), its **line number** and the **group** that applied (the crawler's own token, a vendor-documented fallback, or `*`).
- **Crawler type:** training, AI search / retrieval, or user-triggered fetcher. Blocking each has a different effect (see the FAQ).
- **Counts per type:** how many training and search crawlers are blocked or allowed.
- **`/llms.txt` and `/llms-full.txt`:** present or not (presence only).
- **Sitemaps declared** in robots.txt.
- **Hints, only when visible in the text:** Cloudflare's managed robots.txt block and `Content-Signal` values. They never change the verdicts.
- **Verdict and top fix** in one line each (`en` or `es`).

### Which AI crawlers are covered?

20 crawlers, grouped by what blocking them does. Tokens are matched case-insensitively. Source URLs and the last verification date (`crawlerListVerifiedOn`, currently 2026-10-19) are kept in the Actor's crawler list; vendors change their bots, so treat the list as a snapshot.

| Type | What blocking it does | Tokens |
|---|---|---|
| **Training** | Opts your content out of model training. Does not remove you from search. | `GPTBot` (OpenAI), `ClaudeBot` (Anthropic), `Google-Extended` (Gemini training and grounding), `Applebot-Extended`, `CCBot` (Common Crawl), `Meta-ExternalAgent`, `Amazonbot`, `Bytespider` (ByteDance; vendor documentation not found, so flagged unverified) |
| **Search / retrieval** | Removes you from that product's AI answers and links. | `OAI-SearchBot` (ChatGPT search), `Claude-SearchBot`, `PerplexityBot`, `Googlebot` (Google documents it as the control for AI Overviews / AI Mode), `Applebot` (falls back to the `Googlebot` group if not named), `Amzn-SearchBot`, `DuckAssistBot` |
| **User-triggered fetchers** | Stops fetches made because a user asked an assistant about your page. Several vendors say these may ignore robots.txt. | `ChatGPT-User`, `Claude-User`, `Perplexity-User`, `Meta-ExternalFetcher`, `Amzn-User` |

### What does the output look like?

One dataset item per domain (abbreviated; `sample-output.json` in the Actor source has 5 real, complete items):

```json
{
  "domain": "nytimes.com",
  "status": "checked",
  "robots": { "url": "https://nytimes.com/robots.txt", "outcome": "found", "httpStatus": 200, "directives": 273 },
  "posture": "search_partly_blocked",
  "verdict": "4 of 7 AI search crawlers are blocked or restricted (OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot); those products may not cite this site.",
  "topFix": "If you want to appear in their answers, allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot: remove the Disallow rule or add a \"User-agent: <bot>\" + \"Allow: /\" group.",
  "summary": { "totalChecked": 20, "trainingAllowed": 1, "trainingBlocked": 7, "searchAllowed": 3, "searchBlocked": 4, "userFetchAllowed": 1, "userFetchBlocked": 4 },
  "blockedCrawlers": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "..."],
  "crawlers": [
    { "token": "GPTBot", "category": "training", "status": "blocked", "rootAllowed": false,
      "matchedGroup": "GPTBot", "matchedRule": "Disallow: /", "matchedRuleLine": 233 },
    { "token": "OAI-SearchBot", "category": "search", "status": "blocked", "rootAllowed": false,
      "matchedGroup": "OAI-SearchBot", "matchedRule": "Disallow: /", "matchedRuleLine": 266 }
  ],
  "llmsTxt": { "present": false, "httpStatus": 404 },
  "llmsFullTxt": { "present": false, "httpStatus": 404 },
  "sitemap": { "declared": true, "count": 25 },
  "hints": { "cloudflareManagedContent": false, "contentSignals": null },
  "billableEvents": { "domain-checked": 1 }
}
```

The default **Overview** view shows one row per domain; the **Crawler matrix** view shows one row per domain and crawler. Export as JSON, CSV or Excel from the Output tab.

#### Output fields

| Field | Meaning |
|---|---|
| `status` | `checked` (verdicts included, billed), `inconclusive` (the server answered 401/403/429/5xx: no verdict, not billed), `unreachable` (DNS, TLS, timeout or private address: not billed), `error`, `invalid_input`, `skipped_free_plan`, `skipped_charge_limit` (not processed, not billed). `statusDetail` explains. |
| `robots` | `outcome`, `httpStatus`, size, number of directives, a note when the outcome is not a clean `found`. |
| `posture` | One label: `all_open`, `search_open_training_blocked`, `search_partly_blocked`, `search_blocked`, `all_blocked`, `restricted`, `no_robots_txt`, `inconclusive`. |
| `verdict`, `topFix` | One line each, in the language you chose. |
| `summary` | Counts: `training/search/userFetch` x `Allowed/Blocked/Partial`. |
| `crawlers[]` | Per crawler: `status` (`allowed`, `blocked`, `partial`), `rootAllowed`, `matchedGroup`, `matchedRule`, `matchedRuleLine`, `paths[]` for your extra paths, `disallowRules`, `crawlDelay`, vendor `note`. |
| `blockedCrawlers`, `partialCrawlers`, `allowedCrawlers` | The same result as flat lists of tokens. |
| `llmsTxt`, `llmsFullTxt` | `present` (true / false / null when not checked), `httpStatus`, and for `llms.txt` whether the first line is a `# Title`. |
| `sitemap` | `declared`, `count`, up to 10 URLs from `Sitemap:` lines in robots.txt. |
| `hints` | `cloudflareManagedContent`, `contentSignals`. |

`partial` means the result differs between the checked paths, or `/` is blocked but `Allow` rules open some sections.

A run also stores a `RUN-SUMMARY` record: counts, invalid and skipped inputs, unknown crawler names, events charged and the stop reason.

### How do I check hundreds of domains?

1. Open the Actor, paste your domains in **Domains to check** (one per line; URLs are fine, only the host is used; up to 5,000 per run).
2. Optionally pick a **subset of crawlers** (for example `GPTBot`, `ClaudeBot`, `PerplexityBot`) and **extra paths** such as `/blog/`.
3. Choose the **language** of the verdict and start.
4. Download the dataset, or call it through the API:

```bash
curl -X POST "https://api.apify.com/v2/acts/atalaya~ai-crawler-access-checker/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"domains":["example.com","example.org"],"crawlers":["GPTBot","ClaudeBot"],"language":"en"}'
```

| Input | Default | Description |
|---|---|---|
| `domains` (required) | 3 example domains | Domains or URLs, 1-5,000. Duplicate hosts are checked once; `example.com` and `www.example.com` are different hosts. |
| `crawlers` | all 20 | Subset by robots.txt token, case-insensitive. Unknown tokens are ignored and listed in `RUN-SUMMARY`. |
| `paths` | none | Up to 20 extra paths tested next to `/`. |
| `language` | `en` | `en` or `es`: language of `verdict`, `topFix`, `statusDetail`. Other fields are language-neutral. |
| `checkLlmsTxt` | on | Two small extra requests per domain. Same price either way. |
| `maxConcurrency` | 10 | Different domains checked in parallel (1-20). One request at a time per domain, always. |

### How is the result decided?

The Actor reads each domain's `/robots.txt` and applies [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309), the Robots Exclusion Protocol standard:

- The group naming the crawler's token wins (several groups for the same token are merged); if none names it, the `*` group applies. `Applebot` falls back to a `Googlebot` group because Apple documents that.
- The longest matching rule wins; `Allow` wins a tie. `*` and a trailing `$` work in patterns. A missing file (404) means no restrictions.
- **Deliberate difference:** if robots.txt answers 401, 403, 429 or 5xx, or the connection fails, you get **no verdict** instead of "everything allowed" or "everything blocked". A 403 is very often a firewall rule against unknown bots, so the answer a real AI crawler gets may be different. These domains are not billed.

### How much does it cost?

> **Pricing note.** Pay-per-event. The price below is the planned placeholder; it is set in Apify Console when the Actor is published, so the *Pricing* tab on the Actor page is authoritative.

| Event | Charged when | Price |
|---|---|---|
| `domain-checked` | once per domain whose robots.txt gave a definitive answer (found, or confirmed missing) and whose result was saved | about $0.003 |

Not charged: unreachable domains, `inconclusive` results (401/403/429/5xx), invalid inputs, duplicates, skipped domains, and the optional llms.txt checks.

**Worked examples**

| Run | Calculation | Cost |
|---|---|---|
| 1 domain | 1 x $0.003 | $0.003 |
| 25 domains (the free-plan maximum per run) | 25 x $0.003 | $0.075 |
| 200 client domains, all answered | 200 x $0.003 | $0.60 |
| 1,000 competitor domains, 60 unreachable or firewalled | 940 x $0.003 | $2.82 |
| 5,000 domains, all answered | 5,000 x $0.003 | $15.00 |

**Free plan:** runs started from a free Apify plan process up to **25 domains per run**; the rest are listed in the dataset as `skipped_free_plan`. Paying users have no cap beyond 5,000 domains per run. **Spending limit:** if you set a maximum cost per run, the Actor reserves budget before checking each domain and stops gracefully when the limit cannot pay for another one; the unchecked domains appear as `skipped_charge_limit` and in `RUN-SUMMARY`, so you can run them again.

### Can AI agents use it? (MCP)

Yes. The Actor returns structured JSON, needs no credentials of its own and asks for Apify's *limited permissions* (it only uses its own storages), so agents can call it through [Apify's MCP server](https://mcp.apify.com) from any MCP client. Search for "AI crawler access checker" in the Actor search tool, or call it by ID: `atalaya-ai/ai-crawler-access-checker`.

Example prompt for an agent: *"Use atalaya-ai/ai-crawler-access-checker to check whether GPTBot, ClaudeBot and PerplexityBot are blocked on these domains, then summarize the ones that block a search crawler."* Use the `verdict`, `topFix` and `blockedCrawlers` fields as the answer; `crawlers[].matchedRule` gives the evidence.

### FAQ

#### What is the difference between training, search and user-triggered crawlers?

Training crawlers (GPTBot, ClaudeBot, Google-Extended...) collect content that may be used to train models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot...) index pages so the product can show and link them in answers. User-triggered fetchers (ChatGPT-User, Claude-User...) fetch one page because a person asked about it. Many sites block training bots and allow search bots; blocking a search bot can remove you from that product's answers. Blocking is a business choice, and this Actor does not tell you which to choose.

#### Does blocking GPTBot remove my site from ChatGPT?

According to OpenAI's documentation, `GPTBot` is for training and `OAI-SearchBot` is for ChatGPT search; they are controlled separately. This Actor reports each one on its own so you can see the difference.

#### Is this polite to the sites it checks?

It identifies itself as `AtalayaBot/0.1 (+https://apify.com/atalaya-ai)`. For each domain it sends at most three requests, one after another: `/robots.txt`, `/llms.txt`, `/llms-full.txt`, with a short pause, a 10 s timeout, size caps (about 560 KB for robots.txt, of which the first 500 KiB are parsed as RFC 9309 requires; 64 KB for the llms files) and at most 5 redirects. It skips the llms.txt requests if the site's robots.txt disallows them for `AtalayaBot` or `*`. It refuses localhost and private network addresses. Different domains are checked in parallel; the same domain never is.

#### Can it tell if my CDN or firewall blocks AI bots?

No. See the limitations below. A robots.txt that allows GPTBot says nothing about a firewall rule that blocks it.

#### Does it collect personal data?

No. It reads public robots.txt and llms.txt files and stores only the fields above.

### Limitations

- **robots.txt only.** It does not see firewall, CDN or bot-management blocks (for example Cloudflare's AI Crawl Control or a WAF rule), which can block a crawler that robots.txt allows. `hints.cloudflareManagedContent` only means Cloudflare's managed block appears in the file; Cloudflare may also enforce such blocks at its edge.
- **robots.txt is a request, not a lock.** Compliance is voluntary, and several vendors say user-triggered fetchers may not follow it.
- **No meta tags or headers.** `noindex`, `nosnippet`, `max-snippet` and `X-Robots-Tag` are not checked. Google controls its AI features in Search through Googlebot and snippet directives, not through Google-Extended.
- **Exact host only.** `example.com`, `www.example.com` and `blog.example.com` are three hosts with three robots.txt files. Only ports 80, 443, 8080 and 8443 are accepted.
- **Snapshot of the crawler list.** Tokens change; see `crawlerListVerifiedOn`. `Bytespider` has no first-party documentation found.
- **Presence only for llms.txt.** The Actor does not claim that llms.txt influences AI answers.
- **Our fetch can differ from a real crawler's.** Sites may answer our user agent differently (that is why 401/403/429/5xx give no verdict).
- Not affiliated with OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, ByteDance, DuckDuckGo, Cloudflare or Common Crawl; names are used only to identify their crawlers.

### Español

**Comprueba en masa qué rastreadores de IA permite o bloquea cada dominio en su `robots.txt`.** Pega hasta 5.000 dominios y obtén, por dominio y por rastreador (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, Googlebot, CCBot...), si está permitido o bloqueado, la regla exacta y el grupo que decidió, el tipo (entrenamiento, búsqueda o petición de usuario), si existen `llms.txt` y `llms-full.txt`, los sitemaps declarados y un veredicto de una línea con la mejora principal. Elige `language: es` para recibir `verdict` y `topFix` en español.

- **Precio:** pago por evento `domain-checked`, aproximadamente 0,003 USD por dominio comprobado (precio provisional; el definitivo se fija al publicar). No se cobran dominios inaccesibles, respuestas no concluyentes (401/403/429/5xx), entradas inválidas ni dominios omitidos.
- **Plan gratuito:** hasta 25 dominios por ejecución; el resto se marca como `skipped_free_plan`.
- **Educado con los sitios:** se identifica como `AtalayaBot`, una petición a la vez por dominio, con tiempos máximos y límites de tamaño.
- **Límites:** solo lee `robots.txt` (no ve bloqueos de firewall o CDN como Cloudflare AI Crawl Control), cumple el estándar RFC 9309 y la lista de rastreadores es una instantánea con fecha de verificación.
- **¿Necesitas una auditoría completa de un sitio?** Usa [`atalaya-ai/ai-search-readiness-auditor`](https://apify.com/atalaya-ai/ai-search-readiness-auditor).

# Actor input Schema

## `domains` (type: `array`):

Domains or website URLs, one per line (for example `example.com` or `https://www.example.com/pricing`). Only the host is used; each host is checked once (`example.com` and `www.example.com` are different hosts). Up to 5,000 per run. Free Apify plans check up to 25 domains per run.

## `crawlers` (type: `array`):

Leave empty to check all supported AI crawlers. Or list a subset by robots.txt token (case-insensitive): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Applebot, Applebot-Extended, CCBot, Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot, Amzn-SearchBot, Amzn-User, Bytespider, DuckAssistBot. Unknown tokens are ignored and listed in the run summary.

## `paths` (type: `array`):

The site root `/` is always tested. Add paths such as `/blog/` or `/pricing` to see whether each crawler may fetch them too (up to 20). The deciding robots.txt rule is reported per path.

## `language` (type: `string`):

Language of the one-line `verdict`, the `topFix` and status messages. All other fields (crawler names, statuses, rules) are language-neutral.

## `checkLlmsTxt` (type: `boolean`):

Adds two small requests per domain to report whether /llms.txt and /llms-full.txt exist. Turn off for the fastest robots.txt-only check. The price per domain is the same either way.

## `maxConcurrency` (type: `integer`):

How many different domains are checked at the same time (1-20). Requests to one domain are always sent one at a time with a short pause, whatever this value is.

## Actor input object example

```json
{
  "domains": [
    "nytimes.com",
    "https://www.bbc.com/news"
  ],
  "crawlers": [
    "GPTBot",
    "ClaudeBot",
    "PerplexityBot"
  ],
  "paths": [
    "/blog/",
    "/pricing"
  ],
  "language": "en",
  "checkLlmsTxt": true,
  "maxConcurrency": 10
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset items: one record per domain with the AI crawler verdicts, matched robots.txt rules, llms.txt / sitemap presence, verdict and top fix.

## `runSummary` (type: `string`):

Key-value record with counts, skipped and invalid inputs, billed events and the stop reason.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "nytimes.com",
        "wikipedia.org",
        "github.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("atalaya-ai/ai-crawler-access-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "nytimes.com",
        "wikipedia.org",
        "github.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("atalaya-ai/ai-crawler-access-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "nytimes.com",
    "wikipedia.org",
    "github.com"
  ]
}' |
apify call atalaya-ai/ai-crawler-access-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,atalaya-ai/ai-crawler-access-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/MpGt95DaxVYO8dONo/builds/Gb38cx2pXhggjZYTC/openapi.json