AI Crawler Access Checker: GPTBot, ClaudeBot and robots.txt, in bulk

Paste a list of domains and get, for each one, which AI crawlers (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended and 15 more) its robots.txt allows or blocks, the exact rule and group that decided it, whether it has llms.txt / llms-full.txt, and a one-line verdict with the top fix, in English or Spanish. One cheap, pay-per-domain check you can run on 10, 500 or 5,000 domains. No browser, no API keys.

Built for people who check many sites: SEO and GEO agencies auditing client portfolios, in-house SEOs benchmarking competitors, brands and publishers checking which AI bots they let in, and AI agents that need a factual answer before giving advice.

Need the full picture of one site (page metadata, schema, scored fixes)? Use the companion Actor atalaya-ai/ai-search-readiness-auditor. This one only answers "who can crawl it?", fast and for many domains.

What does it check for each domain?

• robots.txt fetch status: found, missing (404), HTML served instead, access denied (401/403), rate limited (429), server error (5xx) or unreachable. Status codes are interpreted per RFC 9309.
• Allowed or blocked for every AI crawler, for / and for any extra paths you list, with the matched rule (for example Disallow: /), its line number and the group that applied (the crawler's own token, a vendor-documented fallback, or *).
• Crawler type: training, AI search / retrieval, or user-triggered fetcher. Blocking each has a different effect (see the FAQ).
• Counts per type: how many training and search crawlers are blocked or allowed.
• /llms.txt and /llms-full.txt: present or not (presence only).
• Sitemaps declared in robots.txt.
• Hints, only when visible in the text: Cloudflare's managed robots.txt block and Content-Signal values. They never change the verdicts.
• Verdict and top fix in one line each (en or es).

Which AI crawlers are covered?

20 crawlers, grouped by what blocking them does. Tokens are matched case-insensitively. Source URLs and the last verification date (crawlerListVerifiedOn, currently 2026-10-19) are kept in the Actor's crawler list; vendors change their bots, so treat the list as a snapshot.

Type	What blocking it does	Tokens
Training	Opts your content out of model training. Does not remove you from search.	GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Gemini training and grounding), Applebot-Extended, CCBot (Common Crawl), Meta-ExternalAgent, Amazonbot, Bytespider (ByteDance; vendor documentation not found, so flagged unverified)
Search / retrieval	Removes you from that product's AI answers and links.	OAI-SearchBot (ChatGPT search), Claude-SearchBot, PerplexityBot, Googlebot (Google documents it as the control for AI Overviews / AI Mode), Applebot (falls back to the Googlebot group if not named), Amzn-SearchBot, DuckAssistBot
User-triggered fetchers	Stops fetches made because a user asked an assistant about your page. Several vendors say these may ignore robots.txt.	ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User

What does the output look like?

One dataset item per domain (abbreviated; sample-output.json in the Actor source has 5 real, complete items):

{
  "domain": "nytimes.com",
  "status": "checked",
  "robots": { "url": "https://nytimes.com/robots.txt", "outcome": "found", "httpStatus": 200, "directives": 273 },
  "posture": "search_partly_blocked",
  "verdict": "4 of 7 AI search crawlers are blocked or restricted (OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot); those products may not cite this site.",
  "topFix": "If you want to appear in their answers, allow OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot: remove the Disallow rule or add a \"User-agent: <bot>\" + \"Allow: /\" group.",
  "summary": { "totalChecked": 20, "trainingAllowed": 1, "trainingBlocked": 7, "searchAllowed": 3, "searchBlocked": 4, "userFetchAllowed": 1, "userFetchBlocked": 4 },
  "blockedCrawlers": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "..."],
  "crawlers": [
    { "token": "GPTBot", "category": "training", "status": "blocked", "rootAllowed": false,
      "matchedGroup": "GPTBot", "matchedRule": "Disallow: /", "matchedRuleLine": 233 },
    { "token": "OAI-SearchBot", "category": "search", "status": "blocked", "rootAllowed": false,
      "matchedGroup": "OAI-SearchBot", "matchedRule": "Disallow: /", "matchedRuleLine": 266 }
  ],
  "llmsTxt": { "present": false, "httpStatus": 404 },
  "llmsFullTxt": { "present": false, "httpStatus": 404 },
  "sitemap": { "declared": true, "count": 25 },
  "hints": { "cloudflareManagedContent": false, "contentSignals": null },
  "billableEvents": { "domain-checked": 1 }
}

The default Overview view shows one row per domain; the Crawler matrix view shows one row per domain and crawler. Export as JSON, CSV or Excel from the Output tab.

Output fields

Field	Meaning
status	checked (verdicts included, billed), inconclusive (the server answered 401/403/429/5xx: no verdict, not billed), unreachable (DNS, TLS, timeout or private address: not billed), error, invalid_input, skipped_free_plan, skipped_charge_limit (not processed, not billed). statusDetail explains.
robots	outcome, httpStatus, size, number of directives, a note when the outcome is not a clean found.
posture	One label: all_open, search_open_training_blocked, search_partly_blocked, search_blocked, all_blocked, restricted, no_robots_txt, inconclusive.
verdict, topFix	One line each, in the language you chose.
summary	Counts: training/search/userFetch x Allowed/Blocked/Partial.
crawlers[]	Per crawler: status (allowed, blocked, partial), rootAllowed, matchedGroup, matchedRule, matchedRuleLine, paths[] for your extra paths, disallowRules, crawlDelay, vendor note.
blockedCrawlers, partialCrawlers, allowedCrawlers	The same result as flat lists of tokens.
llmsTxt, llmsFullTxt	present (true / false / null when not checked), httpStatus, and for llms.txt whether the first line is a # Title.
sitemap	declared, count, up to 10 URLs from Sitemap: lines in robots.txt.
hints	cloudflareManagedContent, contentSignals.

partial means the result differs between the checked paths, or / is blocked but Allow rules open some sections.

A run also stores a RUN-SUMMARY record: counts, invalid and skipped inputs, unknown crawler names, events charged and the stop reason.

How do I check hundreds of domains?

1. Open the Actor, paste your domains in Domains to check (one per line; URLs are fine, only the host is used; up to 5,000 per run).
2. Optionally pick a subset of crawlers (for example GPTBot, ClaudeBot, PerplexityBot) and extra paths such as /blog/.
3. Choose the language of the verdict and start.
4. Download the dataset, or call it through the API:

curl -X POST "https://api.apify.com/v2/acts/atalaya~ai-crawler-access-checker/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"domains":["example.com","example.org"],"crawlers":["GPTBot","ClaudeBot"],"language":"en"}'

Input	Default	Description
domains (required)	3 example domains	Domains or URLs, 1-5,000. Duplicate hosts are checked once; example.com and www.example.com are different hosts.
crawlers	all 20	Subset by robots.txt token, case-insensitive. Unknown tokens are ignored and listed in RUN-SUMMARY.
paths	none	Up to 20 extra paths tested next to /.
language	en	en or es: language of verdict, topFix, statusDetail. Other fields are language-neutral.
checkLlmsTxt	on	Two small extra requests per domain. Same price either way.
maxConcurrency	10	Different domains checked in parallel (1-20). One request at a time per domain, always.

How is the result decided?

The Actor reads each domain's /robots.txt and applies RFC 9309, the Robots Exclusion Protocol standard:

• The group naming the crawler's token wins (several groups for the same token are merged); if none names it, the * group applies. Applebot falls back to a Googlebot group because Apple documents that.
• The longest matching rule wins; Allow wins a tie. * and a trailing $ work in patterns. A missing file (404) means no restrictions.
• Deliberate difference: if robots.txt answers 401, 403, 429 or 5xx, or the connection fails, you get no verdict instead of "everything allowed" or "everything blocked". A 403 is very often a firewall rule against unknown bots, so the answer a real AI crawler gets may be different. These domains are not billed.

How much does it cost?

Pricing note. Pay-per-event. The price below is the planned placeholder; it is set in Apify Console when the Actor is published, so the Pricing tab on the Actor page is authoritative.

Event	Charged when	Price
domain-checked	once per domain whose robots.txt gave a definitive answer (found, or confirmed missing) and whose result was saved	about $0.003

Not charged: unreachable domains, inconclusive results (401/403/429/5xx), invalid inputs, duplicates, skipped domains, and the optional llms.txt checks.

Worked examples

Run	Calculation	Cost
1 domain	1 x $0.003	$0.003
25 domains (the free-plan maximum per run)	25 x $0.003	$0.075
200 client domains, all answered	200 x $0.003	$0.60
1,000 competitor domains, 60 unreachable or firewalled	940 x $0.003	$2.82
5,000 domains, all answered	5,000 x $0.003	$15.00

Free plan: runs started from a free Apify plan process up to 25 domains per run; the rest are listed in the dataset as skipped_free_plan. Paying users have no cap beyond 5,000 domains per run. Spending limit: if you set a maximum cost per run, the Actor reserves budget before checking each domain and stops gracefully when the limit cannot pay for another one; the unchecked domains appear as skipped_charge_limit and in RUN-SUMMARY, so you can run them again.

Can AI agents use it? (MCP)

Yes. The Actor returns structured JSON, needs no credentials of its own and asks for Apify's limited permissions (it only uses its own storages), so agents can call it through Apify's MCP server from any MCP client. Search for "AI crawler access checker" in the Actor search tool, or call it by ID: atalaya-ai/ai-crawler-access-checker.

Example prompt for an agent: "Use atalaya-ai/ai-crawler-access-checker to check whether GPTBot, ClaudeBot and PerplexityBot are blocked on these domains, then summarize the ones that block a search crawler." Use the verdict, topFix and blockedCrawlers fields as the answer; crawlers[].matchedRule gives the evidence.

FAQ

What is the difference between training, search and user-triggered crawlers?

Training crawlers (GPTBot, ClaudeBot, Google-Extended...) collect content that may be used to train models. Search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot...) index pages so the product can show and link them in answers. User-triggered fetchers (ChatGPT-User, Claude-User...) fetch one page because a person asked about it. Many sites block training bots and allow search bots; blocking a search bot can remove you from that product's answers. Blocking is a business choice, and this Actor does not tell you which to choose.

Does blocking GPTBot remove my site from ChatGPT?

According to OpenAI's documentation, GPTBot is for training and OAI-SearchBot is for ChatGPT search; they are controlled separately. This Actor reports each one on its own so you can see the difference.

Is this polite to the sites it checks?

It identifies itself as AtalayaBot/0.1 (+https://apify.com/atalaya-ai). For each domain it sends at most three requests, one after another: /robots.txt, /llms.txt, /llms-full.txt, with a short pause, a 10 s timeout, size caps (about 560 KB for robots.txt, of which the first 500 KiB are parsed as RFC 9309 requires; 64 KB for the llms files) and at most 5 redirects. It skips the llms.txt requests if the site's robots.txt disallows them for AtalayaBot or *. It refuses localhost and private network addresses. Different domains are checked in parallel; the same domain never is.

Can it tell if my CDN or firewall blocks AI bots?

No. See the limitations below. A robots.txt that allows GPTBot says nothing about a firewall rule that blocks it.

Does it collect personal data?

No. It reads public robots.txt and llms.txt files and stores only the fields above.

Limitations

• robots.txt only. It does not see firewall, CDN or bot-management blocks (for example Cloudflare's AI Crawl Control or a WAF rule), which can block a crawler that robots.txt allows. hints.cloudflareManagedContent only means Cloudflare's managed block appears in the file; Cloudflare may also enforce such blocks at its edge.
• robots.txt is a request, not a lock. Compliance is voluntary, and several vendors say user-triggered fetchers may not follow it.
• No meta tags or headers. noindex, nosnippet, max-snippet and X-Robots-Tag are not checked. Google controls its AI features in Search through Googlebot and snippet directives, not through Google-Extended.
• Exact host only. example.com, www.example.com and blog.example.com are three hosts with three robots.txt files. Only ports 80, 443, 8080 and 8443 are accepted.
• Snapshot of the crawler list. Tokens change; see crawlerListVerifiedOn. Bytespider has no first-party documentation found.
• Presence only for llms.txt. The Actor does not claim that llms.txt influences AI answers.
• Our fetch can differ from a real crawler's. Sites may answer our user agent differently (that is why 401/403/429/5xx give no verdict).
• Not affiliated with OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, ByteDance, DuckDuckGo, Cloudflare or Common Crawl; names are used only to identify their crawlers.

Español

Comprueba en masa qué rastreadores de IA permite o bloquea cada dominio en su robots.txt. Pega hasta 5.000 dominios y obtén, por dominio y por rastreador (GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot, Google-Extended, Googlebot, CCBot...), si está permitido o bloqueado, la regla exacta y el grupo que decidió, el tipo (entrenamiento, búsqueda o petición de usuario), si existen llms.txt y llms-full.txt, los sitemaps declarados y un veredicto de una línea con la mejora principal. Elige language: es para recibir verdict y topFix en español.

• Precio: pago por evento domain-checked, aproximadamente 0,003 USD por dominio comprobado (precio provisional; el definitivo se fija al publicar). No se cobran dominios inaccesibles, respuestas no concluyentes (401/403/429/5xx), entradas inválidas ni dominios omitidos.
• Plan gratuito: hasta 25 dominios por ejecución; el resto se marca como skipped_free_plan.
• Educado con los sitios: se identifica como AtalayaBot, una petición a la vez por dominio, con tiempos máximos y límites de tamaño.
• Límites: solo lee robots.txt (no ve bloqueos de firewall o CDN como Cloudflare AI Crawl Control), cumple el estándar RFC 9309 y la lista de rastreadores es una instantánea con fecha de verificación.
• ¿Necesitas una auditoría completa de un sitio? Usa atalaya-ai/ai-search-readiness-auditor.