AI Crawler Checker — which sites block GPTBot, ClaudeBot and Perplexity

See which AI crawlers any website allows or blocks in its robots.txt, and whether it publishes an llms.txt. One row per site, for 16 crawlers: OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, ByteDance, Meta and Amazon, plus Googlebot and Bingbot as a baseline.

Quick start: press Start with no configuration. It checks twelve well-known news and tech sites so one run shows the whole output shape. Then paste your own domains — yours, your competitors', or a whole market.

Built for SEO and GEO (Generative Engine Optimization) teams, publishers deciding their AI policy, and researchers tracking who opts out of AI.

Sample output

From a real run on 31 October 2026, path /:

domain	robotsTxtStatus	aiSearchAccess	aiTrainingAccess	blocked	llmsTxtFound
nytimes.com	found	none	partial	13 of 16	false
theguardian.com	found	partial	partial	9 of 16	false
bbc.com	found	partial	none	12 of 16	false
cnn.com	found	none	partial	12 of 16	false
medium.com	found	full	partial	6 of 16	false
wikipedia.org	found	full	full	0 of 16	false
stackoverflow.com	refused	–	–	–	false

On that run ClaudeBot, Applebot-Extended and Bytespider were each blocked by 10 of the 12 sites, GPTBot by 6, OAI-SearchBot by 3, and Googlebot by none. The Guardian allows every OpenAI crawler but blocks all three of Anthropic's.

Each row:

{
  "domain": "theguardian.com", "path": "/",
  "robotsTxtFound": true, "robotsTxtStatus": "found",
  "robotsTxtUrl": "https://www.theguardian.com/robots.txt",
  "bots": { "GPTBot": "allowed", "OAI-SearchBot": "allowed", "ClaudeBot": "blocked", "PerplexityBot": "blocked", "…": "…" },
  "blockedBots": ["ClaudeBot", "Claude-SearchBot", "Claude-User", "PerplexityBot", "…"],
  "allowedBots": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "Googlebot", "Bingbot", "…"],
  "namedBots": ["ClaudeBot", "Claude-SearchBot", "…"],
  "blocksAllAi": false, "aiSearchAccess": "partial", "aiTrainingAccess": "partial",
  "llmsTxtFound": false, "llmsTxtBytes": 0, "note": null,
  "checkedAt": "2026-10-31T15:25:30.214Z"
}

Rows are about 1.2 KB, small enough to hand straight to an AI agent.

What each field means

• bots — every crawler, allowed or blocked for the path tested.
• namedBots — crawlers the file names in a group of their own. A site that names GPTBot made a deliberate choice; a site that only blocks * may never have thought about it.
• aiSearchAccess — full, partial or none across the crawlers that put pages into AI answers: OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User.
• aiTrainingAccess — the same across training crawlers: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot.
• blocksAllAi — every AI crawler blocked. Googlebot and Bingbot are not counted.
• robotsTxtStatus — found; none (404, or an HTML page served instead, so everything is allowed); refused (the site answered 401, 403, 418 or 429 to us, so we cannot see what the crawlers see); error (5xx or no answer).
• llmsTxtFound — /llms.txt answers 200 with text. An HTML page served at that address, which many sites do for any unknown path, does not count.

Input

Field	Default	Notes
websites	12 news and tech sites	Domains or URLs. A URL with a path tests that path. Up to 1,000. A bad entry is skipped with a reason in the log and in SUMMARY; the rest still run.
path	/	The path tested for bare domains. Use /blog/ to see whether AI crawlers may read your articles.

Why these checks

robots.txt is how site owners tell each AI vendor what it may do, and each vendor documents its own tokens: OpenAI, Anthropic, Perplexity, Google-Extended, Applebot-Extended, Amazonbot and CCBot. Those pages separate training from search: blocking GPTBot keeps a site out of OpenAI's training data, while OAI-SearchBot decides whether ChatGPT search can show it. A site that blocks both may be giving up citations it did not mean to.

Being cited matters more each month. In the GEO study (arXiv 2311.09735), simple content changes raised a page's visibility in AI answers by up to 40% — but only on pages the engine could read. For the content side, run AI Search Readiness Audit on the page.

The file is read the way the crawlers read it (RFC 9309): a crawler follows the group that names it and ignores * when one does; longest match wins; * and $ wildcards are supported.

FAQ

Does this crawl the site? No. Two requests per site, /robots.txt and /llms.txt, nothing else.

Does "allowed" guarantee the crawler reads my page? No. robots.txt says what is permitted. A firewall or bot wall can still turn a crawler away. If we are refused ourselves, the row says refused and gives no verdict.

Is ChatGPT-User bound by robots.txt? OpenAI says user-initiated fetches may not follow it. The row reports what your file says either way.

What am I charged for? One event per site whose robots.txt was read or confirmed missing. Sites that refuse us or do not answer are returned and not charged.

Works well with

Same publisher, same output conventions, same honest-failure rules:

• ai-search-readiness-audit — A 0–100 score for how citable a page is by ChatGPT, Perplexity and AI Overviews, with fixes.

Disclaimer

This Actor is unofficial and not affiliated with OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, ByteDance, Meta, Amazon or Microsoft. It scrapes no website content and calls no third-party API: it reads two public files that sites publish for automated clients, identifying itself honestly in its user agent. Crawler names are the vendors' own and are used only to describe their crawlers.