AI crawler access checker — GPTBot ClaudeBot robots

Find out which AI crawlers may read a website. Paste domains or URLs and get one policy verdict per site — are GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and the other AI bots allowed, blocked, or blocked only on some paths — checked against robots.txt the way the standard (RFC 9309) defines it, with the matching rule and line number as evidence. For SEO/GEO agencies, publishers and site owners. JSON, CSV or Excel, by API, on a schedule, or from n8n, Make, Zapier and AI agents (MCP). $0.005 per site checked, no subscription.

Independent tool. Not affiliated with OpenAI, Anthropic, Google, Perplexity, Apple, Meta or Common Crawl. Reads publicly available files only (/robots.txt, /llms.txt). A robots.txt verdict is not proof that a CDN or firewall lets the real crawler in — see "What the verdict does and does not tell you".

What does AI crawler access checker do?

• Answers the question behind most AI-visibility audits: am I opted out of AI training but still visible to AI search? Every bot is tagged training, search, user-fetch or ads, and each site gets a policy: open, blocks_training_only, blocks_ai_search, blocks_all, mixed or partial.
• Evaluates robots.txt correctly: wildcards (*), end anchors ($), longest match wins, Allow beats Disallow on a tie, several groups for one bot merged, the first 500 KiB only — and tells you which rule and which line decided each bot and each sample path (/, /blog/, /products/ …).
• Catches the traps: rules for the legacy anthropic-ai / Claude-Web tokens that don't govern ClaudeBot, Google-Extended and Applebot-Extended being control tokens that never show up in your logs, a * block that is not AI-specific, and robots.txt answers that are really an HTML page or a firewall challenge.
• Notes whether /llms.txt exists and looks like Markdown (presence and shape only), reads Content-Signal directives, and flags the CDN it can see.
• Monitors: schedule it and every result says whether the policy changed since the last run ("GPTBot: allowed → blocked").
• Optional, opt-in WAF probe for sites you own — see the limits below.

Who is it for?

You are…	You use it to…
SEO / GEO / AEO agency	Start every AI-visibility audit with a verified access check for a client and its competitors
Publisher or media group	Confirm the opt-out policy is live on every property and benchmark peers
In-house SEO or platform team	Catch the accidental block after a CMS, CDN or firewall change — on a weekly schedule
E-commerce or ads team	Check that ChatGPT search, user-fetch and ads crawlers may fetch product and landing pages
AI-agent builder	Ask "may agent X fetch this URL?" before fetching

Ready-made examples (one click)

Example	What you get
Check if GPTBot and ClaudeBot can crawl your site	Verdict and matching rule for the main training crawlers on four public sites
Audit AI search crawler access for GEO clients	OAI-SearchBot, PerplexityBot, Claude-SearchBot and user-fetch bots per site
Benchmark AI crawler opt-out policies across sites	One policy verdict per site for a whole list
Monitor robots.txt AI crawler policy changes weekly	Schedule-ready watchlist with a change flag
Check ChatGPT search, user-fetch and ads bot access	OAI-SearchBot, ChatGPT-User and OAI-AdsBot per path

How to use it

1. Paste domains or URLs, one per line (or open an example above). A path in a URL (https://example.com/blog/) is checked as an extra sample path for that site.
2. Optionally pick the bots and the sample paths. Leave the bots empty for the default set.
3. Set Max sites — this is also your cost cap — and click Start. Filter the dataset by policy, or download it.

Input

Field	Required	Description
sites	yes	Domains or URLs. One result per site (scheme + host)
bots	no	Which AI crawlers to evaluate. Default: GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, meta-externalfetcher. Optional extras: Applebot, Amazonbot, Bytespider, DuckAssistBot, MistralAI-User
customBots	no	Any other robots.txt user-agent token, e.g. a crawler announced after this Actor was published
samplePaths	no	Paths to evaluate for every bot (default /, max 10). A bot allowed on some but not all gets partial
checkLlmsTxt	no	One extra request per site: is there an /llms.txt? Default on
maxSites	no	Cap on paid results and cost. Default 100
wafProbe, probeBots	no	Opt-in, off by default. For sites you own or may test. See below
respectRobots	no	Skip /llms.txt and probe URLs when the site's robots.txt disallows this Actor (AICrawlerAccessChecker). Default on
maxConcurrency, requestTimeoutSecs	no	Sites in parallel (1–5, default 3) and per-request timeout. At least 0.7 s between requests to one site
trackHistory, historyStoreName	no	Remember verdicts in a named key-value store in your account and add change to the next run

Output

One row per site (abridged real result for jsr.io, 3 Nov 2026):

{
  "site": "https://jsr.io",
  "domain": "jsr.io",
  "robotsUrl": "https://jsr.io/robots.txt",
  "robotsStatus": "found",
  "policy": "blocks_training_only",
  "botsChecked": 14,
  "allowedCount": 8,
  "blockedCount": 6,
  "partialCount": 0,
  "blockedBots": ["GPTBot", "ClaudeBot", "Google-Extended", "Applebot-Extended", "CCBot", "meta-externalagent"],
  "bots": [
    { "bot": "GPTBot", "vendor": "OpenAI", "purpose": "training", "verdict": "blocked", "matchedBy": "bot", "rule": "Disallow: /", "ruleLine": 57 },
    { "bot": "OAI-SearchBot", "vendor": "OpenAI", "purpose": "search", "verdict": "allowed", "matchedBy": "wildcard", "rule": "Allow: /", "ruleLine": 32 }
  ],
  "contentSignals": { "search": "yes", "ai-train": "no", "use": "reference" },
  "llmsTxtPresent": false,
  "cdnHint": "cloudflare",
  "explanation": "Opted out of AI training crawlers (GPTBot, ClaudeBot, …) while AI search and user-fetch bots (OAI-SearchBot, ChatGPT-User, …) are allowed. This is what robots.txt says — it does not prove the CDN/WAF lets the real crawler in.",
  "status": "ok",
  "scrapedAt": "2026-11-03T18:57:54.314Z",
  "schemaVersion": 1
}

Field	Meaning
policy	open · blocks_training_only · blocks_ai_search · blocks_all · mixed · partial (some paths off limits, nothing blocked entirely) · unknown (no verdict possible)
bots[].verdict	allowed, blocked or partial across the sample paths; matchedBy says whether the bot's own group, the * group or no group decided
robotsStatus	found, empty, not_found_allow_all (404/410: no robots.txt, everything allowed) — charged; fetch_blocked, server_error, unreachable, not_robots, blocked_host, invalid_input — free, with an explanation
legacyTokenWarnings	Rules for deprecated tokens that don't protect you
change / changeDetail	baseline, unchanged, changed (a verdict flipped), robots_edited — with change tracking on
wafProbe, wafBlockedBots	Only when the probe is on
status	ok, partial (charged, but llms.txt or the probe could not be completed) or not_found (free)

What the verdict does and does not tell you

• It is what robots.txt says, evaluated per RFC 9309. Reputable AI crawlers say they honour robots.txt; the operators' own documents also say some user-initiated fetchers (for example ChatGPT-User, Perplexity-User, meta-externalfetcher) may not — those rows carry that hint.
• It does not prove access. A CDN, firewall or bot-management rule can block a crawler that robots.txt allows (and the other way round). Login walls, paywalls and JavaScript-only pages are out of scope.
• Bot names change. The registry in the Actor follows the operators' public documentation as of November 2026; entries marked optional were not verified. Add a new crawler with customBots the day it appears.
• Not checked: <meta name="robots"> / X-Robots-Tag directives (such as noai), ai.txt, sitemaps, rate limits, whether a crawler really visited.

The optional WAF probe — read before using

Off by default and meant only for sites you own or are allowed to test. With wafProbe on, the Actor requests up to 3 sample URLs once with a neutral user-agent (its own) and once for each probe bot with an approximate User-Agent string containing that bot's token, then compares status codes and known challenge-page markers (Cloudflare, Akamai, Imperva, DataDome, PerimeterX, AWS WAF, Sucuri, Vercel …). It reports bots a user-agent rule or challenge page treats differently, and says when the neutral request is blocked too (inconclusive).

It cannot prove what the real crawler experiences: it runs from an Apify datacenter IP, and CDNs may verify crawlers by their published IP ranges or reverse DNS. A clean result is not proof the real crawler gets in; a block is not proof the real crawler is blocked. To stay polite it only sends a bot's user-agent to URLs that bot may fetch according to robots.txt, and it skips paths that this Actor's own user-agent may not fetch (unless you turn respectRobots off). The probe adds no extra charge.

How much does it cost?

Pay per event — you only pay for results delivered.

Event	Price	When
Actor start	Apify default	Once per run
Site checked	$0.005	A site whose robots.txt was read and evaluated (includes the llms.txt check and the optional probe)

Free: sites that cannot be read — unreachable, server errors, a robots.txt request blocked for this Actor (401/403/429), a "robots.txt" that is really an HTML page, private hosts, invalid input. Every free row says why.

Examples: 100 sites ≈ $0.50 · 1,000 sites ≈ $5 · a 50-site watchlist run weekly for a month ≈ $1. Set Max sites (or Max cost per run) as your cap. Higher Apify plans get Store discounts automatically.

Schedule it

Weekly (Monday morning) is enough for a watchlist: CDN, CMS and bot-management changes are what flip verdicts. With change tracking on, the dataset tells you which sites changed and exactly what ("GPTBot: allowed → blocked"). Use a different historyStoreName per watchlist.

Use with AI agents (MCP)

Connect the Apify MCP server and ask your agent:

"Use kondasviktor/ai-crawler-access-checker to check whether GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot may crawl example.com and example.org, and tell me which of them are opted out of AI training but still open to AI search."

FAQ

Is a site that "blocks GPTBot" invisible to ChatGPT? Not necessarily. According to OpenAI's crawler documentation, GPTBot is the training crawler; OAI-SearchBot is the one that affects ChatGPT search. That is why results are tagged by purpose.

Why was my site a free row? The Actor could not read a trustworthy robots.txt (see robotsStatus and explanation). fetch_blocked usually means a firewall answered the Actor with 403/429 or a challenge page — the Actor never reports that as "allowed".

What is partial? The bot may crawl some of your sample paths but not others (for example / allowed, /gist/ disallowed). Add the sections you care about to samplePaths.

Does it follow redirects? Yes, up to 5 hops, each checked against private-network rules. The row shows the final robotsUrl. example.com and www.example.com are different sites with their own robots.txt — check the one your visitors use.

Is this legal advice, or proof of consent for AI use? No. It reports a technical signal.

Can you add a bot or fix a verdict? Yes — open an issue with the site and the bot. Fixes within 48 hours.

Limitations

• Static robots.txt evaluation; no proof that a CDN/WAF lets the real crawler in.
• Public sites on ports 80/443 only; no login, no JavaScript rendering, no proxy (so sites that block datacenter traffic give free fetch_blocked rows).
• At most 10 sample paths per site; the probe uses the first 3.
• llms.txt is checked for presence and basic shape only (a validator/generator is planned as a separate tool).

Related Actors

• OpenAI Ads Readiness Checker — AdsBot robots, HTTPS, landing-page and image checks before you run ChatGPT ads
• OpenAI Ads Check & Convert — readiness check plus Google Merchant feed → OpenAI Ads CSV
• Website Change Monitor — get alerted when any page changes

---

Built by Viktor Kondas · Vibe Coder's Life · Custom audits: hello@vibecoderslife.com