AI Crawler Access Test — Can ChatGPT, Claude & Perplexity Read Your Site?

If AI search and answer bots can't fetch your pages, you can't be cited in ChatGPT search, Perplexity or Claude answers, however good your content is. Most "AI readiness" checkers only read robots.txt. This one actually fetches your page as each AI bot and compares the result with a normal browser, so it also catches the block robots.txt never shows: a CDN or firewall rule rejecting AI crawlers.

Passive: a handful of normal GET requests per site. Keyless. Runs as a batch Actor or an instant HTTP endpoint.

What it tests

Test	Detail
robots.txt per bot	18 crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, DuckDuckGo, Common Crawl and ByteDance. Exact RFC 9309 group matching and longest-rule wins, checked for the exact URL path you give.
Search vs training	Each bot is classed as search, user-fetch (an assistant fetching a page for a user) or training. Blocking training bots (GPTBot, ClaudeBot, Google-Extended, CCBot…) is a legitimate choice. Blocking search / user-fetch bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User…) removes you from AI answers. The report keeps these apart.
Live fetch as each bot	The page is fetched with each bot's published User-Agent and compared with a browser fetch. A spoofed Googlebot is used as a control. If it gets through but GPTBot doesn't, that's a confident, deliberate AI-bot block (e.g. a CDN "block AI bots" rule). If the site rejects the fake Googlebot too, it's impostor protection and the result is marked inconclusive rather than guessed.
Content in the HTML	AI crawlers don't run JavaScript. A page that is an empty JS shell is invisible to them, even if it looks fine in a browser.
llms.txt	Presence and validity (H1, summary, linked sections), plus llms-full.txt.
Entity signals	Organization / LocalBusiness JSON-LD, sameAs profile links, content schema (FAQ, Product, Article…), meta description and heading structure. These help models identify you correctly.
Index hygiene	noindex, noai / noimageai directives (meta and X-Robots-Tag), sitemap.

Output

{
  "url": "https://www.nytimes.com",
  "verdict": "ai-invisible",
  "aiReadinessScore": 28,
  "majorSearchBotsBlocked": ["ChatGPT-User", "Claude-SearchBot", "Claude-User", "OAI-SearchBot", "Perplexity-User", "PerplexityBot"],
  "trainingBotsBlocked": ["GPTBot", "ClaudeBot", "Google-Extended", "Applebot-Extended", "CCBot"],
  "findings": [{"severity": "high", "code": "search-bots-disallowed", "detail": "robots.txt blocks AI SEARCH / answer bots: ..."}],
  "summary": "www.nytimes.com -> ai-invisible (28/100): major AI search/answer bots 2/8 not blocked, training bots blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended; ..."
}

Real results from the same run: react.dev → ai-ready (88), notion.com → ai-ready (98), anthropic.com → ai-ready (80, no llms.txt).

Each report also has a bots matrix (owner, purpose, robots rule, live status and block reason), the rendering, llmsTxt, structuredData and directives details, and a prioritised fixes list.

Verdicts: ai-ready · ai-readable-with-gaps · partially-blocked · ai-invisible · allowed-by-robots-live-test-inconclusive (the site rejects even a browser-like automated fetch).

One API call, JSON back

Run it synchronously: one POST starts the run and returns the results in the same response. No run to poll, which is what an AI agent wants mid-task:

curl -X POST "https://api.apify.com/v2/acts/centorprat~ai-crawler-access-test/run-sync-get-dataset-items" -H "Authorization: Bearer <APIFY_TOKEN>" \
     -H "Content-Type: application/json" -d '{"urls": ["example.com", "react.dev"]}'

The body takes the same fields as the normal input. Same pay-per-event price; failed checks aren't charged.

Ready-made tasks

• Is My Site Blocking ChatGPT, Claude or Perplexity?
• llms.txt & AI Readiness Check

For AI agents / MCP

https://mcp.apify.com/?tools=actors,docs,centorprat/ai-crawler-access-test

Use cases

• GEO / AEO audits: the "why aren't we cited?" diagnostic (pair with AI Share of Voice).
• After a CDN / WAF change: confirm you didn't silently block ChatGPT or Perplexity.
• Agencies: batch-test client sites and hand over the fix list.
• Publishers: verify your training opt-out doesn't also block AI search traffic.

FAQ

Is the live test the same as the real bot? It uses each bot's published User-Agent from Apify's servers. Rules that check the bots' real IP ranges can't be reproduced, which is why the Googlebot control exists and inconclusive cases are labelled as such.

Should I block training bots? That's your call. The report treats it as information, not a problem. Blocking search / user-fetch bots is what costs AI visibility.

Does it crawl my site? No. It makes one page fetch per tested bot, plus robots.txt, llms.txt and sitemap.xml.

Cost? One site-tested event per URL. No API keys, no paid scraper.

Related actors

• AI Share of Voice: how often AI models recommend your brand.
• Robots.txt & Sitemap Analyzer
• Structured Data & Schema Validator