AI Crawler Checker: GPTBot, ClaudeBot & Cloudflare Blocks

AI Crawler Checker: can GPTBot, ClaudeBot, PerplexityBot and Gemini read your site?

If an AI crawler can't fetch your page, ChatGPT, Claude, Perplexity and Google AI Overviews can't cite it. This AI crawler checker / GEO audit tests, page by page, whether the crawlers behind ChatGPT search (OAI-SearchBot, ChatGPT-User, GPTBot), Claude, Perplexity, Gemini, Google AI Overviews, Copilot, Apple and DuckDuckGo are allowed by robots.txt and not silently blocked by Cloudflare or another CDN/WAF. Bulk-check your own sites, client sites or competitors. No API key, no login, $5 per 1,000 pages ($0.005 per page).

Why this AI crawler checker

Most robots.txt checkers stop at robots.txt. The most common real-world block in 2025-2026 is a CDN "block AI bots" rule (e.g. Cloudflare's), which robots.txt doesn't show. This Actor re-requests each page with the real user-agents of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot and flags 403s and challenge pages.

	This Actor	robots.txt-only checkers
robots.txt verdict for 19 AI crawlers, with the rule that decided it	Yes	Yes
Cloudflare / WAF bot-block probe	Yes	No
noindex / nosnippet / X-Robots-Tag	Yes	Rarely
JavaScript-only pages (AI crawlers see an empty page)	Yes	No
Answer bots vs training bots separated	Yes	Sometimes
0-100 score + ranked fixes	Yes	Sometimes
Price	$0.005 per page, refused/unreachable pages free	$0.001 to $0.02 per site

It focuses on the things that block AI visibility outright, and labels the rest honestly:

Check	Why it matters
robots.txt for 19 AI crawlers (OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, DuckAssistBot, MistralAI-User, meta-externalagent, Amazonbot, CCBot, Bytespider)	Blocking an answer bot (e.g. OAI-SearchBot, PerplexityBot) means that engine cannot cite you. Blocking a training bot (GPTBot, CCBot) is a policy choice and is reported separately. Uses RFC 9309 matching: exact user-agent groups, longest rule wins, Allow wins ties, unreachable robots.txt = full disallow.
CDN / firewall blocks	Re-requests the page with the published user-agent of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. Challenge pages are detected from real interstitial evidence (Cloudflare cf-mitigated header, challenge markup, challenge title or text on a small page), not from words like "captcha" inside page scripts. A 403 or challenge page while a browser gets 200 usually means a "block AI bots" rule (e.g. Cloudflare) that robots.txt doesn't show.
noindex / nosnippet / X-Robots-Tag	Search-based AI answers (Google AI Overviews and AI Mode, Bing/Copilot) won't use noindex pages; nosnippet keeps text out of AI Overviews.
Content without JavaScript	Most AI crawlers don't run JavaScript. If the raw HTML has almost no text, they see an empty page.
Sitemap, title, meta description, H1	Discovery and basic readability. Sitemaps are looked up from robots.txt (an http:// one is retried over https://), then /sitemap.xml and /sitemap_index.xml.
llms.txt, JSON-LD structured data	Reported as info only: server-log studies found AI crawlers rarely request llms.txt, and controlled tests found no measurable citation effect from schema markup. We tell you, but we don't pretend they're critical.

Each page gets a 0–100 score and a list of issues ranked critical → info. One site-summary record per site rolls everything up.

Ready-made example inputs (copy into the Input JSON tab)

Quick check of one homepage

{"urls": ["https://www.example.com/"]}

Homepage + 25 pages from the sitemap (full GEO audit)

{"urls": ["https://www.example.com/"], "sitemapPagesPerSite": 25}

Benchmark competitors: who blocks ChatGPT and Perplexity?

{"urls": ["https://www.hubspot.com/", "https://www.salesforce.com/", "https://www.pipedrive.com/"], "probeUserAgents": true}

Agency batch: robots.txt only, no CDN probe (fastest)

{"urls": ["https://client-one.com/", "https://client-two.com/", "https://client-three.com/"], "probeUserAgents": false}

Sample output

Real output recorded on 2026-10-17 for https://www.nytimes.com/ (a site that blocks most AI crawlers on purpose). Crawler list trimmed to 5 of 19 for readability.

{
  "type": "page",
  "url": "https://www.nytimes.com/",
  "finalUrl": "https://www.nytimes.com/",
  "status": 403,
  "responseMs": 47,
  "fetchError": null,
  "title": "nytimes.com",
  "metaDescription": null,
  "lang": "en",
  "canonical": null,
  "h1Count": 0,
  "headingCount": 0,
  "noindex": false,
  "nosnippet": false,
  "xRobotsTag": null,
  "metaRobots": null,
  "htmlBytes": 774,
  "wordsWithoutJavaScript": 8,
  "likelyNeedsJavaScript": false,
  "structuredDataTypes": [],
  "aiCrawlerAccess": [
    {
      "bot": "OAI-SearchBot",
      "owner": "OpenAI",
      "purpose": "answer",
      "usedFor": "ChatGPT search results and citations",
      "allowedByRobots": false,
      "matchedGroup": "specific",
      "rule": "Disallow: /"
    },
    {
      "bot": "ChatGPT-User",
      "owner": "OpenAI",
      "purpose": "answer",
      "usedFor": "ChatGPT fetching a page a user asked about",
      "allowedByRobots": false,
      "matchedGroup": "specific",
      "rule": "Disallow: /"
    },
    {
      "bot": "GPTBot",
      "owner": "OpenAI",
      "purpose": "training",
      "usedFor": "OpenAI model training",
      "allowedByRobots": false,
      "matchedGroup": "specific",
      "rule": "Disallow: /"
    },
    {
      "bot": "Claude-SearchBot",
      "owner": "Anthropic",
      "purpose": "answer",
      "usedFor": "Claude search results",
      "allowedByRobots": false,
      "matchedGroup": "specific",
      "rule": "Disallow: /"
    },
    {
      "bot": "Googlebot",
      "owner": "Google",
      "purpose": "search",
      "usedFor": "Google Search, AI Overviews and AI Mode",
      "allowedByRobots": true,
      "matchedGroup": "specific",
      "rule": null
    }
  ],
  "issues": [
    {
      "severity": "high",
      "code": "refused-auditor",
      "message": "The page answered 403 to this audit (a normal browser user-agent from a cloud IP). Bot protection may also be refusing AI crawlers; check your CDN/WAF rules. robots.txt results below are still valid."
    },
    {
      "severity": "critical",
      "code": "answer-bots-blocked",
      "message": "robots.txt blocks OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot. These fetch pages for AI answers/search, so this page cannot be cited by OpenAI, Anthropic, Perplexity, DuckDuckGo."
    },
    {
      "severity": "info",
      "code": "training-bots-blocked",
      "message": "robots.txt blocks training crawlers GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider. That is a policy choice: it keeps content out of future model training but does not stop AI search citations."
    },
    {
      "severity": "low",
      "code": "missing-meta-description",
      "message": "No meta description (used as a summary by some crawlers and search snippets)."
    },
    {
      "severity": "low",
      "code": "missing-h1",
      "message": "No <h1> heading."
    },
    {
      "severity": "info",
      "code": "no-llms-txt",
      "message": "No /llms.txt. Optional: server-log studies found AI crawlers rarely request it, so expect little effect."
    },
    {
      "severity": "info",
      "code": "no-structured-data",
      "message": "No JSON-LD structured data. Useful for Google rich results; controlled tests found no measurable effect on AI citations, so this is low priority."
    }
  ],
  "score": 51,
  "checkedAt": "2026-10-17T19:24:38.918Z"
}

Site summary:

{
  "type": "site-summary",
  "origin": "https://www.nytimes.com",
  "pagesAudited": 1,
  "averageScore": 51,
  "robotsTxt": {
    "found": true,
    "status": 200,
    "error": null,
    "unreachable": false
  },
  "llmsTxt": {
    "found": false,
    "status": 404,
    "bytes": 0
  },
  "sitemap": {
    "found": true,
    "url": "https://www.nytimes.com/sitemaps/new/news.xml.gz",
    "declaredInRobots": true
  },
  "answerBotsBlockedOnHomepageOrFirstPage": [
    "OAI-SearchBot",
    "ChatGPT-User",
    "Claude-SearchBot",
    "Claude-User",
    "PerplexityBot",
    "Perplexity-User",
    "DuckAssistBot"
  ],
  "trainingBotsBlocked": [
    "GPTBot",
    "ClaudeBot",
    "Google-Extended",
    "Applebot-Extended",
    "meta-externalagent",
    "CCBot",
    "Bytespider"
  ],
  "issueCounts": {
    "refused-auditor": 1,
    "answer-bots-blocked": 1,
    "training-bots-blocked": 1,
    "missing-meta-description": 1,
    "missing-h1": 1,
    "no-llms-txt": 1,
    "no-structured-data": 1
  },
  "checkedAt": "2026-10-17T19:24:40.136Z"
}

How to use

1. Paste one or more page URLs (homepage plus your most important product, pricing or blog pages).
2. Optional: set Extra pages from sitemap to audit up to 200 more pages per site from its XML sitemap.
3. Run. Use the Overview table, or export JSON/CSV. Schedule it monthly (or after CDN/robots changes) to catch regressions.

Typical fixes it points to: remove Disallow for OAI-SearchBot/PerplexityBot/Claude-SearchBot, turn off a CDN "block AI bots" rule for answer bots, drop accidental noindex, server-render key content, publish a sitemap.

Pricing

$0.005 per page audited ($5 per 1,000 pages), platform usage included. Site-level robots.txt, sitemap and llms.txt checks are included. Pages that refuse the audit (401/403/429) or can't be reached are reported but not charged. Example: homepage + 24 sitemap pages = $0.125.

FAQ

What is GEO / AEO? Generative Engine Optimization: making your pages readable and citable by AI answer engines (ChatGPT search, Perplexity, Google AI Overviews, Claude). Crawler access is step one.

Should I block GPTBot? That's a policy choice. Blocking training bots (GPTBot, CCBot, Google-Extended) keeps content out of future training; blocking answer bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot) stops AI search from citing you. The report separates the two.

Does it work with Cloudflare? Yes, that's the point of the user-agent probe.

Can I use it from an AI agent? Yes: add https://mcp.apify.com?tools=sefran1/ai-crawler-access-audit to Claude, Cursor or any MCP client.

Limits

• The user-agent probe comes from a cloud IP. Some CDNs block unverified bots that use an AI user-agent but allow the real crawler (verified by IP), so treat a probe block as "check your CDN settings", not proof.
• Being allowed in doesn't guarantee being cited. To measure how often ChatGPT, Gemini and Perplexity actually mention your brand, use AI Visibility Tracker.
• Reads public pages only, never logs in, and makes at most 7 requests per page (1 normal + up to 5 bot user-agents + site files once per site).

Changelog

• 0.2 (2026-10-22): Fewer false positives. Challenge-page detection no longer fires on the word "captcha" inside page JavaScript (e.g. reCAPTCHA feature flags) or on Cloudflare's normal bot-detection script; each probe now shows challengeEvidence. Sitemap discovery retries an http:// sitemap from robots.txt over https:// and falls back to /sitemap.xml and /sitemap_index.xml; a declared sitemap that fails while another works is reported as info (declared-sitemap-unreachable), not "no sitemap".
• 0.1: First release.