AI Crawler Access Audit: can ChatGPT, Claude, Perplexity and Gemini read your site?

If an AI crawler can't fetch your page, the AI can't cite it. This Actor checks, page by page, whether the crawlers behind ChatGPT search, Claude, Perplexity, Gemini, Google AI Overviews, Copilot, Apple and DuckDuckGo are allowed in and can actually read your content. No API key, no login, $0.02 per page.

It focuses on the things that block AI visibility outright, and labels the rest honestly:

Check	Why it matters
robots.txt for 19 AI crawlers (OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, DuckAssistBot, MistralAI-User, meta-externalagent, Amazonbot, CCBot, Bytespider)	Blocking an answer bot (e.g. OAI-SearchBot, PerplexityBot) means that engine cannot cite you. Blocking a training bot (GPTBot, CCBot) is a policy choice and is reported separately. Uses RFC 9309 matching: exact user-agent groups, longest rule wins, Allow wins ties, unreachable robots.txt = full disallow.
CDN / firewall blocks	Re-requests the page with the published user-agent of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. A 403 or challenge page while a browser gets 200 usually means a "block AI bots" rule (e.g. Cloudflare) that robots.txt doesn't show.
noindex / nosnippet / X-Robots-Tag	Search-based AI answers (Google AI Overviews and AI Mode, Bing/Copilot) won't use noindex pages; nosnippet keeps text out of AI Overviews.
Content without JavaScript	Most AI crawlers don't run JavaScript. If the raw HTML has almost no text, they see an empty page.
Sitemap, title, meta description, H1	Discovery and basic readability.
llms.txt, JSON-LD structured data	Reported as info only: server-log studies found AI crawlers rarely request llms.txt, and controlled tests found no measurable citation effect from schema markup. We tell you, but we don't pretend they're critical.

Each page gets a 0–100 score and a list of issues ranked critical → info. One site-summary record per site rolls everything up.

Sample output

Real output recorded on 2026-10-17 for https://www.nytimes.com/ (a site that blocks most AI crawlers on purpose). Crawler list trimmed to 5 of 19 for readability.

{
  "type": "page",
  "url": "https://www.nytimes.com/",
  "finalUrl": "https://www.nytimes.com/",
  "status": 403,
  "responseMs": 47,
  "fetchError": null,
  "title": "nytimes.com",
  "metaDescription": null,
  "lang": "en",
  "canonical": null,
  "h1Count": 0,
  "headingCount": 0,
  "noindex": false,
  "nosnippet": false,
  "xRobotsTag": null,
  "metaRobots": null,
  "htmlBytes": 774,
  "wordsWithoutJavaScript": 8,
  "likelyNeedsJavaScript": false,
  "structuredDataTypes": [],
  "aiCrawlerAccess": [
    {
      "bot": "OAI-SearchBot",
      "owner": "OpenAI",
      "purpose": "answer",
      "usedFor": "ChatGPT search results and citations",
      "allowedByRobots": false,
      "matchedGroup": "specific",
      "rule": "Disallow: /"
    },
    {
      "bot": "ChatGPT-User",
      "owner": "OpenAI",
      "purpose": "answer",
      "usedFor": "ChatGPT fetching a page a user asked about",
      "allowedByRobots": false,
      "matchedGroup": "specific",
      "rule": "Disallow: /"
    },
    {
      "bot": "GPTBot",
      "owner": "OpenAI",
      "purpose": "training",
      "usedFor": "OpenAI model training",
      "allowedByRobots": false,
      "matchedGroup": "specific",
      "rule": "Disallow: /"
    },
    {
      "bot": "Claude-SearchBot",
      "owner": "Anthropic",
      "purpose": "answer",
      "usedFor": "Claude search results",
      "allowedByRobots": false,
      "matchedGroup": "specific",
      "rule": "Disallow: /"
    },
    {
      "bot": "Googlebot",
      "owner": "Google",
      "purpose": "search",
      "usedFor": "Google Search, AI Overviews and AI Mode",
      "allowedByRobots": true,
      "matchedGroup": "specific",
      "rule": null
    }
  ],
  "issues": [
    {
      "severity": "high",
      "code": "refused-auditor",
      "message": "The page answered 403 to this audit (a normal browser user-agent from a cloud IP). Bot protection may also be refusing AI crawlers; check your CDN/WAF rules. robots.txt results below are still valid."
    },
    {
      "severity": "critical",
      "code": "answer-bots-blocked",
      "message": "robots.txt blocks OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot. These fetch pages for AI answers/search, so this page cannot be cited by OpenAI, Anthropic, Perplexity, DuckDuckGo."
    },
    {
      "severity": "info",
      "code": "training-bots-blocked",
      "message": "robots.txt blocks training crawlers GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider. That is a policy choice: it keeps content out of future model training but does not stop AI search citations."
    },
    {
      "severity": "low",
      "code": "missing-meta-description",
      "message": "No meta description (used as a summary by some crawlers and search snippets)."
    },
    {
      "severity": "low",
      "code": "missing-h1",
      "message": "No <h1> heading."
    },
    {
      "severity": "info",
      "code": "no-llms-txt",
      "message": "No /llms.txt. Optional: server-log studies found AI crawlers rarely request it, so expect little effect."
    },
    {
      "severity": "info",
      "code": "no-structured-data",
      "message": "No JSON-LD structured data. Useful for Google rich results; controlled tests found no measurable effect on AI citations, so this is low priority."
    }
  ],
  "score": 51,
  "checkedAt": "2026-10-17T19:24:38.918Z"
}

Site summary:

{
  "type": "site-summary",
  "origin": "https://www.nytimes.com",
  "pagesAudited": 1,
  "averageScore": 51,
  "robotsTxt": {
    "found": true,
    "status": 200,
    "error": null,
    "unreachable": false
  },
  "llmsTxt": {
    "found": false,
    "status": 404,
    "bytes": 0
  },
  "sitemap": {
    "found": true,
    "url": "https://www.nytimes.com/sitemaps/new/news.xml.gz",
    "declaredInRobots": true
  },
  "answerBotsBlockedOnHomepageOrFirstPage": [
    "OAI-SearchBot",
    "ChatGPT-User",
    "Claude-SearchBot",
    "Claude-User",
    "PerplexityBot",
    "Perplexity-User",
    "DuckAssistBot"
  ],
  "trainingBotsBlocked": [
    "GPTBot",
    "ClaudeBot",
    "Google-Extended",
    "Applebot-Extended",
    "meta-externalagent",
    "CCBot",
    "Bytespider"
  ],
  "issueCounts": {
    "refused-auditor": 1,
    "answer-bots-blocked": 1,
    "training-bots-blocked": 1,
    "missing-meta-description": 1,
    "missing-h1": 1,
    "no-llms-txt": 1,
    "no-structured-data": 1
  },
  "checkedAt": "2026-10-17T19:24:40.136Z"
}

How to use

1. Paste one or more page URLs (homepage plus your most important product, pricing or blog pages).
2. Optional: set Extra pages from sitemap to audit up to 200 more pages per site from its XML sitemap.
3. Run. Use the Overview table, or export JSON/CSV. Schedule it monthly (or after CDN/robots changes) to catch regressions.

Typical fixes it points to: remove Disallow for OAI-SearchBot/PerplexityBot/Claude-SearchBot, turn off a CDN "block AI bots" rule for answer bots, drop accidental noindex, server-render key content, publish a sitemap.

Pricing

$0.02 per page audited (robots.txt, sitemap and llms.txt checks per site are included). Pages that refuse the audit (401/403/429) or can't be reached are reported but not charged. Example: homepage + 24 sitemap pages = $0.50.

Limits

• The user-agent probe comes from a cloud IP. Some CDNs block unverified bots that use an AI user-agent but allow the real crawler (verified by IP), so treat a probe block as "check your CDN settings", not proof.
• Being allowed in doesn't guarantee being cited. To measure how often ChatGPT, Gemini and Perplexity actually mention your brand, use AI Brand Visibility Tracker.
• Reads public pages only, never logs in, and makes at most 7 requests per page (1 normal + up to 5 bot user-agents + site files once per site).