AI Crawler Access & GPTBot Checker

See your website the way AI does.

AI assistants and answer engines — ChatGPT search, Claude, Perplexity, Gemini, Meta AI — no longer discover your pages only through Google. They send their own crawlers. If your robots.txt blocks GPTBot or PerplexityBot, or someone added a noai meta tag years ago, your site is invisible in AI answers — and you would never know from your analytics.

This Actor audits any website and tells you exactly:

• which AI crawlers your robots.txt allows or blocks — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, and more (list is fully editable)
• whether /llms.txt and /llms-full.txt exist — the emerging standard that helps AI models index your site
• whether AI opt-out meta tags (noai, noimageai) or X-Robots-Tag headers are present
• your AI Visibility Score (0–100) with a prioritized, actionable fix list

Use cases

• SEO & AI-visibility consultants — audit client sites before selling an "AI search optimization" engagement, then re-audit after fixes.
• Marketing teams — check whether a CMS migration, WAF rule, or new security plugin silently blocked AI crawlers.
• Developers — paste any domain to understand its AI-crawler policy in seconds; export JSON via API for monitoring.

Input

Field	Type	Description
websiteUrls	string list (required)	One or more origins to audit, e.g. https://example.com. Up to 50, audited in parallel.
bots	string list	robots.txt user-agent tokens to check. Defaults cover the major AI crawlers (November 2026).
checkLlmsTxt	boolean	Probe /llms.txt and /llms-full.txt. Default: true.
checkHtmlSignals	boolean	Inspect meta robots tags and X-Robots-Tag headers. Default: true.
requestTimeoutSecs	integer	Per-request timeout. Default: 15.
generateMarkdownReport	boolean	Save a shareable report.md to the key-value store. Default: true.

Output

Dataset — one item per site:

{
  "url": "https://example.com",
  "aiVisibilityScore": 72,
  "verdict": "Partially open",
  "robotsTxt": "found",
  "llmsTxt": "missing",
  "blockedBots": ["GPTBot", "ClaudeBot"],
  "botResults": [ { "bot": "GPTBot", "operator": "OpenAI", "allowed": false, "matched_rule": "Disallow: /", "...": "..." } ],
  "recommendations": [ { "issue": "...", "impact": "...", "how_to_fix": "..." } ]
}

Key-value store — report.md: a ready-to-send markdown report with a per-bot table and recommendations.

How the score works

Start at 100. Subtract for every blocked AI crawler, weighted by how much it matters for AI visibility: search bots (−20), live browsing (−10), model training (−8), open-web aggregators (−5). Missing /llms.txt (−10) and explicit AI opt-out tags (−15) also reduce the score. The score is a transparent heuristic, not a ranking — the per-bot table is the ground truth.

Getting started

1. Run the Actor with the default input (https://www.example.com) to see the output format.
2. Enter your own domain(s).
3. Open Storage → Key-value store → report.md for the human-readable report, or consume the dataset via API / webhooks / Make & Zapier integrations.

FAQ

Does this work on sites behind bot protection? The Actor only fetches robots.txt, /llms.txt and the homepage like a normal browser. Hard-blocked hosts will show error status rather than fail the run.

Does it render JavaScript? No — meta tags and robots.txt are static artifacts. Sites that inject robots directives only via JS are out of scope for this tool.

Is the bot list up to date? The defaults reflect the major AI crawlers as of November 2026. New crawlers appear regularly — you can audit any token by editing the bots input, and updates ship via version bumps.

How is this compliant? The Actor requests three publicly accessible URLs per site, honours redirects and standard timeouts, stores no personal data, and does not bypass any access control.

Limitations

• robots.txt matching follows RFC 9309 semantics (groups, * wildcards, $ anchors, longest-match) with the common "tie goes to Allow" convention; exotic edge cases may differ from individual vendors' parsers.
• Score weights are a published heuristic, not an industry standard.

Changelog

• 0.1.0 — initial release: 15 default AI crawlers, llms.txt probing, HTML/X-Robots-Tag opt-out detection, visibility score, markdown report.