AI Crawler Access Auditor

Can ChatGPT, Perplexity, Claude, Apple and Google actually reach your website? This Actor checks your robots.txt the way the crawlers read it, finds noindex tags and firewall blocks, and tells you in plain English what to fix, with copy-paste snippets for your developer.

No API key needed. One website takes about 20–30 seconds.

Allowing a crawler only makes your pages available to it. It does not guarantee that any AI assistant or search engine will index, cite, recommend or rank your site. This tool checks access, the first step.

What it checks

Check	How	Result type
robots.txt rules per crawler	Parsed per RFC 9309: exact crawler-name groups, catch-all * fallback, longest-match Allow/Disallow, $ and * wildcards, 500 KiB limit, 4xx = "allow all", 5xx/unreachable = "crawlers may stop".	Declared (confirmed)
Apple's Googlebot fallback	If Applebot isn't named, Apple says it follows your Googlebot rules. We apply that.	Declared
noindex / nosnippet	<meta name="robots">, bot-specific meta tags and the X-Robots-Tag header, with bot scoping (googlebot: noindex).	Declared
noai / noimageai	Detected and flagged as informal: the major AI vendors don't document support.	Info
Firewall / bot protection	One polite test request per crawler using its name (homepage only), compared with a normal browser request. Detects Cloudflare, Akamai, Imperva, DataDome, HUMAN, AWS WAF, Sucuri and Vercel challenge or block pages, plus 403/429 responses.	Signal, not proof
robots.txt health	Typos (Dissallow), rules before any User-agent, noindex in robots.txt, HTML served as robots.txt, Cloudflare-managed sections, Content-Signal lines.	Confirmed
llms.txt	Present / missing / HTML soft-404. Informational only: no major AI crawler operator documents using it.	Info

Crawlers covered (14)

Type	Crawlers
AI search (can show your pages in answers)	OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, Applebot
AI user fetches (open a page when someone asks)	ChatGPT-User, Perplexity-User, Claude-User
AI training (separate choice from search)	GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot
Classic search (baseline)	Googlebot, Bingbot

Add up to 10 more names (e.g. Amazonbot, meta-externalagent, Bytespider) with Other robots.txt names. They are evaluated in robots.txt only.

Confirmed vs. signal vs. unknown

Each finding is labelled so you know how much to trust it:

• Confirmed / declared: what your robots.txt, meta tags and headers say. This is deterministic.
• Signal: how your site responded to our test request that used the crawler's name. The requests come from Apify's servers, not OpenAI's, Perplexity's or anyone else's IP addresses. Firewalls that verify real crawlers by IP address may treat the genuine crawler differently, in either direction. Treat a signal as "worth checking", not as proof.
• Unknown: for example, robots.txt timed out, or the site blocks all cloud traffic (including our normal-browser request), so we can't tell.

We never send test requests as a crawler that your robots.txt blocks, and we stop testing a site as soon as it rate-limits us.

Output

One dataset row per website:

• verdict, which is one of open, ai-search-blocked, ai-search-partly-blocked, noindex, possible-firewall-block, some-pages-restricted, robots-unreadable, inconclusive or unknown
• headline and summary in plain English
• aiSearchAccess, userFetchAccess, trainingAccess and searchEngineAccess, each one of allowed, blocked, mixed or unknown
• confirmed[], uncertain[] and suggestedActions[] (with priority)
• fixes[]: robots.txt snippets that keep your existing private-path rules, plus firewall guidance with links to the vendors' published IP ranges
• crawlers[]: for each crawler, the robots.txt decision with the exact rule and line number, the test-request result, and vendor docs
• robotsTxt, pages, llmsTxt and cdn

For each site there's also a readable Markdown report in the key-value store (REPORT-<host>), plus a run SUMMARY.

Example (shortened):

{
  "site": "https://www.example.com",
  "verdict": "ai-search-partly-blocked",
  "headline": "robots.txt blocks some AI search crawlers: OAI-SearchBot and PerplexityBot.",
  "aiSearchAccess": "mixed",
  "trainingAccess": "blocked",
  "fixes": [{ "title": "Allow OAI-SearchBot and PerplexityBot", "snippet": "User-agent: OAI-SearchBot\nUser-agent: PerplexityBot\nAllow: /\nDisallow: /admin" }]
}

Input

Field	Default	Notes
urls	["https://apify.com"]	Up to 25 websites per run, audited one at a time. A page URL audits that page and the homepage.
extraPaths	[]	Up to 4 more paths per site, e.g. /pricing.
liveChecks	true	Firewall test requests (homepage only).
crawlers	all	Limit to some crawlers.
extraUserAgents	[]	Extra robots.txt names.
requestDelayMs	1000	Pause between requests to the same site (500–10000).
aiSummary + openaiApiKey	off	Optional plain-English summary written by OpenAI with your own key (billed to you by OpenAI, typically well under $0.001 per site with gpt-6-luna). The model only rewrites our findings. The audit never needs AI and never uses anyone else's key.

Common fixes it suggests

• Your catch-all User-agent: * with Disallow: / blocks AI search: add a dedicated group for the AI search crawlers that keeps your private-path rules.
• A crawler has its own blocking group: replace that group, or remove just its User-agent line from a shared group. We warn you when deleting a line would make its rules attach to the group above.
• Cloudflare "Block AI bots" / Bot Fight Mode / AI Crawl Control: where to look, and why WAF Skip rules should match the user-agent and the vendor's IP ranges.
• noindex left over from staging: where it is, and what to remove.
• Training vs. search: blocking GPTBot, ClaudeBot or Google-Extended is a separate choice. It doesn't remove you from ChatGPT search, Claude search or Google Search, per those vendors' docs.

Pricing

Pay per event: you're charged once per website audited. Websites we can't reach at all are free. See the Pricing tab.

Next step: is AI actually mentioning you?

Access is necessary but not sufficient. To measure whether ChatGPT, Perplexity, Claude and Gemini actually mention and cite your brand for your buyers' questions, use the AI Search Visibility Tracker.

Limits and honesty notes

• Test requests can't reproduce IP-verified allowlisting. Results are signals.
• When look-alike Googlebot and Bingbot requests are both blocked, we treat it as normal anti-spoofing protection (look-alike-blocked), not a problem, because real search bots are verified by IP.
• Some sites serve a different robots.txt or page to verified crawlers (by IP address). We report the robots.txt we received.
• Bots that execute JavaScript challenges, or that get country-specific responses, may see something different.
• Vendor rules change. The crawler list and its notes were checked against vendor docs on 2026-11-05 and are linked in every result.
• Please audit only sites you own or are authorised to check. The Actor is polite (one site at a time, up to about 20 requests per site, 1 second apart by default), and it identifies its test requests with SteadfeedAuditProbe/1.0.

By Steadfeed.