AI Crawler Robots.txt Audit

Audit whether ChatGPT, Claude, Perplexity, Google AI, Apple and major model-training crawlers may access each website according to its live robots.txt. The result separates AI answer visibility from model-training access, because blocking those are two different business decisions.

A single old rule can quietly block an answer crawler and reduce the chance that a brand appears or gets cited in AI search. The opposite mistake also happens: a publisher wants answer-engine visibility but unintentionally leaves every training crawler open. This Actor turns the live policy into a plain recommendation instead of making someone compare bot names by hand.

What it checks

For each public domain, the Actor:

1. Downloads the live robots.txt without requiring a login, cookies, or credentials.
2. Applies specific user-agent groups before wildcard rules, including Allow and Disallow precedence at the site root.
3. Reports answer/search crawlers separately from training crawlers.
4. Checks whether /llms.txt exists as an additional readiness signal.
5. Returns the exact matched rule for every crawler so an SEO specialist can verify the result.

Search and answer crawlers include OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot. Training or extended-use controls include GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Amazonbot and FacebookBot. Google-Extended does not control inclusion or ranking in Google Search. Bot behavior and vendor naming can change; the output is an audit of the current public file, not a promise of indexing or citation.

Ready-to-run example

{
  "domains": ["example.com", "your-client-site.com"],
  "includeTrainingBots": true,
  "blockedOnly": false,
  "requestTimeoutSecs": 15
}

Every required input has a working default. Enter up to 50 domains or full public URLs. No account or API key is needed.

Output example

{
  "domain": "your-client-site.com",
  "robotsUrl": "https://your-client-site.com/robots.txt",
  "robotsStatus": 200,
  "llmsTxtFound": true,
  "searchVisibilityRisk": true,
  "blockedSearchBots": ["OAI-SearchBot", "PerplexityBot"],
  "blockedTrainingBots": ["GPTBot", "ClaudeBot"],
  "allowedSearchBots": ["ChatGPT-User", "Claude-SearchBot"],
  "recommendation": "Review robots.txt: OAI-SearchBot, PerplexityBot cannot crawl the site root, which may reduce visibility in AI answers.",
  "checkedAt": "2026-07-19T10:00:00.000Z"
}

The full crawlerResults array includes each crawler's company, purpose, allowed status, exact matched rule, and whether the decision came from a specific or wildcard group. A SUMMARY record reports websites checked and sites with an AI search visibility risk.

Who uses it

• SEO and GEO agencies: audit every client after a migration, CDN change, or robots.txt deployment. Use blockedOnly: true on a schedule to keep the result list actionable.
• B2B SaaS teams: verify that an inherited robots.txt file does not block answer engines while normal Google SEO still looks healthy.
• Publishers: distinguish protecting content from model training from disappearing from AI-assisted discovery.
• Web agencies: add an AI crawler access check to launch and maintenance reports without manually tracking every bot name.

Reading the result

searchVisibilityRisk: true means at least one listed answer/search crawler is blocked at /. It does not prove the website is absent from an AI product: vendors can use other indexes and systems. A healthy result also does not guarantee citation. The Actor reports the access policy accurately and leaves the business decision to you.

A missing robots.txt normally means crawling is allowed. A network error is different: the Actor returns status 0, includes the error, and recommends rerunning before changing policy. A missing llms.txt is informational, not automatically a defect.

Data, limits, and responsible use

The Actor reads only public robots.txt and llms.txt files. It collects no personal data, does not bypass access controls, and does not log in. Runs are capped at 50 websites to keep cost and load predictable. Schedule weekly or after deployments rather than polling aggressively.

Unofficial tool disclaimer

This Actor is an unofficial, independent audit tool. It is not affiliated with or endorsed by OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, or Common Crawl. Product names identify public crawler user agents only. Always confirm policy changes against the vendors' current documentation before editing a production robots.txt file.