AI Crawler Access Checker (robots.txt + llms.txt)
Pricing
$1.00 / 1,000 websites
AI Crawler Access Checker (robots.txt + llms.txt)
Check which AI bots each website allows or blocks: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and 25+ more, using RFC 9309 robots.txt matching. Also detects llms.txt and llms-full.txt, sitemaps, meta robots and noai tags. Bulk, fast, no browser.
Pricing
$1.00 / 1,000 websites
Rating
0.0
(0)
Developer
Matt
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
a day ago
Last modified
Categories
Security
Limited permissionsShare
Find out which AI bots a website lets in, for any number of sites. For each site you get a per-bot verdict for 32 AI and search crawlers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot and more. Verdicts follow the official RFC 9309 robots.txt rules: exact user-agent groups, longest match wins, * and $ wildcards. The tool also detects llms.txt / llms-full.txt, sitemap directives, meta robots, X-Robots-Tag and noai tags.
You don't need a browser or proxies, and results come back in seconds.
Why use it
- GEO / AI SEO audits: find out whether your content can show up in ChatGPT search, Perplexity, Claude and Google AI features
- Publisher and competitor research: see who blocks AI training while still allowing AI search, across hundreds of sites
- Monitoring: schedule it with
onlyBlockingto catch a robots.txt change that suddenly blocks AI search bots - llms.txt adoption: see which sites publish
llms.txt, its title and how many links it has
Output per site
| Field | Meaning |
|---|---|
aiPolicy | allows-all, blocks-all, blocks-training-allows-search or partial |
blockedBots / allowedBots | Bot tokens blocked or allowed for the tested path |
trainingBotsBlocked, searchBotsBlocked | Counts by purpose (training, search, user-fetch) |
bots[] | Per bot: operator, purpose, allowed, matchedGroup (its own group or *), matchedRule (e.g. Disallow: /) |
explicitlyNamedBots | Bots the site names in robots.txt |
llmsTxtFound, llmsTxtTitle, llmsTxtLinkCount, llmsFullTxtFound | llms.txt details |
sitemaps, userAgentGroups, robotsStatusCode | Other robots.txt details |
metaRobots, xRobotsTag, noAiDirective, noIndex | Page-level directives |
Under RFC 9309, an unreachable or 5xx robots.txt counts as "disallow all" and a 4xx counts as "allow all".
Input
Leave bots empty to check all 32, or list the tokens you care about.
Example result
Pricing
$1.00 per 1,000 websites (pay-per-event, site-checked). There's no start fee and no charge for platform usage. Sites that can't be reached at all are free.
Notes
- The tool reads only public files (
/robots.txt,/llms.txt,/llms-full.txtand the tested page) and sends an honest user agent. - robots.txt is a request, not an enforcement mechanism. This tool reports what a site declares.
- The bot list is kept up to date. Open an issue if you need a new crawler token added.
