AI Crawler Access Checker: robots.txt & llms.txt
Pricing
from $2.00 / 1,000 domain checkeds
Check which AI crawlers each domain allows in robots.txt: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and 20 more, each with the exact rule that matched. Adds a one-line AI visibility verdict (open to AI search, blocks training …), llms.txt and llms-full.txt checks and sitemap URLs.
Websites to check, one per line, up to 5,000 per run. Bare domains (example.com), subdomains (docs.example.com) and full URLs all work; each hostname becomes one row. Note that example.com and www.example.com can have different robots.txt files.
Also look for /llms.txt and /llms-full.txt and check their basic format (H1 title, summary, link sections, size). Adds two small requests per domain. Same price.
Optional robots.txt tokens to check besides the built-in list of AI crawlers, e.g. Googlebot-Image or YourOwnBot. Up to 20. They are reported per domain but don't change the AI visibility verdict.
