AI Crawler Access Checker
Pricing
Pay per usage
AI Crawler Access Checker
See which AI crawlers each site lets in: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 18 more, read from robots.txt the way RFC 9309 crawlers read it. Also checks llms.txt and noindex/noai tags.
Pricing
Pay per usage
Rating
0.0
(0)
Developer
Sagnato
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
11 hours ago
Last modified
Categories
Features
Share
Find out which AI crawlers a website lets in. For every site you list, this actor reads robots.txt the way crawlers that follow RFC 9309 read it and tells you, for 22 known AI and search crawlers, whether each one is allowed, partly blocked or blocked, and which line decided it.
What it checks
- robots.txt, per crawler: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Bingbot, Applebot, Applebot-Extended, Amazonbot, Meta-ExternalAgent, Meta-ExternalFetcher, DuckAssistBot, CCBot, Bytespider, cohere-ai and MistralAI-User, plus any names you add.
- Each crawler's purpose: training, search, fetching for a user, or an opt-out token, so you can see at a glance whether a site blocks training but stays visible in AI search.
- llms.txt and llms-full.txt: whether the site publishes them, the title, and how many links they hold.
- Home page signals: noindex, noai and noimageai in the X-Robots-Tag header or robots meta tag.
Why the results are right
Many quick checks only look for a Disallow line next to a bot's name. Crawlers don't read robots.txt that way. This actor follows the rules crawlers use:
- A group that names a crawler replaces the
*group completely, so a site that names ClaudeBot with no rules lets it in even if*is blocked. - The longest matching rule wins, and
Allowwins a tie. *and$wildcards in paths are supported.- A missing robots.txt (4xx) allows everything; an unreachable one (5xx or no answer) means "stay out".
Input
| Field | What it does |
|---|---|
sites | Domains or URLs, one per line. Up to 1,000 per run. |
paths | Paths to test each crawler against. Default /. |
extraBots | More user-agent names to test. |
checkLlmsTxt | Look for llms.txt (default on). |
checkHomepage | Read home page noindex/noai signals (default on). |
Output
One dataset item per site, with each crawler's access and the deciding rule, a summary (counts, blocked crawlers, whether training is blocked while search is allowed), llms.txt details and notes. A readable Markdown report is saved as REPORT.md in the run's key-value store.
{"origin": "https://example.com","summary": { "allowed": 19, "partlyBlocked": 0, "blocked": 3, "blockedBots": ["GPTBot", "Google-Extended", "CCBot"], "trainingBlockedSearchAllowed": true, "hasLlmsTxt": true },"bots": [{ "bot": "GPTBot", "operator": "OpenAI", "purpose": "training", "access": "blocked", "rule": "Disallow: /" }]}
Good to know
- robots.txt is a request, not a lock. A firewall or CDN rule (such as a "block AI bots" setting) can still turn crawlers away by address, and no outside check can see that. The actor says so in every result.
- It reads at most three small files per site (robots.txt, llms.txt, the home page), one request at a time per site, at least a second apart, and it obeys robots.txt for its own user agent,
SagnatoBot. - It collects no personal data and uses no AI at run time.