AI Crawler Access Checker avatar

AI Crawler Access Checker

Pricing

Pay per usage

Go to Apify Store
AI Crawler Access Checker

AI Crawler Access Checker

See which AI crawlers each site lets in: GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 18 more, read from robots.txt the way RFC 9309 crawlers read it. Also checks llms.txt and noindex/noai tags.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Sagnato

Sagnato

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

11 hours ago

Last modified

Categories

Features

Limited permissions

Share

Find out which AI crawlers a website lets in. For every site you list, this actor reads robots.txt the way crawlers that follow RFC 9309 read it and tells you, for 22 known AI and search crawlers, whether each one is allowed, partly blocked or blocked, and which line decided it.

What it checks

  • robots.txt, per crawler: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Bingbot, Applebot, Applebot-Extended, Amazonbot, Meta-ExternalAgent, Meta-ExternalFetcher, DuckAssistBot, CCBot, Bytespider, cohere-ai and MistralAI-User, plus any names you add.
  • Each crawler's purpose: training, search, fetching for a user, or an opt-out token, so you can see at a glance whether a site blocks training but stays visible in AI search.
  • llms.txt and llms-full.txt: whether the site publishes them, the title, and how many links they hold.
  • Home page signals: noindex, noai and noimageai in the X-Robots-Tag header or robots meta tag.

Why the results are right

Many quick checks only look for a Disallow line next to a bot's name. Crawlers don't read robots.txt that way. This actor follows the rules crawlers use:

  • A group that names a crawler replaces the * group completely, so a site that names ClaudeBot with no rules lets it in even if * is blocked.
  • The longest matching rule wins, and Allow wins a tie.
  • * and $ wildcards in paths are supported.
  • A missing robots.txt (4xx) allows everything; an unreachable one (5xx or no answer) means "stay out".

Input

FieldWhat it does
sitesDomains or URLs, one per line. Up to 1,000 per run.
pathsPaths to test each crawler against. Default /.
extraBotsMore user-agent names to test.
checkLlmsTxtLook for llms.txt (default on).
checkHomepageRead home page noindex/noai signals (default on).

Output

One dataset item per site, with each crawler's access and the deciding rule, a summary (counts, blocked crawlers, whether training is blocked while search is allowed), llms.txt details and notes. A readable Markdown report is saved as REPORT.md in the run's key-value store.

{
"origin": "https://example.com",
"summary": { "allowed": 19, "partlyBlocked": 0, "blocked": 3, "blockedBots": ["GPTBot", "Google-Extended", "CCBot"], "trainingBlockedSearchAllowed": true, "hasLlmsTxt": true },
"bots": [{ "bot": "GPTBot", "operator": "OpenAI", "purpose": "training", "access": "blocked", "rule": "Disallow: /" }]
}

Good to know

  • robots.txt is a request, not a lock. A firewall or CDN rule (such as a "block AI bots" setting) can still turn crawlers away by address, and no outside check can see that. The actor says so in every result.
  • It reads at most three small files per site (robots.txt, llms.txt, the home page), one request at a time per site, at least a second apart, and it obeys robots.txt for its own user agent, SagnatoBot.
  • It collects no personal data and uses no AI at run time.