AI Crawler & robots.txt Access Checker (GPTBot, Claude)

See which AI crawlers a website allows or blocks: GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more, read straight from robots.txt using the standard matching rules (RFC 9309). One verdict per crawler for each domain, plus a headline policy, the site's sitemaps and whether it publishes llms.txt.

What is the AI Crawler Access Checker?

Publishers are deciding what AI companies may do with their content, and robots.txt is where that decision is written down. Reading it by eye is error-prone: rules are matched by the longest pattern, groups can repeat, wildcards and $ anchors apply. This Actor applies the real matching rules to each crawler and tells you plainly whether it is ALLOWED, RESTRICTED or BLOCKED, and which rule decided it.

What makes it different?

• Real matching, not keyword search. Longest-match, Allow-beats-tie, wildcards, end anchors and merged groups, as in RFC 9309.
• Crawlers grouped by purpose. AI training, AI search and assistants, and classic search engines are reported separately, so blocks AI training but allows AI search is visible at a glance.
• Headline policy per domain, for example blocks-ai-training, open-to-ai-crawlers or no-robots-txt.
• Sitemaps and llms.txt in the same run: status, entry count and newest date for declared sitemaps.
• Unreadable robots.txt handled honestly. A site that refuses the request or is unreachable gets UNKNOWN for every crawler, not a guess. A missing file (404) means no restrictions; a server error means crawlers should assume everything is disallowed.

What data does the checker return?

Field	Example
domain, robotsStatus, robotsHttpStatus	nytimes.com, found, 200
policy	blocks-ai-training-and-ai-search
bots	per crawler: token, operator, category, verdict (ALLOWED, RESTRICTED, BLOCKED or UNKNOWN), explicit, decidedBy, crawlDelay
bots	per crawler: token, operator, category, verdict, explicit, decidedBy, crawlDelay
sitemapsDeclared, sitemaps	declared sitemaps with status, kind, entries and newest lastmod
llmsTxt	{ found: true, url, bytes }
assumption	what a crawler should assume given the robots.txt response

A SUMMARY record shows how many domains were checked and how many fall under each headline policy.

Crawlers checked

Group	Crawlers
AI training	GPTBot, ClaudeBot, anthropic-ai, CCBot, Google-Extended, Applebot-Extended, Bytespider, Meta-ExternalAgent, cohere-ai, Diffbot, Amazonbot
AI search and assistants	OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User, Claude-SearchBot, DuckAssistBot, YouBot
Search engines	Googlebot, Bingbot, Applebot, DuckDuckBot

Google-Extended and Applebot-Extended are control tokens, not separate crawlers: blocking them tells Google or Apple not to use your content for AI, without affecting search.

How to check which AI crawlers a site blocks

1. Paste domains or page URLs, one per line.
2. Choose which crawler groups to evaluate and, optionally, extra paths to test.
3. Click Start.
4. Read the policy and bots columns, or export JSON, CSV or Excel for a spreadsheet of many sites.

{
    "domains": [
        "nytimes.com",
        "en.wikipedia.org"
    ],
    "categories": [
        "training",
        "ai-search"
    ]
}

• domains: only the site's origin is used.
• testPaths: a crawler allowed on / but blocked on a path you list is RESTRICTED.
• categories: training, ai-search, search.

How much does an AI crawler check cost?

You pay per domain checked returned plus a very small start fee. There is no subscription.

Domains	Approximate cost
10	$0.10
100	$1
1,000	$10
5,000	$50

Paid Apify plans get tiered discounts, and you can set a maximum cost per run. You can try it on Apify's free plan without a card.

Who is it for?

• Publishers and SEO teams auditing their own AI-crawler policy.
• Agencies reporting on clients' AI visibility and exposure.
• Researchers and journalists surveying what many sites allow.
• AI and data teams checking which sources permit crawling.

Frequently asked questions

Does robots.txt actually stop AI companies? It is a voluntary convention. Major crawlers from OpenAI, Anthropic, Google and others say they honour it, but it cannot enforce anything, and it does not cover content already collected.

What is the difference between ALLOWED, RESTRICTED and BLOCKED? BLOCKED means the home page is disallowed for that crawler. RESTRICTED means it can fetch the home page but a rule limits some paths (or a path you tested). ALLOWED means no rule limits it.

Why does a site show UNKNOWN for every crawler? Some sites refuse automated requests even for robots.txt (for example with HTTP 403 or 418), or cannot be reached. The Actor then cannot read the rules, reports robots-txt-unreadable and marks every crawler UNKNOWN. It never guesses ALLOWED. (A plain 404 is different: no file means no restrictions, so crawlers are ALLOWED.)

Does blocking Google-Extended remove me from Google Search? No. Google-Extended only controls use of your content for Gemini AI features. Search is controlled by Googlebot.

Is a missing llms.txt a problem? No. It is an optional, emerging convention, and most crawlers do not rely on it yet.

Notes and limits

• Only public robots.txt, sitemap and llms.txt files are read. Private and internal addresses are refused.
• This is a technical reading of a public file, not legal advice about crawling or copyright.