AI Crawler Access & llms.txt Checker

What does AI Crawler Access & llms.txt Checker do?

AI search and assistants can only cite pages their crawlers are allowed to read. This Actor tells you, for any list of websites, which AI crawlers are allowed and which are blocked, and whether the site is ready for AI search.

For each website it reports:

• Allow / block decision for 27 crawlers, grouped by purpose:
  • AI training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider, cohere-ai and more
  • AI search: OAI-SearchBot, Claude-SearchBot, PerplexityBot, YouBot
  • Assistant fetchers (when a user asks an assistant to open a page): ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot, MistralAI-User and more
  • Classic search: Googlebot, Bingbot, Applebot
• Whether each decision comes from a bot-specific rule or the wildcard (*) rule, and whether a bot is only partially blocked
• llms.txt and llms-full.txt: present or not, size, title and number of links
• Content-Signal lines in robots.txt (search, ai-input, ai-train)
• noai / noimageai directives in the meta robots tag or X-Robots-Tag header
• A one-line AI policy summary, such as "Blocks AI training but allows AI search and assistants"

Who is it for?

• SEO and GEO (generative engine optimization) agencies checking that clients are not accidentally blocking AI search
• Publishers and brands auditing their own sites and competitors' AI policies
• Researchers measuring AI crawler blocking across many domains
• Sales teams finding sites that need GEO help (blocked AI search, no llms.txt)

How to use it

1. Enter websites or specific page URLs.
2. Run. Filter by aiPolicySummary, blockedBots or llmsTxt.found.

Input example

{ "websites": ["nytimes.com", "apify.com", "wordpress.org"] }

Output example (shortened)

{
  "domain": "apify.com",
  "aiPolicySummary": "Open: all listed AI crawlers allowed.",
  "blockedBots": [],
  "llmsTxt": { "found": true, "sizeBytes": 16837, "title": "Apify", "linkCount": 73 },
  "contentSignals": ["search=yes, ai-input=yes, ai-train=yes"],
  "bots": [{ "bot": "GPTBot", "operator": "OpenAI", "purpose": "training", "allowed": true, "matchedBy": "wildcard (*) rule" }]
}

How much does it cost?

Event	Price
Actor start	$0.001 per run
Website checked	$0.006 per website

1,000 websites cost about $6. Unreachable websites are free.

Notes

• Decisions follow the robots.txt standard (RFC 9309): the most specific user-agent group applies, and the longest matching path rule wins.
• robots.txt is a request, not a technical block. Some sites also block bots at the firewall, which robots.txt does not show.
• Only robots.txt, llms.txt, llms-full.txt and the page you enter are fetched. Crawler and company names are used only to identify the bots; this tool is not affiliated with any of them.