AI Crawler Access Checker — robots.txt & llms.txt Audit

Can ChatGPT, Claude and Perplexity actually see your website? If AI crawlers are blocked, your brand cannot appear in AI answers — and AI search is where a growing share of buying decisions starts.

This Actor audits any list of websites and reports, per site:

• ✅ Which of 16 AI crawlers are allowed, restricted or fully blocked by robots.txt — OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic (ClaudeBot, Claude-SearchBot, Claude-User), Perplexity (PerplexityBot, Perplexity-User), Google-Extended, Applebot-Extended, Amazonbot, CCBot, Bytespider, meta-externalagent, DuckAssistBot, MistralAI-User
• 📄 Whether the site publishes llms.txt and llms-full.txt (the emerging standard for AI-readable site descriptions)
• 🗺️ Declared sitemaps
• 📊 Two scores: AI Access Score (all crawlers) and AI Search Score (live-search & user-fetch bots only — the ones that put you inside AI answers)

Who is this for

• SEO / GEO / AEO specialists running AI-visibility audits for clients
• Marketing teams checking whether their own site (or competitors) is open to AI search
• Agencies doing bulk audits — pass hundreds of domains in one run

Why it matters

Blocking training bots (GPTBot, CCBot) is a legitimate choice — but many sites unknowingly also block search and user-fetch bots (OAI-SearchBot, Perplexity-User, Claude-User). That removes them from AI answers entirely while competitors stay visible. This Actor makes the difference explicit: the purpose field separates training, search and user-fetch bots, so you can fix the rules that cost you AI traffic without giving your content away for training.

Input

{
    "websites": ["example.com", "competitor.com"],
    "maxConcurrency": 10
}

Output (one item per website)

{
    "website": "https://example.com",
    "aiAccessScore": 75,
    "aiSearchScore": 100,
    "robotsTxt": { "exists": true, "status": 200 },
    "llmsTxt": { "exists": false, "status": 404 },
    "llmsFullTxt": { "exists": false, "status": 404 },
    "sitemaps": ["https://example.com/sitemap.xml"],
    "blockedBots": ["GPTBot", "CCBot", "Bytespider", "meta-externalagent"],
    "bots": [
        { "bot": "GPTBot", "vendor": "OpenAI", "purpose": "training", "verdict": "blocked", "matchedRule": "specific" },
        { "bot": "OAI-SearchBot", "vendor": "OpenAI", "purpose": "search", "verdict": "allowed", "matchedRule": "none" }
    ],
    "summary": "Blocks 4/16 AI crawlers: GPTBot, CCBot, Bytespider, meta-externalagent."
}

FAQ

Does it execute JavaScript or crawl pages? No — it reads robots.txt, llms.txt, llms-full.txt only. Fast and cheap.

Is this legal? Yes. robots.txt and llms.txt are public, machine-readable files intended to be fetched.

How accurate is the verdict? Rules are matched per RFC 9309: a specific User-agent group beats the * wildcard; Disallow: / means full block; partial paths are reported as partial (root still accessible).