AI Crawler Checker – Can GPTBot, ClaudeBot & AI Agents Access Your Site?

Which AI systems can actually read a website? This Actor checks robots.txt rules for 17 AI crawlers, AI agents and search bots, detects bot walls (HTTP 403 for automated clients) and looks for llms.txt, sitemap and structured data – in bulk.

Who uses it

• SEO / GEO agencies: audit AI visibility for clients and prospects in one run.
• E-commerce teams: make sure ChatGPT and AI shopping agents can read your product pages.
• Researchers & journalists: measure how a whole market treats AI crawlers.

What you get

• Status per agent: allowed, partially_blocked (e.g. product paths) or blocked, plus the matching rule group
• Agents: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Bingbot, Applebot, Applebot-Extended, meta-externalagent, Bytespider, CCBot, Amazonbot
• aiAgentsBlocked list and count, searchEnginesBlocked
• Homepage HTTP status for bots (detects Cloudflare/Akamai bot walls)
• llms.txt present, sitemap URL, JSON-LD types on the homepage
• Shop platform detection

How to use it

1. Click Try for free (or Start).
2. Paste your list into the input (see the example below).
3. Run it and download the results as JSON, CSV or Excel, or get them via API.

Example input

{
  "shops": [
    "example.com",
    "another-shop.de"
  ]
}

robots.txt shows the stated policy; the homepage check shows whether bots are actually let in.

Example output (shortened, illustrative values)

{
  "shop": "https://another-shop.de",
  "platform": "Shopify",
  "robotsTxtFound": true,
  "aiAgentsBlocked": [
    "GPTBot",
    "CCBot"
  ],
  "aiAgentsBlockedCount": 2,
  "searchEnginesBlocked": [],
  "homepageHttpStatus": 200,
  "homepageReachableForBots": true,
  "llmsTxtFound": false,
  "homepageJsonLdTypes": [
    "Organization",
    "WebSite"
  ]
}

Pricing

Pay only for results: $5.00 per 1,000 sites (pay per event, no monthly fee). Apify's free plan includes monthly credits, so you can test it for free.

Volume	Price
100 websites	$0.50
1,000 websites	$5.00

Use it via API, n8n, Make or as an AI tool

• API (one call, results back):

curl -X POST "https://api.apify.com/v2/acts/fast_data_asap~ai-crawler-access-checker/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"shops": ["example.com", "another-shop.de"]}'

• n8n / Make / Zapier: use the Apify integration (Run Actor → Get dataset items) or an HTTP request with the URL above.
• Claude, ChatGPT, Cursor (MCP): add https://mcp.apify.com?tools=fast_data_asap/ai-crawler-access-checker as an MCP server, then just ask in plain language.
• RapidAPI: also available as an endpoint of the DACH E-Commerce Data API.

FAQ

What did you find on 100 German shops?
In our test of 100 large DACH shops (Nov 2026), only 9 blocked AI agents in robots.txt, but 12 returned HTTP 403 to automated clients, and 33 had an llms.txt (20 of 24 Shopify stores). Write-up on dev.to (fast_data_asap).

Does "allowed" mean the AI can read my site?
Only if the homepage check also passes. A bot wall (403) blocks crawlers even when robots.txt allows them.

How do I add an llms.txt?
For Shopify stores use the Shopify llms.txt Generator.

Related Fast Data Actors

• Shopify llms.txt Generator
• German Shop Compliance Checker
• DACH Shop Lead Profiler
• All Fast Data Actors

🇩🇪 Auf Deutsch: KI-Crawler-Check für Websites

Prüft, ob GPTBot, ClaudeBot, PerplexityBot, Google-Extended und andere KI-Crawler und KI-Agenten eine Website lesen dürfen (robots.txt) und ob sie technisch überhaupt durchkommen (Bot-Sperre/403). Dazu: llms.txt vorhanden?, Sitemap, strukturierte Daten. Für SEO-/GEO-Agenturen und Shops, die in ChatGPT & Co. sichtbar sein wollen.

Stichworte: GPTBot blockieren prüfen, KI Crawler robots.txt Check, Sichtbarkeit in ChatGPT prüfen.

---

Built and maintained by Fast Data. Questions or a missing feature? Open an issue on this page.