AI Crawler Checker - GPTBot, ClaudeBot, robots.txt & llms.txt

Find out which AI crawlers a website lets in, in bulk: the robots.txt rules for GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 15 more crawlers, whether the site publishes an llms.txt, and whether it uses noai directives.

• 20 crawlers, by company and purpose. OpenAI, Anthropic, Google, Perplexity, Common Crawl, ByteDance, Apple, Meta, Amazon, Microsoft, Cohere, Mistral and DuckDuckGo, each tagged training, search or user (fetches made when a person asks a chatbot).
• Exact robots.txt logic. Rules are applied like the crawlers do (RFC 9309): the crawler's own group, else the * group, longest match wins, * and $ wildcards. Each crawler is allowed, partial (some paths blocked) or blocked (the whole site).
• llms.txt and noai. Detects a real /llms.txt file (not an HTML page answered with status 200) and noai / noimageai in meta robots or X-Robots-Tag.
• Cheap. Three light requests per site: $1 per 1,000 sites.

What can I use it for?

• AI visibility (GEO) audits: check that your site, or your clients' sites, does not block the crawlers of ChatGPT search, Perplexity or Claude by mistake.
• Publisher and market research: see which news sites or competitors block AI training or AI search crawlers.
• Compliance and policy reviews: document your AI crawler policy across all your domains.
• Lead generation for SEO/GEO agencies: find sites without llms.txt or that block AI search crawlers.

Input

{ "urls": ["nytimes.com", "lemonde.fr", "apify.com"] }

Output

One row per website. Example from a real run (lists shortened to 3 crawlers):

{
    "url": "https://lemonde.fr/",
    "robotsTxtUrl": "https://lemonde.fr/robots.txt",
    "robotsTxtStatus": 200,
    "robotsTxtFound": true,
    "sitemaps": ["https://www.lemonde.fr/sitemap_news.xml", "https://www.lemonde.fr/sitemap_index.xml"],
    "aiTrainingBlocked": false,
    "blockedCrawlers": ["ClaudeBot", "Claude-SearchBot", "Claude-User", "anthropic-ai", "Google-Extended", "CCBot", "Bytespider", "Applebot-Extended", "Amazonbot", "cohere-ai", "MistralAI-User", "DuckAssistBot"],
    "allowedCrawlers": [],
    "aiCrawlers": [
        { "bot": "GPTBot", "company": "OpenAI", "purpose": "training", "status": "partial", "disallowRules": 30, "ownGroup": false },
        { "bot": "ClaudeBot", "company": "Anthropic", "purpose": "training", "status": "blocked", "disallowRules": 1, "ownGroup": true },
        { "bot": "Google-Extended", "company": "Google", "purpose": "training", "status": "blocked", "disallowRules": 1, "ownGroup": true }
    ],
    "llmsTxt": { "found": false, "url": "https://lemonde.fr/llms.txt", "bytes": null },
    "metaRobots": "index, follow, noarchive",
    "xRobotsTag": null,
    "noAiDirective": false,
    "summary": "12 of 20 crawlers blocked (ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, CCBot...).",
    "error": null
}

Field	Meaning
aiCrawlers[].status	allowed, partial (some paths disallowed) or blocked (the home page is disallowed)
aiCrawlers[].ownGroup	true when the robots.txt names this crawler; false when the * rules apply
aiTrainingBlocked	true only when every training crawler is fully blocked
llmsTxt.found	A real llms.txt file exists at the site root

If robots.txt answers with a server error (5xx), compliant crawlers treat the whole site as blocked, and so does this Actor.

Pricing

$0.001 per site checked ($1 per 1,000 sites) plus Apify's small run-start fee. No subscription, lower prices on higher Apify plans. Invalid inputs, unreachable sites and internal addresses are free.

Good to know

robots.txt is a request, not a lock: it tells you a site's policy, not what every crawler actually does. Some publishers also block AI crawlers at the firewall level, which robots.txt does not show.

Use it from AI agents (MCP)

Add ai-crawler-checker to the Apify MCP server in your MCP client (Claude, Cursor, VS Code...).

Integrations

Apify API, schedules, webhooks, Make, Zapier, n8n. Export as JSON, CSV, Excel or HTML.

Related tools

Same developer, same pay-per-result pricing, outputs that chain together:

• SEO Audit Tool: on-page SEO of many pages, with an A-F grade and fixes.
• Sitemap URL Extractor: every URL of a website from its sitemaps.
• Web Page to Markdown for LLM & RAG: clean Markdown of web pages for LLMs and RAG.

FAQ

Need the full on-page SEO too? Use SEO Audit. For the sitemap URLs, use Sitemap URL Extractor.

Something wrong or missing? Open an issue on the Issues tab.