AI Crawler Blocking Checker: GPTBot, ClaudeBot & 15 More

See which AI crawlers each website allows or blocks in robots.txt (GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, Applebot-Extended, Meta...), plus search bots, crawl-delay, sitemaps and llms.txt presence.

Why this actor

• Up-to-date AI bot list with per-bot verdicts; llms.txt detection.
• Pay only for results: $1.00 per 1,000 sites, no start fee, failed lookups are free.
• Robust: automatic retries with exponential backoff, respects Retry-After, per-item error rows instead of crashed runs.
• Clean, flat JSON/CSV/Excel output ready for Google Sheets, Make, Zapier, n8n, Clay, or the Apify API.

Use cases

• Publishers auditing their AI-training opt-outs
• AI companies checking crawl permissions before ingestion
• Media research on AI blocking trends
• SEO agencies (AI search visibility)

How to use AI Crawler Blocking Checker: GPTBot, ClaudeBot, Google

1. Configure sites, paths, extraUserAgents in the Input tab (or leave the prefilled sample values to test immediately).
2. Optionally set maxItems to cap the number of results and control spend.
3. Click Start and export structured results as JSON, CSV, Excel, or connect directly via API / MCP / n8n / Make.

Input

Field	Type	Description
sites	array	Domains to check.
paths	array	A bot counts as allowed only if all of these paths are fetchable. Default '/'.
extraUserAgents	array	Any other crawler names to test.
maxItems	integer	0 = no limit. Caps cost.

Example input:

{
  "sites": [
    "nytimes.com",
    "reddit.com",
    "apify.com"
  ]
}

Output

Each dataset item looks like this (real sample from a test run, long fields trimmed):

{
  "site": "https://nytimes.com",
  "robotsStatus": 200,
  "hasRobotsTxt": true,
  "robotsBytes": 8547,
  "aiBotsBlocked": [
    "GPTBot",
    "ChatGPT-User",
    "OAI-SearchBot",
    "\u2026"
  ],
  "aiBotsAllowed": [
    "Amazonbot",
    "MistralAI-User"
  ],
  "aiBotsExplicitlyNamed": [
    "GPTBot",
    "ChatGPT-User",
    "OAI-SearchBot",
    "\u2026"
  ],
  "blocksAllAI": false,
  "searchBotsBlocked": null,
  "wildcardDisallowAll": false,
  "crawlDelay": null,
  "sitemaps": [
    "https://www.nytimes.com/sitemaps/new/news.xml.gz",
    "https://www.nytimes.com/sitemaps/new/sitemap.xml.gz",
    "https://www.nytimes.com/sitemaps/new/collections.xml.gz",
    "\u2026"
  ],
  "hasLlmsTxt": false,
  "perBot": {
    "GPTBot": false,
    "ChatGPT-User": false,
    "OAI-SearchBot": false,
    "ClaudeBot": false,
    "Claude-User": false,
    "anthropic-ai": false,
    "Google-Extended": false,
    "PerplexityBot": false,
    "CCBot": false,
    "Bytespider": false,
    "Applebot-Extended": false,
    "Meta-ExternalAgent": false,
    "Amazonbot": true,
    "cohere-ai": false,
    "Diffbot": false,
    "YouBot": false,
    "DuckAssistBot": false,
    "MistralAI-User": true,
    "Googlebot": true,
    "Bingbot": true,
    "DuckDuckBot": true,
    "YandexBot": true,
    "Baiduspider": true,
    "Applebot": true
  },
  "pathsChecked": [
    "/"
  ],
  "scrapedAt": "2026-10-14T16:24:26+00:00"
}

Rows that could not be processed contain an error field and are not charged.

Extracted output fields

Every dataset row includes structured fields ready for spreadsheets, CRMs, and AI pipelines:

• Fields: site, robotsStatus, hasRobotsTxt, robotsBytes, aiBotsBlocked, aiBotsAllowed, aiBotsExplicitlyNamed, blocksAllAI, searchBotsBlocked, wildcardDisallowAll, crawlDelay, sitemaps, hasLlmsTxt, perBot, pathsChecked, scrapedAt

Comparison: AI Crawler Blocking Checker: GPTBot, ClaudeBot, Google vs alternatives

Feature	This Actor (shawndunn/ai-crawler-blocking-checker)	Typical alternatives
Pricing model	$1 / 1,000 results, $0.00 start fee	$20–$49/mo subscription or per-run start fees
Failed / error lookups	$0.00 (never charged)	Often billed by compute time even on failures
API keys / proxies required	None (works out of the box)	Often requires paid third-party keys or residential proxies
AI Agent & MCP ready	Yes (mcp.apify.com compatible, optional defaults)	Strict required schemas that reject agent calls

Pricing

Pay-per-event: $0.001 per site (= $1.00 / 1,000). Use Max results to cap spend. Apify's free plan credit covers thousands of results per month.

Integrations & API

Run on a schedule in Apify Console, call via the Apify API / Python & JS clients, or connect to Make, Zapier, n8n, Google Sheets, Slack and webhooks. Works as a tool for AI agents via Apify MCP.

Use with AI agents (MCP)

AI Crawler Blocking Checker: GPTBot, ClaudeBot, Google is configured for zero-config AI agent tool use via the Apify MCP Server (no required-field blocking on defaulted inputs).

Streamable HTTP endpoint:

https://mcp.apify.com/?tools=shawndunn/ai-crawler-blocking-checker

Claude Desktop / Cursor / Windsurf mcpServers config:

{
  "mcpServers": {
    "apify-ai-crawler-blocking-checker": {
      "url": "https://mcp.apify.com/?tools=shawndunn/ai-crawler-blocking-checker",
      "headers": {
        "Authorization": "Bearer <YOUR_APIFY_TOKEN>"
      }
    }
  }
}

Direct synchronous API call:

POST https://api.apify.com/v2/acts/shawndunn~ai-crawler-blocking-checker/run-sync-get-dataset-items?token=<YOUR_APIFY_TOKEN>

FAQ

Is this an official tool? No. It is an independent actor and is not affiliated with, sponsored or endorsed by the data source or any trademark owner named here.

Is it legal? The actor only reads publicly available data from public endpoints, without logins, and does not bypass paywalls or access controls. You are responsible for using the data in line with applicable laws (e.g. GDPR) and the source sites' terms.

Something broke? Open an issue in the Issues tab; we typically respond within 24 hours and ship fixes fast.

More tools by shawndunn

• Bulk PageSpeed Insights & Core Web Vitals Checker (Lighthouse) — Audit hundreds of URLs with Google PageSpeed Insights: performance, SEO, accessibility and best-practices scores, LCP, CLS, TBT ($3/1k, no start fee).
• Sitemap URL Extractor — Extract every URL from any website's XML sitemaps: auto-discovery via robots.
• Broken Link Checker & Bulk URL Status Checker — Crawl a website and test every internal and external link, or bulk-check a list of URLs (url, statusCode, ok, finalUrl) ($1/1k, no start fee).
• Hugging Face Hub Explorer: Models, Datasets & Spaces — List and search Hugging Face models, datasets or Spaces with downloads, likes, trending score, pipeline task, library, license ($0.
• Wikipedia Article Extractor: Clean Text, Categories, Wikidata ID — Get clean plain text (intro or full article) for Wikipedia articles in any language, plus categories, Wikidata ID, short ($0.