AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked?

Is your site (or your clients' sites) visible to ChatGPT, Claude, Perplexity and Meta AI, or quietly blocked from them? Give this Actor a list of websites and it tells you, for each one and each AI crawler, whether the crawler is allowed in, and whether it is robots.txt or a firewall doing the blocking. Your own columns stay next to the answers.

What you get for every website

• Verdict: open, partly_blocked, all_blocked or unverified (the site refused our requests, so nothing could be confirmed).
• Per crawler: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Meta-ExternalAgent by default (ChatGPT-User, Google-Extended, Applebot-Extended and CCBot are optional). For each: the robots.txt verdict, which rule group applied, and a real fetch with the crawler's own user agent compared with a normal browser.
• Blocked by what: robots.txt, firewall (a CDN or bot-protection rule answers 403 or a challenge page) or both.
• Search vs training: blocksAiSearch (OAI-SearchBot, PerplexityBot, ChatGPT-User) and blocksTraining (GPTBot, ClaudeBot, Meta, CCBot, Google-Extended) as separate flags, because blocking training and blocking AI search are different decisions.
• llms.txt and llms-full.txt present or not, plus noai / noimageai directives and the CDN in front of the site.
• Fixes: plain sentences saying what is blocked and what to change.

Monitor mode: get told when it changes

Switch on Monitor mode and schedule the run (daily or weekly). The first run records each site's starting point. Every later run reports only what changed: a crawler newly blocked or unblocked in robots.txt, a firewall rule that started or stopped refusing a bot, llms.txt added or removed. Set a Webhook URL (Slack, Zapier, Make, n8n) and it is only called when something changed. Use Only changed sites to keep just those rows. Different watchlists use different Monitor names.

Who uses this

• Agencies and consultants auditing a client list for AI visibility before selling "AI search optimisation".
• Shop owners and publishers who read on Reddit that Meta, OpenAI or Perplexity bots hammer their site, want to block some, and want proof the block works.
• SEO teams who need to know that nobody's CDN update quietly blocked OAI-SearchBot.

Input

Give websites as an Apify dataset, a CSV or Google Sheet link (shared as "anyone with the link can view"), a plain list of URLs, or inline rows. The URL you give is the page tested, so use the homepage. Set Website field if the column is not called website, url or domain; it is detected automatically otherwise.

Example (the default input):

{ "urls": ["https://www.nytimes.com", "https://www.bbc.co.uk", "https://example.com"] }

Output (one row per website)

{
  "checkedUrl": "https://www.bbc.co.uk",
  "checkStatus": "ok",
  "verdict": "all_blocked",
  "summary": "Blocks every checked AI crawler. No llms.txt.",
  "blockedBots": ["GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot", "Meta-ExternalAgent"],
  "blocksAiSearch": true,
  "blocksTraining": true,
  "gptbotRobots": "blocked",
  "gptbotFetch": "served",
  "llmsTxtFound": false,
  "monitorStatus": "unchanged"
}

...Fetch values: served, blocked (the browser got the page, the crawler did not), not_comparable (the browser was refused too), error, not_tested.

Pricing

Pay per event, no subscription:

• $0.005 per URL checked (about $5 per 1,000 sites). Lower on higher Apify plans.
• $0.01 per CSV or Excel export file, only when you ask for one.
• $0.01 per webhook delivered (confirmed 2xx only; in monitor mode only when something changed).

You are never charged for unreachable sites, invalid addresses, duplicates, or rows skipped because of a time or budget limit. Watching 200 sites daily for a month is about $30.

Honest limits

• robots.txt verdicts are exact. The fetch test is a request that looks like the crawler, sent from an Apify data centre. Some bot protection (Cloudflare verified bots, for example) lets the real crawler in by checking its company IP ranges, so a firewall block here can be a false alarm for the genuine crawler, and served does not prove the genuine crawler is let in. Treat firewall results as "worth checking", robots.txt results as fact. A firewall block is only reported when a second try, a couple of seconds later, is refused too.
• Sites that refuse every automated request (some big publishers) come back unverified or not_comparable, not guessed.
• It makes about six light requests per site and never more than two at once to the same site. It reads the page you name, not the whole site.
• llms.txt is an optional convention; no major AI company has confirmed it affects crawling.

FAQ

Does it work for my own site? Yes, that is the main use. Nothing is changed on any site; it only reads.

Can an AI agent call it? Yes. Pay per event works through Apify's MCP server and x402, and the output is plain JSON.

Something looks wrong for one site. Open an issue on the Actor page with the URL and I will look the same day.

If this saved you a manual robots.txt check, a review on the Store helps a lot.