AI Crawler Reality Check

Your robots.txt says GPTBot is welcome. Your server disagrees. You will never find out from robots.txt.

Most AI-visibility tools read robots.txt and tell you what it says. But robots.txt is a declaration, not behaviour. The actual blocking usually happens one layer above it — Cloudflare, a WAF, bot-fight mode, a rate limit keyed on User-Agent. A site can allow GPTBot in robots.txt and still return 403 to every single one of its requests.

This Actor sends live requests as 12 real AI crawlers and compares what your site permits with what it actually serves.

A real example, not a hypothetical

Running this on notion.com returns:

robots.txt allows, but the server does NOT serve this bot (Bingbot: HTTP 403, bot challenge page)

Notion's robots.txt welcomes Bingbot. Their edge returns a challenge page instead. Bing powers Copilot. Nothing in a robots.txt audit would ever show this.

What you get

1. Crawler matrix — for each of 12 bots: what robots.txt declares, what the server actually returned, and whether the response was a bot challenge:

Bot	robots.txt	Live response	Gap
GPTBot	allowed	200	—
ClaudeBot	allowed	200	—
Bingbot	allowed	403 challenge	allows, but does not serve

Bots tested: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Bingbot, Amazonbot, CCBot, meta-externalagent.

2. Extractability — being fetched is not the same as being quoted. For each page: JSON-LD presence and schema types, author attribution, dateModified, FAQ blocks, question-style headings, visible text length, and whether the content only exists after JavaScript runs.

3. llms.txt detection — present or not, and its size.

An honest control that other tools skip

Before comparing bots, the Actor fetches your site with an ordinary browser User-Agent.

If that request also fails, the run is marked inconclusive and explains why — because a site that returns 403 to everyone from a datacenter IP is not running an AI-crawler policy, it is blocking the network. Reporting "12 crawlers blocked" in that case would be a lie.

Verified on openai.com: it returns 403 to GPTBot and to Chrome from the same network. Without the control fetch, this tool would have reported twelve fake findings.

When to use this

• Your content stopped appearing in ChatGPT, Perplexity or AI Overviews and you do not know why
• You added a WAF or turned on bot protection and want to know what it broke
• You are doing GEO/AEO work and need evidence, not assumptions
• You maintain a site for a client and want a before/after artifact

Input

{ "urls": [{ "url": "https://example.com/your-article" }], "additionalPages": ["https://example.com/pricing"] }

Check the exact pages you care about, not just the homepage — protection rules often differ per path.

Limits, stated plainly

• Runs from a datacenter IP. Some sites treat that differently than residential traffic; that is exactly what the control fetch is there to reveal.
• Tests whether a crawler is served, not whether an LLM chose to cite you. No public API exposes citation data; anyone claiming otherwise is guessing.
• robots.txt parsing covers the directives crawlers actually honour, not every edge case of the spec.

---

Built by an engineer who runs a multi-agent system in production and got tired of tools that report the manifest instead of the behaviour.