# AI crawler access checker — GPTBot ClaudeBot robots (`kondasviktor/ai-crawler-access-checker`) Actor

Check whether major AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, OAI-SearchBot, …) are allowed by robots.txt — and flag likely WAF/CDN blocks on sample URLs.

- **URL**: https://apify.com/kondasviktor/ai-crawler-access-checker.md
- **Developed by:** [Viktor Kondas](https://apify.com/kondasviktor) (community)
- **Categories:** SEO tools, AI
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 site checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI crawler access checker — GPTBot ClaudeBot robots

**Find out which AI crawlers may read a website.** Paste domains or URLs and get one policy verdict per site — are GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and the other AI bots allowed, blocked, or blocked only on some paths — checked against robots.txt the way the standard (RFC 9309) defines it, with the matching rule and line number as evidence. For SEO/GEO agencies, publishers and site owners. JSON, CSV or Excel, by API, on a schedule, or from n8n, Make, Zapier and AI agents (MCP). **$0.005 per site checked**, no subscription.

> Independent tool. Not affiliated with OpenAI, Anthropic, Google, Perplexity, Apple, Meta or Common Crawl. Reads publicly available files only (`/robots.txt`, `/llms.txt`). **A robots.txt verdict is not proof that a CDN or firewall lets the real crawler in** — see "What the verdict does and does not tell you".

### What does AI crawler access checker do?

- Answers the question behind most AI-visibility audits: **am I opted out of AI *training* but still visible to AI *search*?** Every bot is tagged `training`, `search`, `user-fetch` or `ads`, and each site gets a policy: `open`, `blocks_training_only`, `blocks_ai_search`, `blocks_all`, `mixed` or `partial`.
- Evaluates robots.txt **correctly**: wildcards (`*`), end anchors (`$`), longest match wins, Allow beats Disallow on a tie, several groups for one bot merged, the first 500 KiB only — and tells you **which rule and which line** decided each bot and each sample path (`/`, `/blog/`, `/products/` …).
- Catches the traps: rules for the legacy `anthropic-ai` / `Claude-Web` tokens that don't govern ClaudeBot, `Google-Extended` and `Applebot-Extended` being control tokens that never show up in your logs, a `*` block that is not AI-specific, and robots.txt answers that are really an HTML page or a firewall challenge.
- Notes whether `/llms.txt` exists and looks like Markdown (presence and shape only), reads `Content-Signal` directives, and flags the CDN it can see.
- **Monitors:** schedule it and every result says whether the policy `changed` since the last run ("GPTBot: allowed → blocked").
- Optional, opt-in **WAF probe** for sites you own — see the limits below.

### Who is it for?

| You are… | You use it to… |
|---|---|
| SEO / GEO / AEO agency | Start every AI-visibility audit with a verified access check for a client and its competitors |
| Publisher or media group | Confirm the opt-out policy is live on every property and benchmark peers |
| In-house SEO or platform team | Catch the accidental block after a CMS, CDN or firewall change — on a weekly schedule |
| E-commerce or ads team | Check that ChatGPT search, user-fetch and ads crawlers may fetch product and landing pages |
| AI-agent builder | Ask "may agent X fetch this URL?" before fetching |

### Ready-made examples (one click)

| Example | What you get |
|---|---|
| [Check if GPTBot and ClaudeBot can crawl your site](https://apify.com/kondasviktor/ai-crawler-access-checker/examples/check-gptbot-claudebot-access) | Verdict and matching rule for the main training crawlers on four public sites |
| [Audit AI search crawler access for GEO clients](https://apify.com/kondasviktor/ai-crawler-access-checker/examples/audit-ai-search-visibility-robots) | OAI-SearchBot, PerplexityBot, Claude-SearchBot and user-fetch bots per site |
| [Benchmark AI crawler opt-out policies across sites](https://apify.com/kondasviktor/ai-crawler-access-checker/examples/benchmark-ai-crawler-opt-out-policies) | One policy verdict per site for a whole list |
| [Monitor robots.txt AI crawler policy changes weekly](https://apify.com/kondasviktor/ai-crawler-access-checker/examples/monitor-ai-crawler-policy-changes) | Schedule-ready watchlist with a `change` flag |
| [Check ChatGPT search, user-fetch and ads bot access](https://apify.com/kondasviktor/ai-crawler-access-checker/examples/check-chatgpt-search-and-ads-bot-access) | OAI-SearchBot, ChatGPT-User and OAI-AdsBot per path |

### How to use it

1. Paste domains or URLs, one per line (or open an example above). A path in a URL (`https://example.com/blog/`) is checked as an extra sample path for that site.
2. Optionally pick the bots and the sample paths. Leave the bots empty for the default set.
3. Set **Max sites** — this is also your cost cap — and click **Start**. Filter the dataset by `policy`, or download it.

### Input

| Field | Required | Description |
|---|---|---|
| `sites` | yes | Domains or URLs. One result per site (scheme + host) |
| `bots` | no | Which AI crawlers to evaluate. Default: GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, meta-externalagent, meta-externalfetcher. Optional extras: Applebot, Amazonbot, Bytespider, DuckAssistBot, MistralAI-User |
| `customBots` | no | Any other robots.txt user-agent token, e.g. a crawler announced after this Actor was published |
| `samplePaths` | no | Paths to evaluate for every bot (default `/`, max 10). A bot allowed on some but not all gets `partial` |
| `checkLlmsTxt` | no | One extra request per site: is there an `/llms.txt`? Default on |
| `maxSites` | no | Cap on paid results and cost. Default 100 |
| `wafProbe`, `probeBots` | no | **Opt-in**, off by default. For sites you own or may test. See below |
| `respectRobots` | no | Skip `/llms.txt` and probe URLs when the site's robots.txt disallows this Actor (`AICrawlerAccessChecker`). Default on |
| `maxConcurrency`, `requestTimeoutSecs` | no | Sites in parallel (1–5, default 3) and per-request timeout. At least 0.7 s between requests to one site |
| `trackHistory`, `historyStoreName` | no | Remember verdicts in a named key-value store **in your account** and add `change` to the next run |

### Output

One row per site (abridged real result for `jsr.io`, 3 Nov 2026):

```json
{
  "site": "https://jsr.io",
  "domain": "jsr.io",
  "robotsUrl": "https://jsr.io/robots.txt",
  "robotsStatus": "found",
  "policy": "blocks_training_only",
  "botsChecked": 14,
  "allowedCount": 8,
  "blockedCount": 6,
  "partialCount": 0,
  "blockedBots": ["GPTBot", "ClaudeBot", "Google-Extended", "Applebot-Extended", "CCBot", "meta-externalagent"],
  "bots": [
    { "bot": "GPTBot", "vendor": "OpenAI", "purpose": "training", "verdict": "blocked", "matchedBy": "bot", "rule": "Disallow: /", "ruleLine": 57 },
    { "bot": "OAI-SearchBot", "vendor": "OpenAI", "purpose": "search", "verdict": "allowed", "matchedBy": "wildcard", "rule": "Allow: /", "ruleLine": 32 }
  ],
  "contentSignals": { "search": "yes", "ai-train": "no", "use": "reference" },
  "llmsTxtPresent": false,
  "cdnHint": "cloudflare",
  "explanation": "Opted out of AI training crawlers (GPTBot, ClaudeBot, …) while AI search and user-fetch bots (OAI-SearchBot, ChatGPT-User, …) are allowed. This is what robots.txt says — it does not prove the CDN/WAF lets the real crawler in.",
  "status": "ok",
  "scrapedAt": "2026-11-03T18:57:54.314Z",
  "schemaVersion": 1
}
```

| Field | Meaning |
|---|---|
| `policy` | `open` · `blocks_training_only` · `blocks_ai_search` · `blocks_all` · `mixed` · `partial` (some paths off limits, nothing blocked entirely) · `unknown` (no verdict possible) |
| `bots[].verdict` | `allowed`, `blocked` or `partial` across the sample paths; `matchedBy` says whether the bot's own group, the `*` group or no group decided |
| `robotsStatus` | `found`, `empty`, `not_found_allow_all` (404/410: no robots.txt, everything allowed) — **charged**; `fetch_blocked`, `server_error`, `unreachable`, `not_robots`, `blocked_host`, `invalid_input` — **free**, with an `explanation` |
| `legacyTokenWarnings` | Rules for deprecated tokens that don't protect you |
| `change` / `changeDetail` | `baseline`, `unchanged`, `changed` (a verdict flipped), `robots_edited` — with change tracking on |
| `wafProbe`, `wafBlockedBots` | Only when the probe is on |
| `status` | `ok`, `partial` (charged, but llms.txt or the probe could not be completed) or `not_found` (free) |

### What the verdict does and does not tell you

- **It is what robots.txt says**, evaluated per RFC 9309. Reputable AI crawlers say they honour robots.txt; the operators' own documents also say some **user-initiated fetchers** (for example ChatGPT-User, Perplexity-User, meta-externalfetcher) may not — those rows carry that hint.
- **It does not prove access.** A CDN, firewall or bot-management rule can block a crawler that robots.txt allows (and the other way round). Login walls, paywalls and JavaScript-only pages are out of scope.
- **Bot names change.** The registry in the Actor follows the operators' public documentation as of November 2026; entries marked optional were not verified. Add a new crawler with `customBots` the day it appears.
- **Not checked:** `<meta name="robots">` / `X-Robots-Tag` directives (such as `noai`), `ai.txt`, sitemaps, rate limits, whether a crawler really visited.

#### The optional WAF probe — read before using

Off by default and meant **only for sites you own or are allowed to test**. With `wafProbe` on, the Actor requests up to 3 sample URLs once with a neutral user-agent (its own) and once for each probe bot with an approximate User-Agent string containing that bot's token, then compares status codes and known challenge-page markers (Cloudflare, Akamai, Imperva, DataDome, PerimeterX, AWS WAF, Sucuri, Vercel …). It reports bots a **user-agent rule or challenge page** treats differently, and says when the neutral request is blocked too (inconclusive).

It **cannot** prove what the real crawler experiences: it runs from an Apify datacenter IP, and CDNs may verify crawlers by their published IP ranges or reverse DNS. A clean result is not proof the real crawler gets in; a block is not proof the real crawler is blocked. To stay polite it only sends a bot's user-agent to URLs that bot may fetch according to robots.txt, and it skips paths that this Actor's own user-agent may not fetch (unless you turn `respectRobots` off). The probe adds no extra charge.

### How much does it cost?

Pay per event — you only pay for results delivered.

| Event | Price | When |
|---|---|---|
| Actor start | Apify default | Once per run |
| Site checked | $0.005 | A site whose robots.txt was read and evaluated (includes the llms.txt check and the optional probe) |

**Free:** sites that cannot be read — unreachable, server errors, a robots.txt request blocked for this Actor (401/403/429), a "robots.txt" that is really an HTML page, private hosts, invalid input. Every free row says why.

**Examples:** 100 sites ≈ $0.50 · 1,000 sites ≈ $5 · a 50-site watchlist run weekly for a month ≈ $1. Set **Max sites** (or Max cost per run) as your cap. Higher Apify plans get Store discounts automatically.

### Schedule it

Weekly (Monday morning) is enough for a watchlist: CDN, CMS and bot-management changes are what flip verdicts. With change tracking on, the dataset tells you which sites `changed` and exactly what ("GPTBot: allowed → blocked"). Use a different `historyStoreName` per watchlist.

### Use with AI agents (MCP)

Connect the Apify MCP server and ask your agent:

> "Use kondasviktor/ai-crawler-access-checker to check whether GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot may crawl example.com and example.org, and tell me which of them are opted out of AI training but still open to AI search."

### FAQ

**Is a site that "blocks GPTBot" invisible to ChatGPT?** Not necessarily. According to OpenAI's crawler documentation, GPTBot is the training crawler; OAI-SearchBot is the one that affects ChatGPT search. That is why results are tagged by purpose.

**Why was my site a free row?** The Actor could not read a trustworthy robots.txt (see `robotsStatus` and `explanation`). `fetch_blocked` usually means a firewall answered the Actor with 403/429 or a challenge page — the Actor never reports that as "allowed".

**What is `partial`?** The bot may crawl some of your sample paths but not others (for example `/` allowed, `/gist/` disallowed). Add the sections you care about to `samplePaths`.

**Does it follow redirects?** Yes, up to 5 hops, each checked against private-network rules. The row shows the final `robotsUrl`. `example.com` and `www.example.com` are different sites with their own robots.txt — check the one your visitors use.

**Is this legal advice, or proof of consent for AI use?** No. It reports a technical signal.

**Can you add a bot or fix a verdict?** Yes — open an issue with the site and the bot. Fixes within 48 hours.

### Limitations

- Static robots.txt evaluation; no proof that a CDN/WAF lets the real crawler in.
- Public sites on ports 80/443 only; no login, no JavaScript rendering, no proxy (so sites that block datacenter traffic give free `fetch_blocked` rows).
- At most 10 sample paths per site; the probe uses the first 3.
- `llms.txt` is checked for presence and basic shape only (a validator/generator is planned as a separate tool).

### Related Actors

- [OpenAI Ads Readiness Checker](https://apify.com/kondasviktor/openai-ads-readiness-checker) — AdsBot robots, HTTPS, landing-page and image checks before you run ChatGPT ads
- [OpenAI Ads Check & Convert](https://apify.com/kondasviktor/openai-ads-check-convert) — readiness check plus Google Merchant feed → OpenAI Ads CSV
- [Website Change Monitor](https://apify.com/kondasviktor/website-change-monitor) — get alerted when any page changes

***

Built by [Viktor Kondas](https://apify.com/kondasviktor) · [Vibe Coder's Life](https://vibecoderslife.com/?utm_source=apify\&utm_medium=readme\&utm_campaign=ai-crawler-access-checker) · Custom audits: hello@vibecoderslife.com

# Actor input Schema

## `sites` (type: `array`):

Domains or URLs, one per line: <code>example.com</code> or <code>https://example.com/blog/</code>. One result per site (scheme + host: <code>example.com</code> and <code>www.example.com</code> are different sites). A path in a URL is checked as an extra sample path for that site. Public sites only.

## `bots` (type: `array`):

Which robots.txt user-agent tokens to evaluate. Leave empty for the default set (the major AI training, search, user-fetch and ads crawlers). <b>Google-Extended</b> and <b>Applebot-Extended</b> are control tokens: they never crawl, robots.txt is the only place they are read. Optional entries have not been verified against the operator’s documentation.

## `customBots` (type: `array`):

Any other robots.txt user-agent token, e.g. a crawler announced after this Actor was published. Letters, digits, <code>.</code> <code>\_</code> <code>-</code> only.

## `samplePaths` (type: `array`):

Paths to evaluate for every bot, e.g. <code>/</code>, <code>/blog/</code>, <code>/products/</code>. A bot that may crawl some of them but not all gets the verdict <code>partial</code>. Maximum 10 per site, including paths typed inside the site URLs.

## `checkLlmsTxt` (type: `boolean`):

One extra request per site: is there an <code>/llms.txt</code>, and does it look like Markdown with a title and links? (Presence and shape only — not a full validation.)

## `maxSites` (type: `integer`):

Hard cap on paid results — sites whose robots.txt was read. Free rows (unreachable, blocked, invalid) do not count. This is also your cost cap: 100 sites = $0.50.

## `wafProbe` (type: `boolean`):

<b>Only for sites you own or are allowed to test.</b> Requests up to 3 sample URLs once with a neutral user-agent and once per probe bot with that bot’s User-Agent token, and compares status codes and challenge-page markers. It shows whether a <i>user-agent rule</i> or a challenge page treats the bot differently. It runs from an Apify datacenter IP, so it <b>cannot prove</b> what the real crawler (which CDNs may verify by IP range) experiences. Adds up to ~30 requests per site; no extra charge.

## `probeBots` (type: `array`):

Crawlers to probe when the WAF probe is on (maximum 8; control tokens cannot be probed). The Actor sends an approximate User-Agent string containing the bot’s token — not the vendor’s exact string.

## `respectRobots` (type: `boolean`):

On (recommended): <code>/llms.txt</code> and probe URLs are skipped when the site’s robots.txt disallows this Actor’s own user-agent (<code>AICrawlerAccessChecker</code>, falling back to <code>\*</code>). Turn off only for sites you own or may test. Reading <code>/robots.txt</code> itself is always done.

## `maxConcurrency` (type: `integer`):

How many different sites are checked at the same time (1–5). Requests to one site are always sequential with a pause of at least 0.7 s.

## `requestTimeoutSecs` (type: `integer`):

Per request. A site that does not answer in time is reported as unreachable (free).

## `trackHistory` (type: `boolean`):

Remember each site’s verdicts in a named key-value store in <b>your</b> account and add <code>change</code> / <code>changeDetail</code> to every result on the next run. Schedule the Actor weekly to get policy-change alerts.

## `historyStoreName` (type: `string`):

Named key-value store that holds the change-tracking state. Use different names to keep separate watchlists apart.

## Actor input object example

```json
{
  "sites": [
    "anthropic.com",
    "github.com"
  ],
  "bots": [
    "GPTBot",
    "OAI-SearchBot",
    "ChatGPT-User",
    "OAI-AdsBot",
    "ClaudeBot",
    "Claude-SearchBot",
    "Claude-User",
    "PerplexityBot",
    "Perplexity-User",
    "Google-Extended",
    "Applebot-Extended",
    "CCBot",
    "meta-externalagent",
    "meta-externalfetcher"
  ],
  "customBots": [],
  "samplePaths": [
    "/"
  ],
  "checkLlmsTxt": true,
  "maxSites": 10,
  "wafProbe": false,
  "probeBots": [
    "GPTBot",
    "OAI-SearchBot",
    "ChatGPT-User",
    "ClaudeBot",
    "PerplexityBot"
  ],
  "respectRobots": true,
  "maxConcurrency": 3,
  "requestTimeoutSecs": 15,
  "trackHistory": true,
  "historyStoreName": "ai-crawler-access-history"
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "anthropic.com",
        "github.com"
    ],
    "bots": [
        "GPTBot",
        "OAI-SearchBot",
        "ChatGPT-User",
        "OAI-AdsBot",
        "ClaudeBot",
        "Claude-SearchBot",
        "Claude-User",
        "PerplexityBot",
        "Perplexity-User",
        "Google-Extended",
        "Applebot-Extended",
        "CCBot",
        "meta-externalagent",
        "meta-externalfetcher"
    ],
    "samplePaths": [
        "/"
    ],
    "maxSites": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("kondasviktor/ai-crawler-access-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sites": [
        "anthropic.com",
        "github.com",
    ],
    "bots": [
        "GPTBot",
        "OAI-SearchBot",
        "ChatGPT-User",
        "OAI-AdsBot",
        "ClaudeBot",
        "Claude-SearchBot",
        "Claude-User",
        "PerplexityBot",
        "Perplexity-User",
        "Google-Extended",
        "Applebot-Extended",
        "CCBot",
        "meta-externalagent",
        "meta-externalfetcher",
    ],
    "samplePaths": ["/"],
    "maxSites": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("kondasviktor/ai-crawler-access-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "anthropic.com",
    "github.com"
  ],
  "bots": [
    "GPTBot",
    "OAI-SearchBot",
    "ChatGPT-User",
    "OAI-AdsBot",
    "ClaudeBot",
    "Claude-SearchBot",
    "Claude-User",
    "PerplexityBot",
    "Perplexity-User",
    "Google-Extended",
    "Applebot-Extended",
    "CCBot",
    "meta-externalagent",
    "meta-externalfetcher"
  ],
  "samplePaths": [
    "/"
  ],
  "maxSites": 10
}' |
apify call kondasviktor/ai-crawler-access-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kondasviktor/ai-crawler-access-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/X67hO6wKoUIouKfvv/builds/iucsBQRt4AynYjlws/openapi.json