# AI Crawler Access Test - Can ChatGPT & AI Bots Read Your Site? (`centorprat/ai-crawler-access-test`) Actor

Tests whether AI search and answer bots (OpenAI, Anthropic, Perplexity, Google, Bing...) can actually read your site: robots.txt per bot (search vs training), a live fetch as each bot vs a browser to catch CDN/WAF blocks, JS-only content, llms.txt and entity schema, with fixes.

- **URL**: https://apify.com/centorprat/ai-crawler-access-test.md
- **Developed by:** [Centor Prat](https://apify.com/centorprat) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $8.50 / 1,000 site testeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler Access Test — Can ChatGPT, Claude & Perplexity Read Your Site?

If AI search and answer bots can't fetch your pages, you can't be cited in ChatGPT search, Perplexity or Claude answers, however good your content is. Most "AI readiness" checkers only read robots.txt. This one **actually fetches your page as each AI bot** and compares the result with a normal browser, so it also catches the block robots.txt never shows: a **CDN or firewall rule rejecting AI crawlers**.

Passive: a handful of normal GET requests per site. Keyless. Runs as a batch Actor **or an instant HTTP endpoint**.

### What it tests

| Test | Detail |
|---|---|
| **robots.txt per bot** | 18 crawlers from OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, DuckDuckGo, Common Crawl and ByteDance. Exact RFC 9309 group matching and longest-rule wins, checked for the exact URL path you give. |
| **Search vs training** | Each bot is classed as **search**, **user-fetch** (an assistant fetching a page for a user) or **training**. Blocking training bots (GPTBot, ClaudeBot, Google-Extended, CCBot…) is a legitimate choice. Blocking **search / user-fetch** bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User…) removes you from AI answers. The report keeps these apart. |
| **Live fetch as each bot** | The page is fetched with each bot's published User-Agent and compared with a browser fetch. **A spoofed Googlebot is used as a control.** If it gets through but GPTBot doesn't, that's a confident, deliberate AI-bot block (e.g. a CDN "block AI bots" rule). If the site rejects the fake Googlebot too, it's impostor protection and the result is marked **inconclusive** rather than guessed. |
| **Content in the HTML** | AI crawlers don't run JavaScript. A page that is an empty JS shell is invisible to them, even if it looks fine in a browser. |
| **llms.txt** | Presence and validity (H1, summary, linked sections), plus `llms-full.txt`. |
| **Entity signals** | Organization / LocalBusiness JSON-LD, `sameAs` profile links, content schema (FAQ, Product, Article…), meta description and heading structure. These help models identify you correctly. |
| **Index hygiene** | `noindex`, `noai` / `noimageai` directives (meta and X-Robots-Tag), sitemap. |

### Output

```json
{
  "url": "https://www.nytimes.com",
  "verdict": "ai-invisible",
  "aiReadinessScore": 28,
  "majorSearchBotsBlocked": ["ChatGPT-User", "Claude-SearchBot", "Claude-User", "OAI-SearchBot", "Perplexity-User", "PerplexityBot"],
  "trainingBotsBlocked": ["GPTBot", "ClaudeBot", "Google-Extended", "Applebot-Extended", "CCBot"],
  "findings": [{"severity": "high", "code": "search-bots-disallowed", "detail": "robots.txt blocks AI SEARCH / answer bots: ..."}],
  "summary": "www.nytimes.com -> ai-invisible (28/100): major AI search/answer bots 2/8 not blocked, training bots blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended; ..."
}
```

Real results from the same run: **react.dev** → `ai-ready` (88), **notion.com** → `ai-ready` (98), **anthropic.com** → `ai-ready` (80, no llms.txt).

Each report also has a `bots` matrix (owner, purpose, robots rule, live status and block reason), the `rendering`, `llmsTxt`, `structuredData` and `directives` details, and a **prioritised `fixes` list**.

**Verdicts:** `ai-ready` · `ai-readable-with-gaps` · `partially-blocked` · `ai-invisible` · `allowed-by-robots-live-test-inconclusive` (the site rejects even a browser-like automated fetch).

### One API call, JSON back

Run it synchronously: one POST starts the run and returns the results in the same response. No run to poll, which is what an AI agent wants mid-task:

```bash
curl -X POST "https://api.apify.com/v2/acts/centorprat~ai-crawler-access-test/run-sync-get-dataset-items" -H "Authorization: Bearer <APIFY_TOKEN>" \
     -H "Content-Type: application/json" -d '{"urls": ["example.com", "react.dev"]}'
```

The body takes the same fields as the normal input. Same pay-per-event price; failed checks aren't charged.

### Ready-made tasks

- [Is My Site Blocking ChatGPT, Claude or Perplexity?](https://apify.com/centorprat/ai-crawler-access-test/examples/is-my-site-blocking-chatgpt)
- [llms.txt & AI Readiness Check](https://apify.com/centorprat/ai-crawler-access-test/examples/llms-txt-and-ai-readiness-check)

### For AI agents / MCP

```
https://mcp.apify.com/?tools=actors,docs,centorprat/ai-crawler-access-test
```

### Use cases

- **GEO / AEO audits:** the "why aren't we cited?" diagnostic (pair with AI Share of Voice).
- **After a CDN / WAF change:** confirm you didn't silently block ChatGPT or Perplexity.
- **Agencies:** batch-test client sites and hand over the fix list.
- **Publishers:** verify your training opt-out doesn't also block AI search traffic.

### FAQ

**Is the live test the same as the real bot?** It uses each bot's published User-Agent from Apify's servers. Rules that check the bots' real IP ranges can't be reproduced, which is why the Googlebot control exists and inconclusive cases are labelled as such.

**Should I block training bots?** That's your call. The report treats it as information, not a problem. Blocking search / user-fetch bots is what costs AI visibility.

**Does it crawl my site?** No. It makes one page fetch per tested bot, plus robots.txt, llms.txt and sitemap.xml.

**Cost?** One `site-tested` event per URL. No API keys, no paid scraper.

### Related actors

- [AI Share of Voice](https://apify.com/centorprat/ai-share-of-voice): how often AI models recommend your brand.
- [Robots.txt & Sitemap Analyzer](https://apify.com/centorprat/robots-sitemap-analyzer)
- [Structured Data & Schema Validator](https://apify.com/centorprat/structured-data-schema-validator)

# Actor input Schema

## `urls` (type: `array`):

e.g. "example.com" or "https://example.com/pricing". One report per URL (the path is checked against robots.txt rules).

## `datasetId` (type: `string`):

Read URLs from another dataset - uses the 'url' / 'website' / 'domain' field.

## `maxSites` (type: `integer`):

Cap how many URLs to test this run.

## `webhookUrl` (type: `string`):

If set, the reports are POSTed here as JSON when the run completes.

## Actor input object example

```json
{
  "urls": [
    "react.dev"
  ],
  "maxSites": 25
}
```

# Actor output Schema

## `reports` (type: `string`):

One report per URL: verdict + AI readiness score, per-bot matrix (owner, search/user-fetch/training, robots.txt rule, live fetch status and block reason), major search bots blocked, training bots blocked, content-in-HTML test, llms.txt validity, structured data / entity signals, meta robots / X-Robots-Tag, sitemap, findings and prioritised fixes.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "react.dev"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("centorprat/ai-crawler-access-test").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["react.dev"] }

# Run the Actor and wait for it to finish
run = client.actor("centorprat/ai-crawler-access-test").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "react.dev"
  ]
}' |
apify call centorprat/ai-crawler-access-test --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,centorprat/ai-crawler-access-test"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/mdcKdpaxDZefxDRuN/builds/KYdRIk2ZwDFnNOYNX/openapi.json