# AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked? (`nerolabs/ai-crawler-access-checker`) Actor

Shows if GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Meta-ExternalAgent are allowed or blocked on each site in a dataset, CSV or Sheet: robots.txt, a real fetch per bot, llms.txt, plus change monitoring. Inputs: datasetId, fileUrl, urls. $0.005 per URL. Agent-ready: x402, MCP.

- **URL**: https://apify.com/nerolabs/ai-crawler-access-checker.md
- **Developed by:** [Adam Pearce](https://apify.com/nerolabs) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.50 / 1,000 url checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler Access Checker: Is GPTBot or ClaudeBot Blocked?

**Is your site (or your clients' sites) visible to ChatGPT, Claude, Perplexity and Meta AI, or quietly blocked from them?** Give this Actor a list of websites and it tells you, for each one and each AI crawler, whether the crawler is allowed in, and whether it is robots.txt or a firewall doing the blocking. Your own columns stay next to the answers.

### What you get for every website

- **Verdict**: `open`, `partly_blocked`, `all_blocked` or `unverified` (the site refused our requests, so nothing could be confirmed).
- **Per crawler**: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Meta-ExternalAgent by default (ChatGPT-User, Google-Extended, Applebot-Extended and CCBot are optional). For each: the robots.txt verdict, which rule group applied, and a **real fetch** with the crawler's own user agent compared with a normal browser.
- **Blocked by what**: `robots.txt`, `firewall` (a CDN or bot-protection rule answers 403 or a challenge page) or both.
- **Search vs training**: `blocksAiSearch` (OAI-SearchBot, PerplexityBot, ChatGPT-User) and `blocksTraining` (GPTBot, ClaudeBot, Meta, CCBot, Google-Extended) as separate flags, because blocking training and blocking AI search are different decisions.
- **llms.txt and llms-full.txt** present or not, plus `noai` / `noimageai` directives and the CDN in front of the site.
- **Fixes**: plain sentences saying what is blocked and what to change.

### Monitor mode: get told when it changes

Switch on **Monitor mode** and schedule the run (daily or weekly). The first run records each site's starting point. Every later run reports only what changed: a crawler newly blocked or unblocked in robots.txt, a firewall rule that started or stopped refusing a bot, `llms.txt` added or removed. Set a **Webhook URL** (Slack, Zapier, Make, n8n) and it is only called when something changed. Use **Only changed sites** to keep just those rows. Different watchlists use different **Monitor names**.

### Who uses this

- **Agencies and consultants** auditing a client list for AI visibility before selling "AI search optimisation".
- **Shop owners and publishers** who read on Reddit that Meta, OpenAI or Perplexity bots hammer their site, want to block some, and want proof the block works.
- **SEO teams** who need to know that nobody's CDN update quietly blocked OAI-SearchBot.

### Input

Give websites as an **Apify dataset**, a **CSV or Google Sheet link** (shared as "anyone with the link can view"), a plain **list of URLs**, or inline rows. The URL you give is the page tested, so use the homepage. Set **Website field** if the column is not called website, url or domain; it is detected automatically otherwise.

Example (the default input):

```json
{ "urls": ["https://www.nytimes.com", "https://www.bbc.co.uk", "https://example.com"] }
```

### Output (one row per website)

```json
{
  "checkedUrl": "https://www.bbc.co.uk",
  "checkStatus": "ok",
  "verdict": "all_blocked",
  "summary": "Blocks every checked AI crawler. No llms.txt.",
  "blockedBots": ["GPTBot", "OAI-SearchBot", "ClaudeBot", "PerplexityBot", "Meta-ExternalAgent"],
  "blocksAiSearch": true,
  "blocksTraining": true,
  "gptbotRobots": "blocked",
  "gptbotFetch": "served",
  "llmsTxtFound": false,
  "monitorStatus": "unchanged"
}
```

`...Fetch` values: `served`, `blocked` (the browser got the page, the crawler did not), `not_comparable` (the browser was refused too), `error`, `not_tested`.

### Pricing

Pay per event, no subscription:

- **$0.005 per URL checked** (about $5 per 1,000 sites). Lower on higher Apify plans.
- $0.01 per CSV or Excel export file, only when you ask for one.
- $0.01 per webhook delivered (confirmed 2xx only; in monitor mode only when something changed).

You are **never charged** for unreachable sites, invalid addresses, duplicates, or rows skipped because of a time or budget limit. Watching 200 sites daily for a month is about $30.

### Honest limits

- **robots.txt verdicts are exact.** The fetch test is a request that *looks like* the crawler, sent from an Apify data centre. Some bot protection (Cloudflare verified bots, for example) lets the real crawler in by checking its company IP ranges, so a `firewall` block here can be a false alarm for the genuine crawler, and `served` does not prove the genuine crawler is let in. Treat firewall results as "worth checking", robots.txt results as fact. A firewall block is only reported when a second try, a couple of seconds later, is refused too.
- Sites that refuse every automated request (some big publishers) come back `unverified` or `not_comparable`, not guessed.
- It makes about six light requests per site and never more than two at once to the same site. It reads the page you name, not the whole site.
- llms.txt is an optional convention; no major AI company has confirmed it affects crawling.

### FAQ

**Does it work for my own site?** Yes, that is the main use. Nothing is changed on any site; it only reads.

**Can an AI agent call it?** Yes. Pay per event works through Apify's MCP server and x402, and the output is plain JSON.

**Something looks wrong for one site.** Open an issue on the Actor page with the URL and I will look the same day.

If this saved you a manual robots.txt check, a review on the Store helps a lot.

# Actor input Schema

## `datasetId` (type: `string`):

An Apify dataset with one row per website, for example a Google Maps or lead scraper's output. Every original column is kept and the AI crawler verdicts are added alongside. Use the picker rather than typing an ID.

## `fileUrl` (type: `string`):

A public link to a CSV, TSV, Excel, JSON or JSON Lines file with one row per website. A normal Google Sheets link works: share it as 'Anyone with the link can view'. Used when no dataset is given.

## `fileFormat` (type: `string`):

Leave on 'Detect automatically' unless the link has no file extension and the server reports the wrong content type.

## `sheetName` (type: `string`):

Which sheet to read from an Excel workbook. Defaults to the first sheet.

## `urls` (type: `array`):

A plain list of websites for a quick run with no spreadsheet (bare domains like example.com are fine). The URL you give is the page tested, so use the homepage unless you care about a specific path. Use the dataset, file or Google Sheet inputs above to keep your own columns next to the results.

## `data` (type: `array`):

Rows as inline JSON, an alternative to a dataset or file. Each object needs a field holding the website.

## `urlField` (type: `string`):

The column holding the website to check. Left empty it is detected automatically, preferring columns named website, domain, homepage or url.

## `bots` (type: `array`):

Which crawlers to test. Default: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Meta-ExternalAgent. Google-Extended and Applebot-Extended are robots.txt control tokens with no crawler of their own, so only their robots.txt rule is checked.

## `testRealFetch` (type: `boolean`):

Besides reading robots.txt, request the URL once per crawler with its published user agent and once as a normal browser, to catch firewalls and CDNs that refuse AI crawlers even though robots.txt allows them. About six requests per site. Turn off for a robots.txt-only check.

## `checkLlmsTxt` (type: `boolean`):

Also look for /llms.txt and /llms-full.txt on each site. Free, two extra requests per site.

## `monitor` (type: `boolean`):

Remember each site's result and report what changed since the last run (a crawler newly blocked or unblocked, llms.txt added or removed). Run it on a schedule. The first run of a site just records its starting point. Off by default.

## `monitorName` (type: `string`):

Names the saved memory for monitor mode, so different watchlists do not mix. Use the same name on every scheduled run of one watchlist. Default: 'default'.

## `keep` (type: `string`):

Filtering happens after each site is checked. 'Only changed sites' is the monitor-mode alert list.

## `keepOriginalFields` (type: `boolean`):

Keep every column from the input row next to the results, so they line up with your own data.

## `proxyConfiguration` (type: `object`):

Off by default and usually not needed. Requests then come from an Apify data centre. Turning a proxy on changes which address the sites see, which can change firewall results.

## `concurrency` (type: `integer`):

How many sites to check in parallel. Never more than two requests at a time go to any one site.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one request before giving up. A site that never answers is never charged.

## `maxItems` (type: `integer`):

A hard ceiling on how many rows are read from the input, as a safety net on a large dataset.

## `exportFormats` (type: `array`):

Also write the results as a downloadable CSV or Excel file, linked from the run's output. $0.01 per file, charged only when the run checked at least one URL.

## `outputDatasetName` (type: `string`):

Also append every kept row to a named dataset that persists across runs, for example a history of checks. Not charged again.

## `webhookUrl` (type: `string`):

POST the run summary to this URL when the run finishes, for Slack, Zapier, Make, n8n or your own API. In monitor mode it is sent only when something changed. Charged $0.01 only on a confirmed 2xx response.

## Actor input object example

```json
{
  "fileFormat": "auto",
  "urls": [
    "https://www.nytimes.com",
    "https://www.bbc.co.uk",
    "https://example.com"
  ],
  "bots": [
    "GPTBot",
    "OAI-SearchBot",
    "ClaudeBot",
    "PerplexityBot",
    "Meta-ExternalAgent"
  ],
  "testRealFetch": true,
  "checkLlmsTxt": true,
  "monitor": false,
  "keep": "all",
  "keepOriginalFields": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "concurrency": 10,
  "requestTimeoutSecs": 15,
  "exportFormats": []
}
```

# Actor output Schema

## `results` (type: `string`):

One row per website with a verdict (open, partly blocked, all blocked), which crawlers are blocked and by what, llms.txt, and what to fix. Your original columns are kept.

## `checkSummary` (type: `string`):

URLs checked, verdict counts, blocks per crawler, monitor-mode changes, export links and warnings.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://www.nytimes.com",
        "https://www.bbc.co.uk",
        "https://example.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("nerolabs/ai-crawler-access-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://www.nytimes.com",
        "https://www.bbc.co.uk",
        "https://example.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("nerolabs/ai-crawler-access-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://www.nytimes.com",
    "https://www.bbc.co.uk",
    "https://example.com"
  ]
}' |
apify call nerolabs/ai-crawler-access-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nerolabs/ai-crawler-access-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/mYLg7Gb09IhNxjki9/builds/WXWN11lIJbxQF0woq/openapi.json