# AI Crawler Checker: robots.txt GPTBot, ClaudeBot & llms.txt (`swillz/ai-crawler-access-checker`) Actor

Bulk check which AI crawlers a website allows or blocks in robots.txt: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and 18 more. Adds llms.txt, Content-Signal, meta robots and a verdict (training vs AI search). Misses are free.

- **URL**: https://apify.com/swillz/ai-crawler-access-checker.md
- **Developed by:** [William Jin](https://apify.com/swillz) (community)
- **Categories:** AI, SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.60 / 1,000 site checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**AI Crawler Access Checker** tells you, for any list of websites, **which AI crawlers are allowed or blocked in robots.txt**: OpenAI's GPTBot, OAI-SearchBot and ChatGPT-User, Anthropic's ClaudeBot, Claude-SearchBot and Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, Meta, Amazon, Common Crawl (CCBot), Bytespider and more (23 crawlers, plus any user-agent you add).

It separates **AI training bots** from **AI search and assistant bots**, so you see at a glance whether a site can still be **cited in ChatGPT, Claude and Perplexity answers** (`aiSearchAllowed`) or only blocks model training (`blocks_ai_training_only`). It also reads **llms.txt / llms-full.txt**, Cloudflare **Content-Signal** (`ai-train`, `ai-input`, `search`), meta robots and X-Robots-Tag (`noai`, `noimageai`, `noindex`).

Rules are evaluated like a real crawler, following **RFC 9309**: the bot's own group overrides `*`, longest match wins, `Allow` wins ties, `*` and `$` wildcards, redirects followed. Only public files are read (robots.txt, llms.txt and the one page you test, if robots.txt allows it). No logins, no personal data.

**When to use:** GEO / AI SEO audits, checking whether your own site or clients are invisible to AI search, publisher and competitor research, lead lists of sites that block (or allow) AI, monitoring robots.txt changes, or filtering sites before an AI scraping job.
**Input:** domains or URLs, e.g. `{"domains":["nytimes.com","stripe.com"]}`.
**Output:** one row per site with `aiAccessVerdict`, `aiSearchAllowed`, blocked and allowed bot lists, per-bot matched rules, llms.txt details and meta robots. From $1.60 per 1,000 sites.

### What can AI Crawler Access Checker do?

- **Bulk check robots.txt for AI bots:** thousands of sites per run, 10 in parallel by default.
- **Training vs AI search, separated:** `trainingBotsBlocked`, `answerBotsBlocked`, `aiSearchAllowed` and a one-word verdict: `allows_all_ai`, `blocks_ai_training_only`, `blocks_some_ai_search` or `blocks_all_ai`.
- **Explain every decision:** for each bot, the user-agent group that matched and the exact rule (e.g. `GPTBot` → `Disallow: /`).
- **llms.txt check:** found or not, title, summary, sections, link count and whether it follows the llms.txt format; plus llms-full.txt.
- **Content signals and meta tags:** Cloudflare Content-Signal, meta robots, X-Robots-Tag, `noai` / `noimageai`, and a hint when a firewall answers 403 to bots.
- **Any path, any bot:** test `/blog/` instead of the whole site, or add your own tokens in `customUserAgents`.
- **Chain from other Actors:** pass a `datasetId` from a lead list, Google Maps or SERP scrape and it reads the website field.

### What data does it return?

| Field | Example | Meaning |
|---|---|---|
| `aiAccessVerdict` | `blocks_some_ai_search` | One-word summary for AI crawlers (Googlebot and Bingbot not counted) |
| `aiSearchAllowed` | `false` | No AI search or assistant bot is blocked, so the site can be cited in AI answers |
| `blockedBots` | `["GPTBot","ClaudeBot",…]` | Crawlers disallowed on the tested path |
| `allowedBots` | `["Bingbot","Applebot"]` | Crawlers allowed |
| `trainingBotsBlocked` / `answerBotsBlocked` | `10` / `8` | Counts by purpose |
| `googlebotAllowed` / `bingbotAllowed` | `true` | Classic search access (also feeds Google AI Overviews and Copilot) |
| `bots` | `[{"bot":"GPTBot","purpose":"training","access":"blocked","matchedRule":"Disallow: /"}]` | Per-bot details |
| `contentSignal` | `{"search":true,"ai-train":false}` | Cloudflare Content-Signal for all crawlers |
| `llmsTxt` | `{"found":true,"title":"Stripe","linkCount":308}` | llms.txt details |
| `homepage` | `{"httpStatus":200,"noai":false}` | Meta robots and X-Robots-Tag of the tested page |
| `sitemaps`, `crawlDelaySecs` | | From robots.txt |
| `status`, `error` | `checked` | Free rows explain why a site could not be checked |

### How to check which AI bots a website blocks (step by step)

1. Open the Actor and paste your domains or URLs into **Domains or website URLs**.
2. Optionally pick specific crawlers, a path to test, or turn off the llms.txt and page checks.
3. Click **Start**. A few hundred sites take about a minute.
4. Filter the dataset by `aiAccessVerdict` or `aiSearchAllowed`, or download it as CSV, Excel or JSON.

### How much does it cost?

You pay only for sites checked, through **pay-per-event** pricing:

- **$1.60–2.00 per 1,000 sites**, depending on your Apify plan: Free $2.00, Starter $2.00, Scale $1.80, Business and up $1.60.
- **Never charged:** sites that don't resolve or answer, robots.txt behind a firewall (401/403/429), invalid inputs and duplicates. They're still listed as free rows with a reason.
- A negligible start fee per run.

| Job | Free plan | Starter | Business and up |
|---|---|---|---|
| Audit 50 client sites | $0.10 | $0.10 | $0.08 |
| 10,000-site research list | $20.00 | $20.00 | $16.00 |
| Weekly check of your 200 competitors | ≈ $1.73 / month | ≈ $1.73 / month | ≈ $1.39 / month |

Prices drop on bigger Apify plans (Starter = Bronze tier, Scale = Silver, Business and up = Gold). Set a **maximum cost per run** in the run options and the Actor stops cleanly at that limit.

### Input example

```json
{
  "domains": ["nytimes.com", "stripe.com", "https://www.cloudflare.com"],
  "bots": [],
  "customUserAgents": ["YouBot"],
  "testPath": "/",
  "checkLlmsTxt": true,
  "checkHomepage": true
}
```

Leave `bots` empty to check all 23 known AI and search crawlers.

### Output example

Real result for nytimes.com (shortened):

```json
{
  "domain": "nytimes.com",
  "status": "checked",
  "robotsTxtUrl": "https://www.nytimes.com/robots.txt",
  "aiAccessVerdict": "blocks_some_ai_search",
  "blockedBots": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "Claude-SearchBot", "Claude-User", "anthropic-ai", "PerplexityBot", "Perplexity-User", "Google-Extended", "Applebot-Extended", "Meta-ExternalAgent", "Meta-ExternalFetcher", "DuckAssistBot", "CCBot", "Bytespider", "cohere-ai", "Diffbot"],
  "partiallyBlockedBots": ["Googlebot", "Amazonbot"],
  "allowedBots": ["Bingbot", "Applebot", "MistralAI-User"],
  "trainingBotsBlocked": 10,
  "answerBotsBlocked": 8,
  "aiSearchAllowed": false,
  "googlebotAllowed": true,
  "bingbotAllowed": true,
  "bots": [
    {"bot": "GPTBot", "operator": "OpenAI", "purpose": "training", "access": "blocked", "allowed": false, "matchedUserAgent": "GPTBot", "matchedRule": "Disallow: /"}
  ],
  "llmsTxt": {"found": false, "url": "https://www.nytimes.com/llms.txt", "httpStatus": 404},
  "homepage": {"fetched": true, "httpStatus": 403, "noindex": false, "noai": false, "note": "The page answered HTTP 403 to our checker; a firewall or bot protection may also block AI crawlers here even though robots.txt allows them."},
  "checkedAt": "2026-11-02T17:30:09.328Z"
}
```

And stripe.com allows every crawler and publishes an llms.txt: `"aiAccessVerdict": "allows_all_ai"`, `"llmsTxt": {"found": true, "title": "Stripe", "linkCount": 308, "followsSpec": true}`.

### Use cases

- **GEO / AI SEO audits:** make sure your site, or a client's, isn't accidentally invisible to ChatGPT search, Claude or Perplexity.
- **Agencies:** audit a whole client list in one run and export it to a sheet.
- **Publisher and competitor research:** which news, e-commerce or SaaS sites block AI training, and who still allows AI search.
- **Lead generation:** find sites that block AI search (a GEO pitch) or that lack llms.txt.
- **Responsible scraping:** check a list of sites before you crawl them with an AI agent.
- **Monitoring:** schedule weekly runs to catch robots.txt changes.

### Use with AI agents (MCP, ChatGPT, Claude, Cursor, n8n)

AI agents can find and run this Actor through the [Apify MCP server](https://mcp.apify.com). Add it to Claude Desktop, Cursor, VS Code or any MCP client:

```json
{"mcpServers":{"apify":{"url":"https://mcp.apify.com?tools=swillz/ai-crawler-access-checker"}}}
```

Or run it locally: `npx -y @apify/actors-mcp-server --tools swillz/ai-crawler-access-checker` with `APIFY_TOKEN` set. In n8n or Make, use the Apify node with Actor `swillz/ai-crawler-access-checker`.

Example prompts:

- "Does nytimes.com block GPTBot and ClaudeBot?"
- "Which of these 100 sites block AI search bots, and which only block training?"
- "Do my competitors have an llms.txt?"

Minimal input:

```json
{"domains": ["nytimes.com"]}
```

**Keep it fast:** set `checkHomepage` and `checkLlmsTxt` to false to read only robots.txt.

**Cap the cost:** pass `maxTotalChargeUsd` ("Maximum cost per run") when you call the Actor. The run stops cleanly at the cap, keeps the results so far, lists unchecked sites as free `skipped_charge_limit` rows, and says why in the status message and the OUTPUT record.

### Integrations and API

- **API and clients:** run it from Python or JavaScript with the Apify client, or call the REST API and download the dataset as CSV or JSON.
- **No-code:** use the official Apify apps for **Make**, **n8n** and **Zapier**. Choose *Run Actor*, then `swillz/ai-crawler-access-checker`.
- **AI agents (MCP):** add it as a tool for Claude, Cursor or other MCP clients through `https://mcp.apify.com?tools=swillz/ai-crawler-access-checker`.
- **Google Sheets, Slack and webhooks:** set these up from the Integrations tab of any run.

### AI Crawler Access Checker vs alternatives

| | This Actor | Single-site web checkers | Generic robots.txt testers |
|---|---|---|---|
| Bulk (thousands of sites), API, MCP | ✅ | ❌ | ❌ |
| 23 AI crawlers incl. search vs training split | ✅ | Partly | ❌ (one bot at a time) |
| Matched group and rule per bot | ✅ | ❌ | ✅ |
| llms.txt, Content-Signal, noai meta | ✅ | Partly | ❌ |
| Free rows for unreachable sites | ✅ | — | — |

### FAQ

**Is robots.txt the whole story?** No. robots.txt is a request that well-behaved crawlers follow. Some sites also block bots at the firewall (e.g. Cloudflare's AI bot blocking). When robots.txt or the tested page answers 403/429 to our checker we flag it, but we can't see firewall rules directly.

**What's the difference between training and search bots?** Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot…) collect data for model training. Search and user bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot…) fetch pages to answer and cite in AI answers. Blocking training only keeps you citable.

**Does blocking Google-Extended remove me from Google?** No. Google-Extended only controls Gemini training and grounding. Googlebot (reported separately) controls Google Search, including AI Overviews.

**What counts as one site?** Each unique host. `nytimes.com` and `www.nytimes.com` are separate inputs only if you enter both.

**What if there's no robots.txt?** A 404 means every crawler is allowed (RFC 9309). The row says so and is charged as a normal check.

**Is it legal?** It reads only public files any crawler reads (robots.txt, llms.txt and one page, only if robots.txt allows it), identifies itself with a clear user-agent and collects no personal data.

### Related tools

More tools from the same developer, built to work together:

- [Career Site Jobs Scraper](https://apify.com/swillz/career-site-jobs-scraper): jobs from Greenhouse, Lever, Ashby, Workable and Recruitee boards, with salary fields, as a jobs API or CSV.
- [Workday Jobs Scraper](https://apify.com/swillz/workday-jobs-scraper): every job from any myworkdayjobs.com career site, past the 2,000-result cap.
- [Company Website Enrichment](https://apify.com/swillz/company-website-enrichment): turn a list of domains into company name, emails, phones, social links and tech stack.
- [AI Extract](https://apify.com/swillz/ai-extract): structured JSON from any web page with a plain-English prompt or a JSON schema. AI included.
- [Google Sheets Import & Export](https://apify.com/swillz/google-sheets-import-export): send any Apify dataset to a Google Sheet (append, replace or dedupe) or read a sheet as JSON.
- [XML Sitemap URL Extractor & Sitemap to CSV](https://apify.com/swillz/sitemap-url-extractor): every URL from a website's sitemap, with lastmod, HTTP status and new-page monitoring.
- [AI Brand Visibility Tracker](https://apify.com/swillz/ai-brand-visibility-tracker): track whether ChatGPT mentions, ranks and cites your brand vs competitors (GEO).
- [Email Provider Detector](https://apify.com/swillz/email-provider-detector): bulk check whether domains use Google Workspace, Microsoft 365, Proofpoint or Mimecast, plus SPF, DMARC and DKIM.
- [Hiring Signals by Company Domain](https://apify.com/swillz/hiring-signals-by-domain): from a list of company domains, find which ATS each uses and how many open roles, by department and location.
- [EU VAT Number Validator (VIES)](https://apify.com/swillz/eu-vat-validator): bulk check EU VAT numbers against official VIES, with company name and address.
- [EORI Number Validator](https://apify.com/swillz/eori-validator): bulk check EU and UK EORI numbers for customs, with trader name where published.
- [LEI Lookup (GLEIF)](https://apify.com/swillz/lei-lookup): find a company's Legal Entity Identifier, legal name, address, status and parent from official GLEIF data.
- [Sanctions Screener](https://apify.com/swillz/sanctions-screener): screen company and person names against official OFAC, UN, EU and UK sanctions lists with fuzzy matching.
- [HTS Code Lookup (USITC)](https://apify.com/swillz/hts-code-lookup): US HTS codes and import duty rates from the official tariff schedule, by code or product description.
- [Federal Contracts & Grants by Company](https://apify.com/swillz/federal-awards-by-company): federal contracts, grants and loans any US company won, from official USAspending.gov data.
- [Grants.gov Opportunity Search](https://apify.com/swillz/grants-gov-opportunity-search): open and forecasted US federal grants by keyword, agency or eligibility, with deadlines, award ceilings and funding.

**Popular workflows**

- **GEO audit:** check which AI crawlers a site lets in with this Actor, see whether ChatGPT actually mentions and cites the brand with [AI Brand Visibility Tracker](https://apify.com/swillz/ai-brand-visibility-tracker), and report both in a sheet with [Google Sheets Import & Export](https://apify.com/swillz/google-sheets-import-export).
- **What AI can read:** list every page from the sitemap with [XML Sitemap URL Extractor & Sitemap to CSV](https://apify.com/swillz/sitemap-url-extractor), then test key sections for AI crawlers with this Actor (`testPath`).

Source code: https://github.com/williamjin02/apify-actors/tree/main/actors/ai-crawler-access-checker

# Actor input Schema

## `domains` (type: `array`):

Sites to check, e.g. \["nytimes.com","https://www.stripe.com"]. A bare domain is checked at https:// and follows redirects (e.g. to www). Each host is checked once.

## `bots` (type: `array`):

Leave empty to check all 23 known crawlers: OpenAI, Anthropic, Perplexity, Google, Microsoft, Apple, Meta, Amazon, DuckDuckGo, Mistral, Common Crawl, ByteDance, Cohere and Diffbot.

## `customUserAgents` (type: `array`):

Any other crawler tokens to check against robots.txt, e.g. \["YouBot","Timpibot"]. Up to 20.

## `testPath` (type: `string`):

Path the bot rules are evaluated for (and the page whose meta robots is read). Default "/" (the whole site). Use e.g. "/blog/" to check a section.

## `checkLlmsTxt` (type: `boolean`):

Look for /llms.txt (title, summary, sections, link count) and /llms-full.txt.

## `checkHomepage` (type: `boolean`):

Fetch the tested page (only if robots.txt allows generic crawlers) and report meta robots, X-Robots-Tag, noindex, noai and noimageai.

## `includeBotDetails` (type: `boolean`):

Add the bots array (operator, purpose, access, matched user-agent and rule for each crawler). Turn off for compact rows; the blocked/allowed lists stay.

## `datasetId` (type: `string`):

Optional dataset ID from another Actor run (e.g. a lead list or Google Maps scrape). Domains are read from domainField.

## `domainField` (type: `string`):

Field to read from datasetId, e.g. "website". Auto-detected from domain, website, url and similar if empty.

## `maxDomains` (type: `integer`):

Stop after this many unique domains (1-100000). Default 1000.

## `concurrency` (type: `integer`):

Sites checked in parallel (1-30). Default 10.

## `timeoutSecs` (type: `integer`):

Per HTTP request (3-60). Default 15.

## Actor input object example

```json
{
  "domains": [
    "nytimes.com",
    "stripe.com",
    "apify.com"
  ],
  "testPath": "/",
  "checkLlmsTxt": true,
  "checkHomepage": true,
  "includeBotDetails": true,
  "maxDomains": 1000,
  "concurrency": 10,
  "timeoutSecs": 15
}
```

# Actor output Schema

## `results` (type: `string`):

Example: {"domain":"nytimes.com","status":"checked","aiAccessVerdict":"blocks_some_ai_search","aiSearchAllowed":false,"blockedBots":\["GPTBot","ClaudeBot",...],"googlebotAllowed":true,"llmsTxt":{"found":false}}. Miss: {"status":"unreachable","error":"..."}.

## `summary` (type: `string`):

JSON: domainsChecked, byStatus, byVerdict, mostBlockedBots, llmsTxtFound, botsChecked, zeroResultsReason, nextSteps.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "nytimes.com",
        "stripe.com",
        "apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("swillz/ai-crawler-access-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "nytimes.com",
        "stripe.com",
        "apify.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("swillz/ai-crawler-access-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "nytimes.com",
    "stripe.com",
    "apify.com"
  ]
}' |
apify call swillz/ai-crawler-access-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,swillz/ai-crawler-access-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/6UCNTUWUc1PTiEuqT/builds/uNVkiJVpULhCRYfRw/openapi.json