# AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot (`locomint/ai-crawler-checker`) Actor

AI crawler checker: read each website's robots.txt and see whether GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and 13 other AI crawlers are allowed or blocked, with the line that decided each verdict, plus Content-Signal and llms.txt.

- **URL**: https://apify.com/locomint/ai-crawler-checker.md
- **Developed by:** [Locomint](https://apify.com/locomint) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / 1,000 result delivereds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

This AI crawler checker reads each website's robots.txt and reports, for 18 AI crawlers
including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot, whether that crawler may
fetch the site, quoting the line that decided every verdict. It also reads the site's
Content-Signal, llms.txt and ai.txt.

### What it does

For each domain the actor fetches four files in parallel over a direct connection:
`/robots.txt`, `/llms.txt`, `/ai.txt` and the homepage. Nothing else on the site is requested.
Then it answers two questions: may each agent fetch the path you care about, and what does the
site say about AI training as a whole.

**One verdict per agent**, in `agents`:

| Field | What goes in it |
|---|---|
| `agent`, `vendor`, `purpose` | The user-agent token, who runs it and what it is for (training, search, user-triggered fetch, open crawl, commercial crawl). |
| `allowed` | `true`, `false`, or `null` when robots.txt could not be read. |
| `rule` | The line that decided it, such as `Disallow: /`. `null` when the group has no line matching the path, which means allowed. |
| `matched_by` | `User-agent: GPTBot` when the site names the agent, `User-agent: *` when the wildcard group applied, `no matching group`, `no robots.txt`, or `robots.txt could not be read`. |
| `explicit` | `true` when the site named this agent rather than relying on `*`. |
| `crawl_delay` | The group's `Crawl-delay`, when it sets one. |

Matching follows robots.txt conventions: a group that names the agent beats the `*` group, the
longest matching rule wins, an `Allow` wins a tie, and `*` and `$` work as wildcards.
`allowed_by_agent` repeats the verdicts as a flat `{agent: true/false/null}` map, which is the
easiest field to use in a spreadsheet.

**Site-level answers**

| Field | What goes in it |
|---|---|
| `blocked_agents` | Every tracked agent, plus any you added, that the rules block for the path. |
| `blocks_any_ai_agent` | `true` when `blocked_agents` is not empty; `null` when the rules could not be read. |
| `ai_training_allowed` | `false` when the site publishes `Content-Signal: ai-train=no`, whatever its per-agent rules say. Otherwise `true` only when all seven training crawlers (GPTBot, ClaudeBot, Google-Extended, Bytespider, Applebot-Extended, meta-externalagent, cohere-ai) are allowed. |
| `content_signal_ai_train`, `content_signals` | What the Content-Signal line says about training, and the line itself. |
| `llms_txt`, `ai_txt` | `true` when the file answers 200 with content that is not an HTML page. A site that serves its HTML 404 page for every path does not count. |
| `pay_per_crawl` | `true` when the homepage or robots.txt answers HTTP 402 Payment Required. |
| `text_without_javascript`, `javascript_dependent` | Words of homepage text in the HTML before any script runs, and `true` below 50 words. A crawler that does not run JavaScript sees little of such a site. |
| `robots_status`, `robots_present`, `robots_bytes`, `robots_truncated`, `sitemaps` | The HTTP status of robots.txt, whether it had content, its size, whether it was cut at 3 MB, and up to 20 sitemap URLs. |
| `status`, `warnings` | `ok` or `unreadable`, and a sentence for anything unusual. |

#### Unknown is not allowed

The rule this checker is built around: **a robots.txt that could not be read gives null
verdicts, never "allowed".** "We do not know the rules" and "there are no rules" are different
answers, and treating the first as the second is how a checker tells you that you may crawl a
site that forbids it.

- A 200 answer is read as written. An empty file means no rules.
- Any answer whose body is plainly a ruleset is parsed whatever its status code.
- A 4xx other than 429 means no rules, per RFC 9309: every agent is allowed, `matched_by` says
  `no robots.txt`.
- A 5xx, a timeout, a refused connection, a 403, a 429 or a challenge page means unknown:
  `status` is `unreadable`, every `allowed` is `null`, `ai_training_allowed` and
  `blocks_any_ai_agent` are `null`, and `warnings` says why. The site is not retried from
  another address.

### Who it is for

- **SEO and content teams** checking that their own robots.txt says what they meant. Blocking
  GPTBot for training while keeping OAI-SearchBot for search is a one-line mistake in either
  direction, and each verdict names the line responsible.
- **Publishers** auditing every site and brand they run in one pass, with the evidence kept.
- **AI, RAG and data teams** who must honour robots.txt before crawling. Run the source list
  first, keep the rows where your agent is `true`, drop `false` and `null`, and store `rule`,
  `matched_by` and `checked_at` as the record of what the site said on the day.
- **Agencies** reporting on clients' AI crawler access, including `llms.txt` and pages that
  depend on JavaScript.
- **Researchers** measuring how many sites in a list block AI training.

### How to use it

In the Apify Console:

1. Paste domains or URLs into **Websites**, one per line, up to 5,000. Only the host is used, so
   `https://www.nytimes.com/section/world` and `nytimes.com` are the same site.
2. Leave **AI crawlers to check** empty for all 18, or list the ones you care about. Any other
   user-agent token works too, including your own crawler's.
3. Set **Path to check** if you intend to fetch something other than the homepage, for example
   `/blog/`.
4. Turn **Include sites whose robots.txt could not be read** off if you only want answers you
   can act on; those sites are then left out and not charged.
5. Start the run and export the dataset as CSV, Excel or JSON.

From the API, this call starts a run, waits for it and returns the rows:

```bash
curl -X POST \
  "https://api.apify.com/v2/acts/locomint~ai-crawler-checker/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["nytimes.com", "vercel.com"], "agents": ["GPTBot", "ClaudeBot"]}'
```

The synchronous endpoint waits up to 300 seconds. For a long list, start the run with
`POST https://api.apify.com/v2/acts/locomint~ai-crawler-checker/runs?token=YOUR_APIFY_TOKEN`
and read its dataset when the run finishes. With the Python client (`pip install
apify-client`):

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("locomint/ai-crawler-checker").call(
    run_input={"domains": ["nytimes.com", "vercel.com", "python.org"], "includeUnreadable": False}
)
for row in client.dataset(run.default_dataset_id).iterate_items():
    print(row["domain"], row["ai_training_allowed"], row["blocked_agents"])
```

### Input example

```json
{
  "domains": ["nytimes.com", "vercel.com", "python.org"],
  "agents": ["GPTBot", "ClaudeBot", "Google-Extended", "PerplexityBot"],
  "path": "/",
  "checkLlmsTxt": true,
  "checkHomepage": true,
  "includeUnreadable": true,
  "concurrency": 5,
  "siteTimeout": 30
}
```

| Field | Default | Allowed | What it does |
|---|---|---|---|
| `domains` | required | 1 to 5,000 | Domains or URLs. The host is used without `www.`; duplicates are dropped. |
| `agents` | all 18 | up to 50 names | The 18 tracked names match in any case (`gptbot` becomes `GPTBot`). Other tokens are checked as written, with `vendor` and `purpose` null. |
| `path` | `/` | up to 300 characters | The path every verdict is about. A leading `/` is added when missing. |
| `checkLlmsTxt` | `true` | true / false | Fetch `/llms.txt` and `/ai.txt`. Off: both fields are `null`. |
| `checkHomepage` | `true` | true / false | Fetch the homepage for the 402 and JavaScript checks. Off: those fields are `null`, and `pay_per_crawl` looks at robots.txt only. |
| `includeUnreadable` | `true` | true / false | Off: sites whose robots.txt could not be read are left out and not charged. |
| `concurrency` | 5 | 1 to 10 | Sites checked at the same time. |
| `siteTimeout` | 30 | 5 to 90 seconds | A site with no answer by then is delivered as `unreadable`. |

### Output example

A real row, from a run on 11 September 2026 with all 18 agents. `allowed_by_agent` and `agents`
held all 18 entries; four and two are shown. Vercel's robots.txt blocks none of the AI crawlers,
but it publishes `Content-Signal: ai-train=no`, so `ai_training_allowed` is `false`.

```json
{
  "domain": "vercel.com",
  "input": "vercel.com",
  "path": "/",
  "status": "ok",
  "ai_training_allowed": false,
  "blocks_any_ai_agent": false,
  "blocked_agents": [],
  "content_signal_ai_train": false,
  "content_signals": "search=yes, ai-input=yes, ai-train=no",
  "allowed_by_agent": {"GPTBot": true, "OAI-SearchBot": true, "ChatGPT-User": true, "ClaudeBot": true},
  "agents": [
    {"agent": "GPTBot", "allowed": true, "rule": null, "matched_by": "User-agent: *", "explicit": false, "crawl_delay": null, "vendor": "OpenAI", "purpose": "training"},
    {"agent": "ClaudeBot", "allowed": true, "rule": null, "matched_by": "User-agent: *", "explicit": false, "crawl_delay": null, "vendor": "Anthropic", "purpose": "training"}
  ],
  "robots_url": "https://vercel.com/robots.txt",
  "robots_status": 200,
  "robots_present": true,
  "robots_bytes": 629,
  "robots_truncated": false,
  "sitemaps": ["https://vercel.com/sitemap.xml"],
  "llms_txt": true,
  "ai_txt": false,
  "homepage_status": 200,
  "pay_per_crawl": false,
  "text_without_javascript": 138,
  "javascript_dependent": false,
  "warnings": [],
  "checked_at": "2026-09-11T18:33:01+00:00"
}
```

In the same run, nytimes.com blocked 16 of the 18 agents, each through a group that names it:

```json
{"agent": "GPTBot", "allowed": false, "rule": "Disallow: /", "matched_by": "User-agent: GPTBot", "explicit": true, "crawl_delay": null, "vendor": "OpenAI", "purpose": "training"}
```

Googlebot and Amazonbot were allowed by groups of their own. python.org blocked none and had no
Content-Signal, so its `ai_training_allowed` was `true`.

### Pricing

| Event | Price |
|---|---|
| Domain checked (one dataset row) | $0.0005, which is $0.50 per 1,000 until 26 September 2026, then $0.003, which is $3.00 per 1,000 |
| Actor start | $0.00005 per GB of run memory, charged once per run |

Worked example: 10,000 domains is two runs of 5,000, so 10,000 x $0.0005 = $5.00 plus two start
events of $0.00005 each for 1 GB runs. From 26 September 2026 the price per domain becomes
$0.003.

You pay only these event prices; Apify compute is not billed to you separately, and the four
files per domain are one charge. Unreadable sites are charged by default because the row is
still an answer ("unknown"); set `includeUnreadable` to `false` and they are left out and cost
nothing. Entries that are not a domain or URL produce no row and no charge. If you set a
maximum cost per run, the actor stops before the row that would pass it.

### FAQ

#### Does "allowed" mean I may use the site's content?

It means the site's robots.txt does not disallow that crawler from that path. robots.txt
governs crawling; it is not a licence to the content, and a site's terms of service can say
more. `ai_training_allowed` reflects robots.txt and Content-Signal only.

#### Why are all the verdicts null for one site?

Its robots.txt could not be read: the site answered 5xx, 403 or 429, served a challenge page,
refused the connection or did not answer in time. `warnings` gives the reason. Unknown is kept
as unknown rather than guessed; turn `includeUnreadable` off to leave such sites out.

#### Why does a site with no robots.txt show every agent allowed?

Under RFC 9309 a missing file (a 404 or another 4xx apart from 429) means there are no rules,
so everything is allowed. `matched_by` says `no robots.txt`, which keeps that case apart from a
site that wrote rules and allowed the agent. A 402 on robots.txt also counts as no rules, so
read `pay_per_crawl` next to the verdicts.

#### Why is ai\_training\_allowed false when no agent is blocked?

The site publishes a Content-Signal directive with `ai-train=no`, as vercel.com did in the
example above. The site said so in words, and that settles the training question whatever the
per-agent rules allow; `content_signals` quotes the directive.

#### Can I check my own crawler or a path other than the homepage?

Yes. Any user-agent token in `agents` is checked as written, and `path` makes every verdict
about that path, so `/blog/` can be blocked on a site whose homepage is open. The path is
evaluated against the rules; it is not fetched.

#### What counts as having llms.txt?

A 200 answer at `/llms.txt` with content that does not start as an HTML page. Many sites answer
every unknown path with their HTML error page and a 200 status, and those do not count. `null`
means the file was not checked or the site gave no answer.

#### How big a robots.txt can it read?

Up to 3 MB, which covers the largest files in normal use; some sites publish several hundred
kilobytes and put their AI rules at the end. If a file is ever cut, `robots_truncated` is
`true` and `warnings` says a rule near the end may be missing.

### Limits

- Four files per domain: robots.txt, llms.txt, ai.txt and the homepage. No other page is
  fetched, and `path` is evaluated against the rules, not requested.
- The host is checked without `www.`. A site that answers only on its `www.` name comes back
  `unreadable`; subdomains such as `blog.example.com` are checked as given.
- A 403 or 429 on robots.txt is treated as a block, so the verdicts are `null`, not "no file".
- When a file names the same agent in two separate groups, only the first group is read.
- Meta robots tags, `X-Robots-Tag` headers, TDM reservation files and terms of service are not
  read. Content-Signal is read from robots.txt only.
- Verdicts describe the file at `checked_at`. Sites change their rules; re-run before a crawl.
- 5,000 domains and 50 agents per run, 30 seconds per site by default (5 to 90). A challenge is
  never retried from another address.

### Other Locomint actors

- [Google Maps Scraper & Email Extractor – Business Leads](https://apify.com/locomint/google-maps-scraper): Search terms and a city in, business records with website contacts out.
- [Google Maps Scraper – Multi-City Lead Lists with Emails](https://apify.com/locomint/google-maps-multi-city-scraper): Many categories across many cities in one deduplicated run.
- [Google Maps Place Details Scraper – Bulk Place ID Lookup](https://apify.com/locomint/google-maps-place-details-scraper): Place IDs or place-page links in, full records out.
- [Website Email Scraper – Contact Details, Socials & WhatsApp](https://apify.com/locomint/website-content-contact-extractor): Contact points from website URLs you supply.
- [Website Content Crawler – Markdown for AI, Emails & Contacts](https://apify.com/locomint/website-crawler-content-contacts): A whole site as Markdown, with its contact points.
- [Bulk Email Verifier & Validator](https://apify.com/locomint/bulk-email-verifier): Checks whether addresses can receive mail.
- [Company Enrichment API – Domain to Emails, Socials & Tech](https://apify.com/locomint/company-enrichment): A domain in, its contacts and technologies out.
- [Schema Markup Validator & Generator – JSON-LD Checker](https://apify.com/locomint/schema-markup-validator): Checks and generates schema.org markup.

# Actor input Schema

## `domains` (type: `array`):

Domains or URLs, one per line. Only the host is used, so https://www.nytimes.com/section/world and nytimes.com are the same site. Up to 5,000 per run.

## `agents` (type: `array`):

Leave empty to check all 18 we track: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, Googlebot, PerplexityBot, Perplexity-User, CCBot, Bytespider, Amazonbot, Applebot-Extended, meta-externalagent, cohere-ai, Diffbot and Timpibot. Any other user-agent name works too.

## `path` (type: `string`):

The path you intend to fetch, for example /blog/. Every verdict answers whether that crawler may fetch this path.

## `checkLlmsTxt` (type: `boolean`):

Report whether the site publishes /llms.txt and /ai.txt.

## `checkHomepage` (type: `boolean`):

Fetch the homepage to report pay-per-crawl (HTTP 402) and whether it has readable text before any JavaScript runs.

## `includeUnreadable` (type: `boolean`):

Those rows arrive with every verdict null (unknown, never allowed) and count as a result. Turn this off to leave them out and not pay for them.

## `concurrency` (type: `integer`):

How many sites to check at the same time.

## `siteTimeout` (type: `integer`):

A site that has not answered by then is delivered with status unreadable.

## Actor input object example

```json
{
  "domains": [
    "nytimes.com",
    "vercel.com",
    "python.org"
  ],
  "path": "/",
  "checkLlmsTxt": true,
  "checkHomepage": true,
  "includeUnreadable": true,
  "concurrency": 5,
  "siteTimeout": 30
}
```

# Actor output Schema

## `sites` (type: `string`):

One row per domain: whether AI training is allowed, which AI crawlers are blocked, a verdict per crawler with the robots.txt line that decided it, Content-Signal, llms.txt and ai.txt, and warnings.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "nytimes.com",
        "vercel.com",
        "python.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("locomint/ai-crawler-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "nytimes.com",
        "vercel.com",
        "python.org",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("locomint/ai-crawler-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "nytimes.com",
    "vercel.com",
    "python.org"
  ]
}' |
apify call locomint/ai-crawler-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,locomint/ai-crawler-checker"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/oLMBmvDPxIk8iNpgd/builds/uz44cgnfrw7VdetUa/openapi.json