# AI Crawler Access Audit (`gridlinetools/ai-crawler-access-audit`) Actor

See which AI and search crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot and 150+ more) each site allows or blocks in robots.txt, and whether llms.txt is published. One row per URL, at most two small requests per site, no page content.

- **URL**: https://apify.com/gridlinetools/ai-crawler-access-audit.md
- **Developed by:** [Gridline Tools](https://apify.com/gridlinetools) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 site audits

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler Access Audit: which AI crawlers your robots.txt allows or blocks, and whether llms.txt exists

For every URL on your list the Actor reads the site's `robots.txt`, evaluates it for more than 150 known AI and search crawler tokens (GPTBot, ClaudeBot, Google-Extended, CCBot and others) and tells you which of them may access that URL and which rule made the decision. It also checks whether the site publishes `llms.txt`. At most two plain GET requests are sent per origin (`/robots.txt` and `/llms.txt`), evaluation follows RFC 9309 (a dedicated `User-agent` group beats `*`, the longest matching rule wins), and the bot dictionary ships inside the Actor, so nothing is downloaded at run time. The Actor reports what your `robots.txt` says to each bot; it does not observe what the bots actually do.

### What it is good for

- **AI visibility report for an agency:** run dozens of client sites at once and get, per URL, which AI bots are `allowed` or `disallowed` and by which rule; the dataset is ready to be turned into an "AI visibility" report.
- **Verify that your AI-training block really works:** check that `GPTBot`, `ClaudeBot`, `Google-Extended`, `CCBot` or `Bytespider` are disallowed on every path that matters, and that the search-type bots that bring you traffic (`OAI-SearchBot`, `Claude-SearchBot`, `PerplexityBot`) were not blocked by accident. `botsNotRespecting` tells you how many of the checked bots declare they ignore `robots.txt` anyway.
- **A building block for automations:** one normalized JSON record per URL with a bundled dictionary of ~160 AI bot tokens (operator, category, stated respect for `robots.txt`), so you do not maintain your own list of user agents.
- **llms.txt check:** whether the site publishes `/llms.txt` with a valid H1 title, and how many sections and links it has (structure only, never the content).

### How to use it

1. Open the Actor and paste your list of URLs into the `urls` field, one per line (or pass a JSON array through the API).
2. Leave `bots` empty to check the whole dictionary, or list the bot tokens you care about (for example `GPTBot`, `ClaudeBot`); turn `checkLlmsTxt` off if you only need the robots.txt decisions. The other defaults work for most runs.
3. Click Start. When the run finishes, the status message reads `Processed N of M URLs (X with errors, Y not charged)`.
4. Download the results from the Dataset tab as JSON, CSV or Excel, or read them through the API.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | list of strings | small self-test list | Absolute http(s) URLs, one per line; duplicates skipped; max 1,000 per run |
| `bots` | list of strings | (all known bots) | AI bot tokens to check, e.g. `GPTBot`, `Bytespider`. Case-insensitive matching; duplicates use first spelling. Invalid tokens are dropped. Empty = check all ~160 bots from the built-in dictionary. |
| `checkLlmsTxt` | boolean | `true` | Whether to also fetch and inspect `llms.txt` from each origin. Set to `false` to skip llms.txt checks entirely (saves requests). |
| `timeoutSecs` | integer 1–60 | 15 | Maximum time per file (applied separately to `robots.txt` and `llms.txt`). |
| `maxConcurrency` | integer 1–50 | 10 | URLs processed in parallel. |

### Output

One dataset record per input URL. Common fields:

| Field | Type | Meaning |
|---|---|---|
| `inputValue` | string | URL as provided (credentials masked) |
| `error` | string or null | `INVALID_URL`, `TIMEOUT`, `NETWORK_ERROR`, `TOO_MANY_REDIRECTS` or `null` when the server responded |
| `billable` | boolean | `true` when the record was charged; `false` for records written free of charge (`INVALID_URL`, `TIMEOUT`, `NETWORK_ERROR`, `TOO_MANY_REDIRECTS`) |
| `errorMessage` | string or null | Human-readable reason |
| `checkedAt` | string | ISO 8601 timestamp |

Robot-specific fields (one `robots.txt` fetch per origin, cached for the run):

| Field | Type | Meaning |
|---|---|---|
| `robotsUrl` | string or null | `<origin>/robots.txt`; `null` for `INVALID_URL` |
| `robotsStatus` | int or null | HTTP status of the `robots.txt` response |
| `robotsState` | `ok` / `missing` / `unreachable` / `invalid` / null | State of `robots.txt`; `null` only for `INVALID_URL` |
| `robotsTruncated` | boolean | Whether the file exceeded 512,000 bytes |
| `redirectCount` | int 0–5 | Number of redirects followed for `robots.txt` |
| `robotsFromCache` | boolean | `true` when `robots.txt` was reused from the same origin earlier in the run |
| `sitemaps` | string\[] | `Sitemap:` URLs from `robots.txt`; `[]` when file is not `ok` |

Audit fields, one `BotDecision` object per bot checked:

| Field | Type | Meaning |
|---|---|---|
| `bots` | BotDecision\[] | One entry per bot token, in input / dictionary order |
| `botsChecked` | int | Number of bots checked |
| `botsAllowed` | int | Bots with `allowed: true` |
| `botsDisallowed` | int | Bots with `allowed: false` |
| `botsExplicit` | int | Bots with a dedicated `User-agent` line in `robots.txt` |
| `botsNotRespecting` | int | Bots flagged as `respect: "no"` in the dictionary (regardless of the decision) |
| `defaultGroupAllowed` | boolean or null | Decision for the `*` wildcard group |
| `defaultGroupRule` | `{ type, pattern }` or null | Rule that decided for the default group |

Each `BotDecision` object:

| Field | Type | Meaning |
|---|---|---|
| `bot` | string | Bot token (spelling from the dictionary when known) |
| `operator` | string or null | Organization behind the bot |
| `category` | string or null | Category from the dictionary (AI Assistants, AI Data Scrapers, etc.) |
| `respectsRobotsTxt` | `yes` / `no` / `unclear` / null | Stated respect from the `ai.robots.txt` community list |
| `allowed` | boolean or null | RFC 9309 decision for this bot and URL; `null` at `INVALID_URL` |
| `explicit` | boolean | Whether the file has a group exactly matching this bot token |
| `matchedGroup` | string or null | Bot token (spelling from file) or `*` (wildcard) of the group that decided |
| `matchedRule` | `{ type, pattern }` or null | Longest-matching `Allow`/`Disallow` rule that decided |
| `crawlDelay` | number or null | `Crawl-delay` value from the deciding group, in seconds |

`llms.txt` fields (fetched in parallel with `robots.txt`, cached per origin):

| Field | Type | Meaning |
|---|---|---|
| `llmsTxtUrl` | string or null | `<origin>/llms.txt`; `null` for `INVALID_URL`; set even when `checkLlmsTxt` is `false` |
| `llmsTxtStatus` | int or null | HTTP status of the `llms.txt` response |
| `llmsTxtState` | `present` / `absent` / `invalid` / `unreachable` / `skipped` / null | `skipped` when `checkLlmsTxt: false`; `null` only for `INVALID_URL` |
| `llmsTxtBytes` | int or null | UTF-8 size in bytes of the text read (after truncation to 512,000); `null` when no text |
| `llmsTxtTitle` | string or null | H1 title from `llms.txt` (≤ 200 chars); `null` when no H1 on the first non-empty line |
| `llmsTxtSections` | int | Number of `## ` section headings |
| `llmsTxtLinks` | int | Number of list lines (`- ` / `* `) containing an `http(s)` Markdown link |
| `llmsTxtTruncated` | boolean | Whether `llms.txt` exceeded 512,000 bytes |
| `llmsTxtFromCache` | boolean | `true` when `llms.txt` was reused from the same origin |

Dictionary metadata:

| Field | Type | Meaning |
|---|---|---|
| `dictionaryVersion` | string | Snapshot date `YYYY-MM-DD` from the bot dictionary |
| `dictionarySize` | int | Total number of bot tokens in the dictionary snapshot |

A `SUMMARY` record in the default key-value store holds the run counters (`inputEntries`, `inputItems`, `skippedDuplicates`, `droppedEmptyOrNonString`, `truncatedOverLimit`, `resumedSkipped`, `processed`, `checked` = charged, `failed`, `notBilled`, `notSavedAfterLimit`, `unexpectedErrors`, `stoppedByChargeLimit`).

### Pricing

This Actor uses Apify pay per event pricing. Each run costs a small fixed start fee plus a price per URL audited, that is per URL whose server answered the `robots.txt` request; the exact amounts are shown in the Pricing tab. `INVALID_URL`, `TIMEOUT`, `NETWORK_ERROR` and `TOO_MANY_REDIRECTS` records are written free of charge. There are no additional platform usage fees.

A URL is charged only when its record is written, and every charged record is written. When the run's "Maximum cost per run" is reached, the Actor stops and reports how many URLs were left unprocessed.

### Limits and behavior

- Up to 1,000 URLs per run; exact duplicates are skipped.
- Private, loopback, link-local and `localhost` addresses, URLs with embedded credentials and blocked ports are reported as `INVALID_URL` without any request and without charge. Credentials are masked in `inputValue` (`user:***@host`). A user name without a password is masked as well (`https://***@host/`); a password containing a raw `/`, `\`, `?` or `#` is not recognized as credentials, so percent-encode such characters.
- A run interrupted by a platform migration resumes without processing or charging the same URL twice.
- At most two files are fetched per origin: `robots.txt` and `llms.txt` (one file when `checkLlmsTxt` is `false`), each following up to 5 redirects; the actual page is never requested.
- `robots.txt` and `llms.txt` are cached per origin within a run: subsequent URLs from the same origin reuse the result without additional requests.
- Maximum 5 redirects per file; redirects to private addresses are rejected (`unreachable`).
- Maximum 512,000 bytes read per file; larger files are truncated at the last complete line.
- No retries: each file is fetched once.
- `timeoutSecs` is applied independently to `robots.txt` and `llms.txt`.
- When `robots.txt` is `unreachable` (5xx, timeout, network error), all bots are considered `disallowed` per RFC 9309 §2.3.1.4.
- When `robots.txt` is `missing` (4xx) or `invalid` (NUL byte), all bots are considered `allowed`.
- The record is billed (`billable: true`) whenever the server responded to the `robots.txt` request, including 5xx errors and redirects that end without a valid location. Only `INVALID_URL`, `TIMEOUT`, `NETWORK_ERROR`, and `TOO_MANY_REDIRECTS` are not billed.
- Errors fetching `llms.txt` do not affect billing or the `error` field; they are reported only in `llmsTxtState`.
- `llms.txt` is inspected only for structure (H1 title, section count, link count); no file content enters the record beyond the H1 title.

### FAQ

**Why is Bytespider shown as not respecting robots.txt even when the decision for it is `allowed`?**

The `respectsRobotsTxt` field reflects the community-maintained `ai.robots.txt` database, not the actual decision for this particular URL. Bytespider is listed with `respect: "no"` in that database, so every record for Bytespider shows `respectsRobotsTxt: "no"` regardless of whether the site's `robots.txt` allows or disallows it.

**Why does a 500 error on robots.txt block all bots?**

Per RFC 9309 §2.3.1.4, when a `robots.txt` file cannot be fetched due to a server error, crawlers must assume full disallow. The Actor follows the same rule: `robotsState: "unreachable"` → every bot gets `allowed: false`. Note that this record is still charged since the server did respond (with a 5xx code).

**What happens when a site doesn't have an llms.txt?**

The Actor reports `llmsTxtState: "absent"` (4xx status) or `"unreachable"` (5xx/timeout/network error). The `error` field and billing are unaffected; they depend only on the `robots.txt` outcome.

**Can I check just a few bots instead of all 160+?**

Yes, pass a list of bot tokens in the `bots` input field (e.g. `["GPTBot", "ClaudeBot"]`). Case-insensitive matching; the spelling from the built-in dictionary is used in the output. Tokens not found in the dictionary produce records with `operator: null`, `category: null`, `respectsRobotsTxt: null`.

**Is the bot dictionary up to date?**

The dictionary is a snapshot of the `ai-robots-txt/ai.robots.txt` community repository, bundled with the Actor at build time. The snapshot date is visible in every record as `dictionaryVersion`. The dictionary is not fetched from the internet at runtime.

### Other Actors by Gridline Tools

Other checks from the same publisher, each with one row per address and the same pricing rule (start fee plus price per checked item):

- [HTTP Status and Headers Checker](https://apify.com/gridlinetools/http-status-headers-checker): Audit a list of URLs in one run: status code, every hop of the redirect chain and the final URL, response time, server and content type, and six key security headers.
- [Robots.txt Checker](https://apify.com/gridlinetools/robots-txt-checker): Test a list of URLs against each site's robots.txt for a chosen user agent (RFC 9309).
- [RDAP Domain Lookup and Expiry Checker](https://apify.com/gridlinetools/rdap-domain-lookup): Query RDAP registries for a list of domains and get registrar, creation and expiry dates, EPP statuses, name servers and DNSSEC in one table.

### Disclaimer

The Actor sends plain HTTP GET requests to `<origin>/robots.txt` and `<origin>/llms.txt` only; it never requests the actual page content. Only the H1 title and section/link counts from `llms.txt` are stored in the dataset; no file content beyond the title enters any record. The bot dictionary is a snapshot of the `ai-robots-txt/ai.robots.txt` repository (MIT license, snapshot date in `dictionaryVersion`). You are responsible for having the right to query those URLs and for complying with the terms of service of the sites you check. Gridline Tools does not store the checked URLs or results beyond your own Apify account.

# Actor input Schema

## `urls` (type: `array`):

Absolute, publicly reachable URLs starting with http:// or https://, one per line. The path of each URL is the page being checked (add /, /blog/, /docs/… as separate lines). Exact duplicates are skipped. Maximum 1,000 per run. Private/localhost addresses, embedded credentials and blocked ports are reported as INVALID_URL and not charged. The default list is a small self-test (Apify also runs it automatically every day to verify the Actor works).

## `bots` (type: `array`):

User-agent product tokens to evaluate, one per line (e.g. GPTBot, ClaudeBot, Google-Extended, CCBot, MyBot). Leave empty to check the full built-in dictionary of AI crawlers (150+ tokens, with operator and whether the bot respects robots.txt). Tokens outside the dictionary are allowed; '\*' reports the default group as a bot. Letters, digits, '\_', '.' and '-' only; invalid entries are skipped with a warning.

## `checkLlmsTxt` (type: `boolean`):

Also fetch <origin>/llms.txt and report whether it exists and has the required shape (an H1 title). Turn off to skip the second request; llmsTxtState is then 'skipped'.

## `timeoutSecs` (type: `integer`):

Maximum time for fetching one file (robots.txt and llms.txt are timed separately). On timeout of robots.txt the record gets error = TIMEOUT and is not charged.

## `maxConcurrency` (type: `integer`):

How many URLs are processed in parallel. Lower it if many URLs point to one host.

## Actor input object example

```json
{
  "urls": [
    "https://example.com",
    "https://example.org"
  ],
  "bots": [],
  "checkLlmsTxt": true,
  "timeoutSecs": 15,
  "maxConcurrency": 10
}
```

# Actor output Schema

## `results` (type: `string`):

Default dataset: one record per input URL.

## `summary` (type: `string`):

SUMMARY record in the default key-value store: input, processed, checked, failed and invalid counters, charge-limit flags.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "bots": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("gridlinetools/ai-crawler-access-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "bots": [] }

# Run the Actor and wait for it to finish
run = client.actor("gridlinetools/ai-crawler-access-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "bots": []
}' |
apify call gridlinetools/ai-crawler-access-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gridlinetools/ai-crawler-access-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/aCYffkznGeAYGEOk1/builds/NpgYHWedxZ5c5iSWm/openapi.json