# AI Crawler & robots.txt Readiness Audit (`aicrawlready/ai-crawler-robots-readiness-audit`) Actor

Checks each domain's robots.txt for 21 vendor-documented AI crawler tokens (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot and more): allowed, blocked or not mentioned, with the deciding rule and line. Also reports Sitemap lines, llms.txt presence and Content-Signal lines.

- **URL**: https://apify.com/aicrawlready/ai-crawler-robots-readiness-audit.md
- **Developed by:** [AI Crawl Ready](https://apify.com/aicrawlready) (community)
- **Categories:** SEO tools, Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 domain auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler & robots.txt Readiness Audit

Give it a list of domains. For each one it fetches `robots.txt` and reports whether 21 AI crawler and fetcher tokens are **allowed**, **blocked**, or **not mentioned**. Every token on the list is documented by its vendor. For each bot you get the rule that decided the result and its line number. Each domain also gets these checks:

- whether a `Sitemap:` line is declared
- whether `/llms.txt` exists (presence only)
- any `Content-Signal:` lines (listed, not evaluated)
- warnings for things that make the result less reliable

You get one dataset row per domain.

To check a single site in the browser, use the free checker at [aicrawlready.com](https://aicrawlready.com). Full input and output documentation for this Actor is at [aicrawlready.com/tools](https://aicrawlready.com/tools).

### What it checks

For every domain:

1. **robots.txt fetch.** It sends one `GET https://<host>/robots.txt` and follows up to 5 redirects. The row records the HTTP status, the final URL after redirects, the redirect chain, the content type and the size.
2. **Rule evaluation for each bot.** It evaluates rules per RFC 9309 plus Google's documented matching behaviour:
   - User-agent values are compared case-insensitively on the product token.
   - Every group that names the bot is merged. If none names it, the `*` group(s) apply. If there is no `*` group either, everything is allowed.
   - The longest matching pattern wins, and `Allow` wins a tie. `*` and `$` are supported.
   - The root path `/` is always checked. Paths you add in `checkPaths` are checked too. Paths are matched against the rules only; they are never fetched.
3. **HTTP status handling (RFC 9309):**
   - 2xx: the rules are evaluated.
   - 4xx, including 401, 403 and 404: no restrictions, so every bot is reported as allowed. A 401 or 403 also adds a bot-protection warning.
   - 5xx or 429: treated as a complete disallow, so every bot is reported as blocked, with a warning.
   - More than 5 redirects: treated as unavailable, like a 4xx.
4. **Sitemap.** Checks whether any `Sitemap:` line is present and lists up to 50 URLs.
5. **llms.txt.** Sends one GET for `/llms.txt` and reads at most 64 KiB. The status is one of:

   - `present`: 2xx with a non-HTML body
   - `html-response`: 2xx but the body is HTML, which is often a soft 404
   - `absent`: 404 or other 4xx, or an empty body
   - `unknown`: 401, 403, 429, 5xx, a timeout or an error

   This is a factual presence check only. The audit makes no claim about whether any AI system reads or respects llms.txt.
6. **Content-Signal.** `Content-Signal:` lines are a non-standard extension. They are listed with their line numbers and are **not** evaluated.
7. **Warnings.** A warning is added for:
   - 401/403 responses, which may be bot protection
   - 5xx/429 responses
   - redirects, including host changes
   - files larger than 500 KiB, where the rest is ignored (Google documents the same 500 KiB limit)
   - robots.txt served as HTML
   - a file with no user-agent groups
   - timeouts
   - redirects to private addresses, which are refused

#### What "status" means for each bot

| status | Meaning |
|---|---|
| `blocked` | `/` is disallowed for this token. The cause can be a group that names the bot, the `*` fallback, or a 5xx/429 robots.txt. |
| `allowed` | A group names this token, and `/` is allowed. |
| `not-mentioned` | No group names this token, and `/` is not blocked. That covers three cases: the `*` group allows it, no group applies, or robots.txt returned a 4xx. |

Each bot also has these fields:

- `mention`: `specific`, `wildcard`, `none` or `n/a`
- `verdict`: `allowed` or `blocked`, for `/`
- `rule`: `{type, pattern, line, text}`, the winning rule (`null` when no rule matched)
- `line`: the line number of that rule
- `paths[]`: one verdict for each checked path

### Limits (please read)

- **Rules only, no crawling.** Only `robots.txt` and `/llms.txt` are requested. The actor does not fetch pages, render JavaScript, or read `<meta name="robots">` tags or `X-Robots-Tag` headers.
- **Only `/` and the paths you list are checked.** A bot shown as allowed at `/` can still be blocked from other sections.
- **Standard `*` fallback only.** Some vendors document their own fallback rules. Amazon says Amzn-SearchBot may follow rules written for other search bots when it isn't named. Apple says Applebot follows Googlebot rules when Applebot isn't named. These vendor-specific fallbacks are not modelled.
- **robots.txt is advisory, not enforcement.** It is a request that well-behaved crawlers choose to honour. Several vendors say that user-triggered fetchers may not follow it; these are marked `userTriggered: true`. This audit reports what the file says, not what any crawler actually does.
- **Responses can differ by client.** A CDN or bot-protection layer may give this actor a different response (for example 403) than it gives real crawlers. When that seems likely, the row carries a warning.
- **Scheme.** Bare domains are checked over `https://`. If you need `http://`, pass the full URL.
- **Bot list is fixed.** The list contains 21 tokens, each verified against vendor documentation on 2026-09-25. `Claude-Web`, `anthropic-ai` and `Bytespider` are deliberately left out because current vendor docs for them could not be verified. Other user-agent groups that appear in a file do not change the result for the listed tokens.

### Safety and politeness

- Only `http://` and `https://` on the default ports are allowed. URLs with credentials are refused.
- Private, loopback, link-local, CGNAT, multicast and documentation IP ranges (IPv4 and IPv6) are blocked.
  - The check runs at DNS-resolution time, on the address the socket actually connects to.
  - It runs again on every redirect hop.
- Timeouts are 10 s per request, covering all redirect hops together. Response bodies are capped at 500 KiB for robots.txt and 64 KiB for llms.txt.
- Each domain gets at most 2 requests (robots.txt, then llms.txt), made one after the other. By default 5 domains run in parallel (`maxConcurrency`, 1–20). Duplicate origins are skipped.
- Requests identify themselves with the user agent `AI-Crawler-Readiness-Audit/0.1 (Apify actor; fetches robots.txt and llms.txt only)`.

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `domains` | string\[] | — (required) | Domains or URLs; each is reduced to its origin. 1–1000 entries. |
| `bots` | string\[] | all 21 | Optional filter; tokens from the table below, case-insensitive. |
| `checkPaths` | string\[] | `[]` | Extra paths besides `/` (max 20), e.g. `/blog/`. |
| `includeRawRobots` | boolean | `false` | Adds the robots.txt text to each row. |
| `maxConcurrency` | integer | `5` | 1–20 domains in parallel. |

```json
{
  "domains": ["example.com", "https://www.example.org/some/page"],
  "bots": ["GPTBot", "ClaudeBot", "Google-Extended"],
  "checkPaths": ["/blog/"],
  "includeRawRobots": false,
  "maxConcurrency": 5
}
```

### Output

Each row is one domain. Here is an illustrative item, trimmed to one bot:

```json
{
  "input": "example.com",
  "domain": "example.com",
  "origin": "https://example.com",
  "robotsUrl": "https://example.com/robots.txt",
  "finalUrl": "https://www.example.com/robots.txt",
  "fetchStatus": 200,
  "robotsState": "ok",
  "robotsTxt": { "contentType": "text/plain", "bytes": 412, "truncated": false, "redirects": 1,
                 "redirectChain": ["https://example.com/robots.txt", "https://www.example.com/robots.txt"], "groupCount": 3 },
  "checkedPaths": ["/", "/blog/"],
  "bots": [
    {
      "token": "GPTBot", "vendor": "OpenAI",
      "purpose": "Crawls content that may be used to train OpenAI foundation models",
      "docs": "https://platform.openai.com/docs/bots", "userTriggered": false,
      "status": "blocked", "mention": "specific", "verdict": "blocked",
      "rule": { "type": "disallow", "pattern": "/", "line": 5, "text": "Disallow: /" },
      "line": 5, "groupAgents": ["GPTBot"], "groupLines": [4],
      "paths": [
        { "path": "/", "verdict": "blocked", "rule": { "type": "disallow", "pattern": "/", "line": 5, "text": "Disallow: /" } },
        { "path": "/blog/", "verdict": "blocked", "rule": { "type": "disallow", "pattern": "/", "line": 5, "text": "Disallow: /" } }
      ]
    }
  ],
  "summary": { "botsChecked": 21, "allowed": 1, "blocked": 4, "notMentioned": 16,
               "namedInRobots": 5, "blockedViaWildcard": 0, "blockedOnAnyPath": 4 },
  "sitemap": { "declared": true, "count": 1, "urls": ["https://www.example.com/sitemap.xml"] },
  "llmsTxt": { "url": "https://example.com/llms.txt", "status": "absent", "httpStatus": 404,
               "finalUrl": "https://www.example.com/llms.txt", "contentType": "text/html", "bytes": 1256, "note": null },
  "contentSignals": [{ "line": 2, "value": "search=yes, ai-train=no" }],
  "warnings": [
    "robots.txt redirected 1 time(s) to https://www.example.com/robots.txt (host changed from example.com to www.example.com); rules were read from the final URL.",
    "Found 1 Content-Signal line(s) (a non-standard usage-preference extension). They are listed for reference only and are not evaluated."
  ],
  "error": null,
  "audited": true,
  "checkedAt": "2026-09-25T12:00:00.000Z"
}
```

Rows can also report failures:

- **Failed domains**, such as DNS failures, timeouts or refused private addresses: `audited: false`, `error: {code, message}`, and empty `bots`.
- **Invalid inputs**: `error.code: "INVALID_INPUT"`.

The dataset's **Overview** view shows one table row per domain with the key columns.

### Bot tokens checked

Each token was verified against the vendor's own documentation on 2026-09-25.

| Token | Vendor | What the vendor says it controls | Vendor docs |
|---|---|---|---|
| GPTBot | OpenAI | Content that may be used to train foundation models | [OpenAI crawlers](https://platform.openai.com/docs/bots) |
| OAI-SearchBot | OpenAI | Appearing in ChatGPT search results | [OpenAI crawlers](https://platform.openai.com/docs/bots) |
| ChatGPT-User | OpenAI | User actions in ChatGPT/GPTs (vendor notes robots.txt rules may not apply) | [OpenAI crawlers](https://platform.openai.com/docs/bots) |
| ClaudeBot | Anthropic | Content that may go into model training | [Anthropic help center](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Claude-SearchBot | Anthropic | Search result quality | [Anthropic help center](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Claude-User | Anthropic | Fetches when a user asks Claude | [Anthropic help center](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) |
| Google-Extended | Google | Gemini training and grounding (control token; no separate crawler; not a Search ranking signal) | [Google common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers) |
| PerplexityBot | Perplexity | Perplexity search results (not model training) | [Perplexity crawlers](https://docs.perplexity.ai/guides/bots) |
| Perplexity-User | Perplexity | User-requested fetches (vendor says it generally ignores robots.txt) | [Perplexity crawlers](https://docs.perplexity.ai/guides/bots) |
| CCBot | Common Crawl | Common Crawl archive | [CCBot](https://commoncrawl.org/ccbot) |
| Applebot-Extended | Apple | Training Apple foundation models (control token; doesn't crawl) | [About Applebot](https://support.apple.com/en-us/119829) |
| Meta-ExternalAgent | Meta | Uses such as training AI models or indexing | [Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) |
| Meta-WebIndexer | Meta | Meta AI search results/citations | [Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) |
| Meta-ExternalFetcher | Meta | User-requested fetches (vendor says it may bypass robots.txt) | [Meta web crawlers](https://developers.facebook.com/docs/sharing/webmasters/web-crawlers) |
| Amazonbot | Amazon | Amazon products; may be used to train Amazon AI models | [About Amazonbot](https://developer.amazon.com/amazonbot) |
| Amzn-SearchBot | Amazon | Search experiences such as Alexa (not training) | [About Amazonbot](https://developer.amazon.com/amazonbot) |
| Amzn-User | Amazon | User actions such as Alexa queries (may not follow all robots.txt directives) | [About Amazonbot](https://developer.amazon.com/amazonbot) |
| DuckAssistBot | DuckDuckGo | AI-assisted answers (not training) | [DuckAssistBot](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) |
| MistralAI-Training | Mistral AI | Training datasets | [Mistral crawlers](https://docs.mistral.ai/robots) |
| MistralAI-Index | Mistral AI | Indexing for Mistral search | [Mistral crawlers](https://docs.mistral.ai/robots) |
| MistralAI-User | Mistral AI | User-requested fetches | [Mistral crawlers](https://docs.mistral.ai/robots) |

### Pricing

The actor uses pay-per-event pricing with a single event, **`domain-audited`**:

- **Charged:** once per domain whose robots.txt returned an HTTP response (any status) and was fully evaluated.
- **Not charged:** invalid inputs, DNS failures, timeouts, and refused private-address targets. These still get a row with an `error`.

The Actor owner sets the price in Apify Console; see the Actor's pricing tab for the current amount. The run stops starting new domains once your per-run cost limit is reached. Details are in [PRICING.md](PRICING.md).

### Related tools

- [aicrawlready.com](https://aicrawlready.com): free single-site AI crawler and robots.txt checker, plus guides.
- [aicrawlready.com/tools](https://aicrawlready.com/tools): docs for this Actor, the AI Crawler Checker Chrome extension and the public JSON APIs.

### Local development

```bash
npm install
npm test            # matcher, input normalizer, and mocked-network integration tests
npm run build       # TypeScript -> dist/
npm run local -- nytimes.com vercel.com anthropic.com   # stubbed Actor, writes local-output.json
npx apify-cli run -p                                    # real SDK with local storage (./storage)
npx apify-cli validate-schema
```

The matcher and bot list were copied from a companion Next.js tool (`seo-microtool`) and are kept in sync by hand.

# Actor input Schema

## `domains` (type: `array`):

Domains (example.com) or URLs (https://www.example.com/page). Each entry is reduced to its origin (scheme + host); bare domains use https. robots.txt and /llms.txt are fetched from that origin. Duplicates (same origin) are skipped. Maximum 1000 entries per run.

## `bots` (type: `array`):

Limit the audit to these AI crawler tokens. Leave empty to check all 21 vendor-documented tokens.

## `checkPaths` (type: `array`):

Additional URL paths to evaluate besides / (for example /blog/ or /docs/page?x=1). Max 20. Paths are matched against the rules; they are not fetched.

## `includeRawRobots` (type: `boolean`):

Add the fetched robots.txt text (up to 500 KiB) to each dataset item.

## `maxConcurrency` (type: `integer`):

How many domains are audited in parallel. Each domain gets at most two requests (robots.txt, /llms.txt), made one after the other.

## Actor input object example

```json
{
  "domains": [
    "vercel.com",
    "anthropic.com"
  ],
  "bots": [],
  "checkPaths": [],
  "includeRawRobots": false,
  "maxConcurrency": 5
}
```

# Actor output Schema

## `overview` (type: `string`):

robots.txt status, AI crawler summary counts, sitemap and llms.txt status, Content-Signal lines and warnings per domain.

## `results` (type: `string`):

Complete dataset items including the per-bot verdicts, deciding rules and line numbers.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "vercel.com",
        "anthropic.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("aicrawlready/ai-crawler-robots-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "vercel.com",
        "anthropic.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("aicrawlready/ai-crawler-robots-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "vercel.com",
    "anthropic.com"
  ]
}' |
apify call aicrawlready/ai-crawler-robots-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,aicrawlready/ai-crawler-robots-readiness-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/BGsuYgcq11rreSjd8/builds/isoeHxlFElfXAa91C/openapi.json