# AI Crawler Policy Audit — robots.txt, llms.txt, noai (`multiplex/ai-crawl-policy`) Actor
See exactly which AI crawlers a site allows. Per domain it reads robots.txt for 25 named AI agents (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot…), plus ai.txt, llms.txt, noai meta, X-Robots-Tag and tdmrep.json, and quotes the rule behind every answer. Alerts on policy changes.
- **URL**: https://apify.com/multiplex/ai-crawl-policy.md
- **Developed by:** [Daniel James](https://apify.com/multiplex) (community)
- **Categories:** SEO tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
Pay per usage
This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage
## What's an Apify Actor?
Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## AI Crawler Policy Audit — robots.txt, llms.txt, noai
**Two questions, one answer set.**
> **Publishers:** which AI crawlers may take your content right now — and did that change this week?
> **AI teams:** may we crawl this domain — and can we prove we checked?
Give it a list of domains. Get back, per domain and per AI user agent: **allowed / not allowed**, **the
exact rule that decided it**, and **which file that rule came from**. Plus a one-word verdict per
domain and an alert when any of it moves.
Most "AI blocker" checkers grep robots.txt for a handful of agent names. This one implements RFC 9309
properly — group selection, longest-match precedence, `Allow` beating `Disallow` on ties, `*` and `$`
wildcards, `Crawl-delay` — then reads the four other surfaces a site can express AI policy on, and
tells you which one actually decided the answer.
***
### Why it exists
Site owners now express AI policy in five different places that contradict each other constantly. A
publisher blocks `GPTBot` in robots.txt, publishes an `llms.txt` inviting models in, and has a
`noai` tag left over from a 2023 CMS plugin. An AI team fetching that domain has to decide which one
governs. This actor answers that question the same way every time, shows its working, and records
the answer so the next run can tell you what moved.
| You are | What you get |
|---|---|
| A publisher | The full picture of what you are exposing, including the files you forgot you published. `aiPolicyStance` in one column across your whole portfolio. |
| A publisher with an editorial line on AI | Proof of what your policy said on a given date, and an alert the day it changes. |
| An AI / agent team | A respect-check before you crawl: `domain, agent, allowed, rule, source` — five columns, machine-readable, with the rule quoted so a compliance reviewer can follow it. |
| A researcher or journalist | Adoption data. Who blocks whom, who publishes `llms.txt`, who has reserved TDM rights, and how that moves over time. |
| Anyone doing competitive work | An alert when a competitor's AI policy shifts — often the first public sign of a licensing deal or a legal position. |
***
### What it checks — at most six requests per domain
| # | Request | What it answers |
|---|---|---|
| 1 | `/robots.txt` | The AI-agent groups, per-agent permission, path restrictions, `Crawl-delay`. **The primary evidence.** |
| 2 | `/` (homepage) | `` and the `X-Robots-Tag` response header — `noai`, `noimageai`, `noindex`. |
| 3 | `/llms.txt` | Present? Valid? Title, summary, sections, link count. |
| 4 | `/llms-full.txt` | Present? (Existence only — the file is not read.) |
| 5 | `/ai.txt` | Present? Which agents it names, whether it disallows everything, which asset types it names. |
| 6 | `/.well-known/tdmrep.json` | A TDM rights reservation for `/`, and the policy URL if one is given. |
Requests to one host are sequential and spaced. `robots.txt` is fetched first and **honoured for this
actor's own requests** — if a site disallows `ai-crawl-policy` from `/llms.txt`, that check is
skipped and reported as skipped, never guessed.
***
### Output
Two row types in one dataset, distinguished by `rowType`. The Apify UI ships three views:
**Domain summary**, **Per agent**, and **Changes only**.
#### Per domain + agent
```json
{
"id": "0d9c…",
"rowType": "agent",
"domain": "example.com",
"agent": "GPTBot",
"allowed": false,
"rule": "Disallow: /",
"source": "robots",
"changeType": "changed",
"previousAllowed": true,
"previousSource": "none",
"namedInRobots": true,
"pathRestrictions": 0,
"crawlDelay": null,
"agentVendor": "OpenAI",
"agentKind": "training",
"agentVerified": true,
"agentNote": "Crawls content that may be used to train OpenAI foundation models.",
"agentDocs": "https://developers.openai.com/api/docs/bots",
"status": "ok",
"checkedAt": "2026-08-19T11:02:44.118Z",
"previousCheckedAt": "2026-08-12T11:00:07.554Z",
"warning": null
}
```
`allowed` is `true`, `false`, or **`null`**. `null` means *unknown* — the audit could not read
`robots.txt`, so no permission is asserted. It never means "probably fine".
`source` says which surface produced the answer: `robots` · `meta` · `ai.txt` · `tdmrep` · `llms.txt`
· `none`. `none` means nothing on the site addresses this agent and RFC 9309's default (permission)
applies.
#### Per domain
```json
{
"rowType": "summary",
"domain": "example.com",
"aiPolicyStance": "partial",
"previousStance": "unstated",
"changeType": "changed",
"hasRobots": true,
"hasAiTxt": true,
"hasLlmsTxt": true,
"hasLlmsFull": false,
"hasTdmRep": true,
"agentsChecked": 25,
"agentsAllowed": 21,
"agentsBlocked": 4,
"agentsUnknown": 0,
"aiAgentsNamedInRobots": ["ccbot", "claudebot", "google-extended", "gptbot"],
"aiSpecific": true,
"robotsCrawlDelay": 2,
"llmsTxtSummary": "\"Example Docs\", 2 section(s), 3 link(s), 1 optional",
"aiTxtSummary": "2 agent group(s), 4 rule(s), asset types: *.jpg *.png, disallows all agents at /",
"tdmRepSummary": "TDM rights RESERVED for \"/\" — policy: https://example.com/tdm-policy.json",
"metaRobots": "index, follow, max-snippet:-1, noai, noimageai",
"xRobotsTag": "noai",
"noai": true,
"noimageai": true,
"noindex": false,
"status": "ok",
"fetches": 6,
"warning": null
}
```
***
### `aiPolicyStance` — the exact criteria
The one-word verdict. These four rules are the whole definition; nothing else feeds it.
| Stance | Criteria |
|---|---|
| **`blocked`** | Every agent that could be decided came back `allowed: false`. |
| **`partial`** | Some agents blocked and some allowed — **or** every agent is allowed at `/` but the site wrote path-level `Disallow` rules inside a group it named an AI agent in. |
| **`open`** | Every agent allowed **and** the site actually said something about AI: it named an AI agent in robots.txt, or published `ai.txt`, `llms.txt` or `tdmrep.json`, or set a `noai` tag. Deliberate openness. |
| **`unstated`** | Every agent allowed and **nothing** on the site mentions AI. The default the web ships with. |
| `null` | The domain could not be audited. `status` is `failed` and `warning` says why. |
`open` and `unstated` are deliberately different columns. "We welcome AI crawlers" and "nobody here
has thought about it" produce identical robots.txt behaviour and completely different conversations.
**Read `aiSpecific` alongside a `blocked` verdict.** A staging host with `User-agent: * / Disallow: /`
blocks every AI crawler without anyone making an AI decision. That row reports
`aiPolicyStance: "blocked"`, `aiSpecific: false`, `aiAgentsNamedInRobots: []`.
***
### Which surface wins — the precedence, stated openly
Five surfaces can speak about one agent and they routinely disagree. This order is a judgement, and
it is written here rather than buried in the code:
1. **`robots.txt`** — the only surface with access-control semantics that crawlers actually implement.
If it speaks, it decides.
2. **`meta` / `X-Robots-Tag` `noai`** — travels with the bytes; what sites set today.
3. **`ai.txt`** — a proposal nobody enforces. Evidence of intent, not of control.
4. **`tdmrep.json`** — a rights reservation, not a fetch rule. Strongest legally, weakest
operationally, so it decides only when nothing above it did.
5. **`llms.txt`** — can only ever say *yes*. Read as an explicit invitation when nothing above it
expressed a restriction.
Two consequences worth knowing:
- **`noindex` never blocks an AI agent.** It is a search directive that predates every AI crawler;
treating it as an AI opt-out would put a false "blocked" on a large share of the web. It is
reported on the summary row and left out of the decision. The one exception is a tag addressed to
an agent by name (``).
- **A `tdmrep` refusal is not a fetch refusal.** `allowed: false` with `source: "tdmrep"` means
*mining these bytes is reserved*, and the `rule` text says so.
***
### What these files actually are — no inflation
Only one of the five is a standard. Saying otherwise would misrepresent a publisher's legal position.
| Surface | Status | Enforced by crawlers? |
|---|---|---|
| `robots.txt` | **RFC 9309** — a published IETF standard | Yes, broadly. The one real access control here. |
| `X-Robots-Tag` / `meta name="robots"` | Long-standing convention; `noindex` widely honoured | `noindex` yes; **`noai` is a convention with no registry and no known enforcement** |
| `llms.txt` | **A proposal.** Authored by Jeremy Howard at llmstxt.org, v2 (3 Sep 2024), open for community input. No standards body. `llms-full.txt` is not even in the proposal — it is a convention | Read by some tools; permits nothing and forbids nothing |
| `ai.txt` | **A proposal** originated by Spawning.ai. No standards body, no registered media type | No crawler is known to enforce it |
| `tdmrep.json` | **W3C Community Group Final Report** (2 Feb 2024). The document states in terms: *"It is not a W3C Standard nor is it on the W3C Standards Track"* | Rarely, but it is the machine-readable form the EU DSM Article 4 opt-out is usually pointed at — the most likely of these to carry legal weight |
Full provenance, including which agent tokens were confirmed against the vendor's own documentation
and which were not, is in **[SOURCES.md](SOURCES.md)**.
***
### Tracking changes
Set `changesOnly: true`. State is kept per domain in the named key-value store
**`ai-crawl-policy-state`**, so it survives across runs. Schedule the actor daily or weekly and the
dataset becomes an alert feed.
A domain counts as **changed** when any of these move:
- an agent's permission flips (`allowed`)
- the **surface** that decided an agent's answer changes, even if the permission does not
- a policy file appears or disappears
- the stance moves
The second one matters more than it looks. When a site moves `CCBot` from "blocked by a site-wide
`noai` tag" to "blocked by a named robots.txt group", nothing a crawler does changes — but the
publisher has just taken a deliberate, targeted position. That row reports
`previousAllowed: false → allowed: false`, `previousSource: "meta" → source: "robots"`,
`changeType: "changed"`.
**Failures are always emitted, even in changes-only mode.** A silent row would read as "nothing
changed" when the truth is "this domain was never checked".
***
### Politeness and compliance
This actor audits crawler policy, so it has to be beyond reproach about its own behaviour.
- **`robots.txt` is honoured, with no opt-out**, for this actor's own user agent
(`ai-crawl-policy`). A disallowed check is skipped and reported as skipped.
- **An unreachable `robots.txt` fails closed.** RFC 9309 §2.3.1.4 requires treating it as a full
disallow, so nothing else is fetched and every agent answer for that domain is `null`, not `true`.
A `404` is different: §2.3.1.3 says an unavailable `robots.txt` means no restrictions.
- **A site's `Crawl-delay` overrides the configured delay whenever it is longer.** Never shorter.
- **Sequential per host**, at most six requests per domain, identifying user agent, `Retry-After`
respected, exponential backoff, 1 MB read cap, hard 200-domain ceiling per run.
- Only public files are read. Nothing is submitted, posted or logged in anywhere.
***
### Input
| Field | Type | Default | Notes |
|---|---|---|---|
| `domains` | array | — | **Required.** `example.com`, `www.example.com` or a full URL — the path is dropped, because a policy surface belongs to an origin, not a page. |
| `agents` | array | the built-in registry | Narrow the audit, e.g. `["GPTBot","ClaudeBot","Google-Extended"]`. A name the registry does not know is still audited exactly as written and flagged as unrecognised — never dropped. |
| `changesOnly` | boolean | `false` | Emit only what moved. See above. |
| `maxDomains` | integer | `0` (all) | Capped at 200 per run. |
| `delaySecs` | integer | `1` | Politeness pause per host; a site's `Crawl-delay` can only raise it. |
***
### The agent registry
25 tokens by default, each carrying `agentKind` — and the kinds are **not** interchangeable:
| `agentKind` | Meaning | Blocking it means |
|---|---|---|
| `training` | Collects content for model training | Your content stays out of the model |
| `search` | Indexes for an answer engine | **You disappear from that product's results.** Usually not what a publisher wants |
| `user` | Fetches a page because a human asked for it in a chat. Several vendors state these do not follow robots.txt | Often nothing — a stated preference, not an enforced block |
| `control` | **Not a crawler.** A token that exists only to be written in robots.txt as an opt-out signal (`Google-Extended`, `Applebot-Extended`) | A permission is withdrawn; no fetch is prevented |
Every row carries `agentVerified`. `true` means the exact token was read from the vendor's own
documentation when this actor was built, and `agentDocs` links to that page. `false` means the token
is widely published in third-party block-lists but was not confirmed first-party. Both are audited
identically — the flag just lets you tell a citable finding from a best-effort one. Details and
the confirmed-vs-not list: **[SOURCES.md](SOURCES.md)**.
***
### Limits — what this does not do
- **It does not run JavaScript.** It reads served HTML and text files. A `noai` tag injected
client-side will be missed. Sites that set it that way are rare and the header is the common path.
- **It reads the homepage only** for `meta`/`X-Robots-Tag`. A section-level `noai` on `/archive/`
will not be seen. Path-level rules in `robots.txt` *are* seen, and reported as `pathRestrictions`.
- **It evaluates permission at `/`.** A domain that allows `/blog` and blocks `/archive` for
`ClaudeBot` reports `allowed: true` with `pathRestrictions: 1` and a `partial` stance, not a
per-path grid.
- **It reports policy, not behaviour.** Several vendors state plainly that their user-initiated
agents ignore `robots.txt` entirely. A `false` in this dataset is what the site *asked for*, which
is not always what happens.
- **It is not legal advice.** `tdmrep` reservations and `noai` tags may or may not bind anyone
depending on jurisdiction and facts. This actor reports what is published; a lawyer decides what
it means.
***
### Development
```bash
npm install
npm test # node --test — 67 tests, fixtures only, no network
npm start # requires an INPUT in ./storage
```
Tests cover the RFC 9309 matcher (including the strict-token-matching case that stops a
`GPTBot-Image` rule being reported against `GPTBot`), all four secondary parsers, the source
precedence, the four stance criteria, and a three-run monitor cycle. Nothing in the suite touches the
network. The end-to-end verification against a local fixture server is recorded in
[SOURCES.md](SOURCES.md).
# Actor input Schema
## `domains` (type: `array`):
One domain per line — `example.com`, `www.example.com` or a full URL (the path is dropped; a policy surface belongs to an origin, not a page). Each domain costs at most six requests: robots.txt, the homepage, llms.txt, llms-full.txt, ai.txt and /.well-known/tdmrep.json.
## `agents` (type: `array`):
Leave empty to audit the built-in registry of 25 AI user agents (OpenAI, Anthropic, Google, Perplexity, Common Crawl, Meta, Apple, ByteDance, Amazon and more). Supply your own list to narrow it — for example just `GPTBot, ClaudeBot, Google-Extended`. Names you supply that the registry does not know are still audited, exactly as written, and flagged as unrecognised rather than dropped.
## `changesOnly` (type: `boolean`):
Emit a row only when a domain's AI policy moved since the last run of this actor on that domain — an agent's permission flipped, the deciding rule changed, or a policy file appeared or vanished. The first run emits everything as `new`. Failures are ALWAYS emitted even here: a silent row would read as "nothing changed" when the truth is "never checked". State is kept in the named key-value store `ai-crawl-policy-state`.
## `maxDomains` (type: `integer`):
0 = every domain supplied, bounded by an internal 200-domain ceiling.
## `delaySecs` (type: `integer`):
Politeness pause between two requests to the same host. A site's own `Crawl-delay` in robots.txt overrides this whenever it is longer. Requests to one host are always sequential.
## Actor input object example
```json
{
"domains": [
"apify.com",
"bbc.co.uk",
"nytimes.com"
],
"agents": [],
"changesOnly": false,
"maxDomains": 0,
"delaySecs": 1
}
```
# Actor output Schema
## `policy` (type: `string`):
Every row: one summary per domain plus one row per domain and AI user agent, each carrying the permission, the rule that decided it and which surface the rule came from.
## `summaries` (type: `string`):
One row per domain — the stance, which policy files exist, and how many agents are blocked. The shape to put on a dashboard.
## `agentMatrix` (type: `string`):
The domain-by-agent permission grid reduced to five columns — the shape a respect-check pipeline reads directly.
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"domains": [
"apify.com",
"bbc.co.uk",
"nytimes.com"
]
};
// Run the Actor and wait for it to finish
const run = await client.actor("multiplex/ai-crawl-policy").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = { "domains": [
"apify.com",
"bbc.co.uk",
"nytimes.com",
] }
# Run the Actor and wait for it to finish
run = client.actor("multiplex/ai-crawl-policy").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"domains": [
"apify.com",
"bbc.co.uk",
"nytimes.com"
]
}' |
apify call multiplex/ai-crawl-policy --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"type": "http",
"url": "https://mcp.apify.com/?tools=fetch-actor-details,multiplex/ai-crawl-policy"
}
}
}
```
The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).
## OpenAPI specification
Download the OpenAPI definition: https://api.apify.com/v2/actors/oAOib8RuaIAmZaPcH/builds/3jzKLYYzB9uUUz987/openapi.json