AI Crawler Policy Audit — robots.txt, llms.txt, noai

Two questions, one answer set.

Publishers: which AI crawlers may take your content right now — and did that change this week? AI teams: may we crawl this domain — and can we prove we checked?

Give it a list of domains. Get back, per domain and per AI user agent: allowed / not allowed, the exact rule that decided it, and which file that rule came from. Plus a one-word verdict per domain and an alert when any of it moves.

Most "AI blocker" checkers grep robots.txt for a handful of agent names. This one implements RFC 9309 properly — group selection, longest-match precedence, Allow beating Disallow on ties, * and $ wildcards, Crawl-delay — then reads the four other surfaces a site can express AI policy on, and tells you which one actually decided the answer.

---

Why it exists

Site owners now express AI policy in five different places that contradict each other constantly. A publisher blocks GPTBot in robots.txt, publishes an llms.txt inviting models in, and has a noai tag left over from a 2023 CMS plugin. An AI team fetching that domain has to decide which one governs. This actor answers that question the same way every time, shows its working, and records the answer so the next run can tell you what moved.

You are	What you get
A publisher	The full picture of what you are exposing, including the files you forgot you published. aiPolicyStance in one column across your whole portfolio.
A publisher with an editorial line on AI	Proof of what your policy said on a given date, and an alert the day it changes.
An AI / agent team	A respect-check before you crawl: domain, agent, allowed, rule, source — five columns, machine-readable, with the rule quoted so a compliance reviewer can follow it.
A researcher or journalist	Adoption data. Who blocks whom, who publishes llms.txt, who has reserved TDM rights, and how that moves over time.
Anyone doing competitive work	An alert when a competitor's AI policy shifts — often the first public sign of a licensing deal or a legal position.

---

What it checks — at most six requests per domain

#	Request	What it answers
1	/robots.txt	The AI-agent groups, per-agent permission, path restrictions, Crawl-delay. The primary evidence.
2	/ (homepage)	<meta name="robots"> and the X-Robots-Tag response header — noai, noimageai, noindex.
3	/llms.txt	Present? Valid? Title, summary, sections, link count.
4	/llms-full.txt	Present? (Existence only — the file is not read.)
5	/ai.txt	Present? Which agents it names, whether it disallows everything, which asset types it names.
6	/.well-known/tdmrep.json	A TDM rights reservation for /, and the policy URL if one is given.

Requests to one host are sequential and spaced. robots.txt is fetched first and honoured for this actor's own requests — if a site disallows ai-crawl-policy from /llms.txt, that check is skipped and reported as skipped, never guessed.

---

Output

Two row types in one dataset, distinguished by rowType. The Apify UI ships three views: Domain summary, Per agent, and Changes only.

Per domain + agent

{
  "id": "0d9c…",
  "rowType": "agent",
  "domain": "example.com",
  "agent": "GPTBot",
  "allowed": false,
  "rule": "Disallow: /",
  "source": "robots",
  "changeType": "changed",
  "previousAllowed": true,
  "previousSource": "none",
  "namedInRobots": true,
  "pathRestrictions": 0,
  "crawlDelay": null,
  "agentVendor": "OpenAI",
  "agentKind": "training",
  "agentVerified": true,
  "agentNote": "Crawls content that may be used to train OpenAI foundation models.",
  "agentDocs": "https://developers.openai.com/api/docs/bots",
  "status": "ok",
  "checkedAt": "2026-08-19T11:02:44.118Z",
  "previousCheckedAt": "2026-08-12T11:00:07.554Z",
  "warning": null
}

allowed is true, false, or null. null means unknown — the audit could not read robots.txt, so no permission is asserted. It never means "probably fine".

source says which surface produced the answer: robots · meta · ai.txt · tdmrep · llms.txt · none. none means nothing on the site addresses this agent and RFC 9309's default (permission) applies.

Per domain

{
  "rowType": "summary",
  "domain": "example.com",
  "aiPolicyStance": "partial",
  "previousStance": "unstated",
  "changeType": "changed",
  "hasRobots": true,
  "hasAiTxt": true,
  "hasLlmsTxt": true,
  "hasLlmsFull": false,
  "hasTdmRep": true,
  "agentsChecked": 25,
  "agentsAllowed": 21,
  "agentsBlocked": 4,
  "agentsUnknown": 0,
  "aiAgentsNamedInRobots": ["ccbot", "claudebot", "google-extended", "gptbot"],
  "aiSpecific": true,
  "robotsCrawlDelay": 2,
  "llmsTxtSummary": "\"Example Docs\", 2 section(s), 3 link(s), 1 optional",
  "aiTxtSummary": "2 agent group(s), 4 rule(s), asset types: *.jpg *.png, disallows all agents at /",
  "tdmRepSummary": "TDM rights RESERVED for \"/\" — policy: https://example.com/tdm-policy.json",
  "metaRobots": "index, follow, max-snippet:-1, noai, noimageai",
  "xRobotsTag": "noai",
  "noai": true,
  "noimageai": true,
  "noindex": false,
  "status": "ok",
  "fetches": 6,
  "warning": null
}

---

aiPolicyStance — the exact criteria

The one-word verdict. These four rules are the whole definition; nothing else feeds it.

Stance	Criteria
blocked	Every agent that could be decided came back allowed: false.
partial	Some agents blocked and some allowed — or every agent is allowed at / but the site wrote path-level Disallow rules inside a group it named an AI agent in.
open	Every agent allowed and the site actually said something about AI: it named an AI agent in robots.txt, or published ai.txt, llms.txt or tdmrep.json, or set a noai tag. Deliberate openness.
unstated	Every agent allowed and nothing on the site mentions AI. The default the web ships with.
null	The domain could not be audited. status is failed and warning says why.

open and unstated are deliberately different columns. "We welcome AI crawlers" and "nobody here has thought about it" produce identical robots.txt behaviour and completely different conversations.

Read aiSpecific alongside a blocked verdict. A staging host with User-agent: * / Disallow: / blocks every AI crawler without anyone making an AI decision. That row reports aiPolicyStance: "blocked", aiSpecific: false, aiAgentsNamedInRobots: [].

---

Which surface wins — the precedence, stated openly

Five surfaces can speak about one agent and they routinely disagree. This order is a judgement, and it is written here rather than buried in the code:

1. robots.txt — the only surface with access-control semantics that crawlers actually implement. If it speaks, it decides.
2. meta / X-Robots-Tag noai — travels with the bytes; what sites set today.
3. ai.txt — a proposal nobody enforces. Evidence of intent, not of control.
4. tdmrep.json — a rights reservation, not a fetch rule. Strongest legally, weakest operationally, so it decides only when nothing above it did.
5. llms.txt — can only ever say yes. Read as an explicit invitation when nothing above it expressed a restriction.

Two consequences worth knowing:

• noindex never blocks an AI agent. It is a search directive that predates every AI crawler; treating it as an AI opt-out would put a false "blocked" on a large share of the web. It is reported on the summary row and left out of the decision. The one exception is a tag addressed to an agent by name (<meta name="GPTBot" content="noindex">).
• A tdmrep refusal is not a fetch refusal. allowed: false with source: "tdmrep" means mining these bytes is reserved, and the rule text says so.

---

What these files actually are — no inflation

Only one of the five is a standard. Saying otherwise would misrepresent a publisher's legal position.

Surface	Status	Enforced by crawlers?
robots.txt	RFC 9309 — a published IETF standard	Yes, broadly. The one real access control here.
X-Robots-Tag / meta name="robots"	Long-standing convention; noindex widely honoured	noindex yes; noai is a convention with no registry and no known enforcement
llms.txt	A proposal. Authored by Jeremy Howard at llmstxt.org, v2 (3 Sep 2024), open for community input. No standards body. llms-full.txt is not even in the proposal — it is a convention	Read by some tools; permits nothing and forbids nothing
ai.txt	A proposal originated by Spawning.ai. No standards body, no registered media type	No crawler is known to enforce it
tdmrep.json	W3C Community Group Final Report (2 Feb 2024). The document states in terms: "It is not a W3C Standard nor is it on the W3C Standards Track"	Rarely, but it is the machine-readable form the EU DSM Article 4 opt-out is usually pointed at — the most likely of these to carry legal weight

Full provenance, including which agent tokens were confirmed against the vendor's own documentation and which were not, is in SOURCES.md.

---

Tracking changes

Set changesOnly: true. State is kept per domain in the named key-value store ai-crawl-policy-state, so it survives across runs. Schedule the actor daily or weekly and the dataset becomes an alert feed.

A domain counts as changed when any of these move:

• an agent's permission flips (allowed)
• the surface that decided an agent's answer changes, even if the permission does not
• a policy file appears or disappears
• the stance moves

The second one matters more than it looks. When a site moves CCBot from "blocked by a site-wide noai tag" to "blocked by a named robots.txt group", nothing a crawler does changes — but the publisher has just taken a deliberate, targeted position. That row reports previousAllowed: false → allowed: false, previousSource: "meta" → source: "robots", changeType: "changed".

Failures are always emitted, even in changes-only mode. A silent row would read as "nothing changed" when the truth is "this domain was never checked".

---

Politeness and compliance

This actor audits crawler policy, so it has to be beyond reproach about its own behaviour.

• robots.txt is honoured, with no opt-out, for this actor's own user agent (ai-crawl-policy). A disallowed check is skipped and reported as skipped.
• An unreachable robots.txt fails closed. RFC 9309 §2.3.1.4 requires treating it as a full disallow, so nothing else is fetched and every agent answer for that domain is null, not true. A 404 is different: §2.3.1.3 says an unavailable robots.txt means no restrictions.
• A site's Crawl-delay overrides the configured delay whenever it is longer. Never shorter.
• Sequential per host, at most six requests per domain, identifying user agent, Retry-After respected, exponential backoff, 1 MB read cap, hard 200-domain ceiling per run.
• Only public files are read. Nothing is submitted, posted or logged in anywhere.

---

Input

Field	Type	Default	Notes
domains	array	—	Required. example.com, www.example.com or a full URL — the path is dropped, because a policy surface belongs to an origin, not a page.
agents	array	the built-in registry	Narrow the audit, e.g. ["GPTBot","ClaudeBot","Google-Extended"]. A name the registry does not know is still audited exactly as written and flagged as unrecognised — never dropped.
changesOnly	boolean	false	Emit only what moved. See above.
maxDomains	integer	0 (all)	Capped at 200 per run.
delaySecs	integer	1	Politeness pause per host; a site's Crawl-delay can only raise it.

---

The agent registry

25 tokens by default, each carrying agentKind — and the kinds are not interchangeable:

agentKind	Meaning	Blocking it means
training	Collects content for model training	Your content stays out of the model
search	Indexes for an answer engine	You disappear from that product's results. Usually not what a publisher wants
user	Fetches a page because a human asked for it in a chat. Several vendors state these do not follow robots.txt	Often nothing — a stated preference, not an enforced block
control	Not a crawler. A token that exists only to be written in robots.txt as an opt-out signal (Google-Extended, Applebot-Extended)	A permission is withdrawn; no fetch is prevented

Every row carries agentVerified. true means the exact token was read from the vendor's own documentation when this actor was built, and agentDocs links to that page. false means the token is widely published in third-party block-lists but was not confirmed first-party. Both are audited identically — the flag just lets you tell a citable finding from a best-effort one. Details and the confirmed-vs-not list: SOURCES.md.

---

Limits — what this does not do

• It does not run JavaScript. It reads served HTML and text files. A noai tag injected client-side will be missed. Sites that set it that way are rare and the header is the common path.
• It reads the homepage only for meta/X-Robots-Tag. A section-level noai on /archive/ will not be seen. Path-level rules in robots.txt are seen, and reported as pathRestrictions.
• It evaluates permission at /. A domain that allows /blog and blocks /archive for ClaudeBot reports allowed: true with pathRestrictions: 1 and a partial stance, not a per-path grid.
• It reports policy, not behaviour. Several vendors state plainly that their user-initiated agents ignore robots.txt entirely. A false in this dataset is what the site asked for, which is not always what happens.
• It is not legal advice. tdmrep reservations and noai tags may or may not bind anyone depending on jurisdiction and facts. This actor reports what is published; a lawyer decides what it means.

---

Development

npm install
npm test      # node --test — 67 tests, fixtures only, no network
npm start     # requires an INPUT in ./storage

Tests cover the RFC 9309 matcher (including the strict-token-matching case that stops a GPTBot-Image rule being reported against GPTBot), all four secondary parsers, the source precedence, the four stance criteria, and a three-run monitor cycle. Nothing in the suite touches the network. The end-to-end verification against a local fixture server is recorded in SOURCES.md.