AI Crawler & robots.txt Readiness Audit

Give it a list of domains. For each one it fetches robots.txt and reports whether 21 AI crawler and fetcher tokens are allowed, blocked, or not mentioned. Every token on the list is documented by its vendor. For each bot you get the rule that decided the result and its line number. Each domain also gets these checks:

• whether a Sitemap: line is declared
• whether /llms.txt exists (presence only)
• any Content-Signal: lines (listed, not evaluated)
• warnings for things that make the result less reliable

You get one dataset row per domain.

To check a single site in the browser, use the free checker at aicrawlready.com. Full input and output documentation for this Actor is at aicrawlready.com/tools.

What it checks

For every domain:

1. robots.txt fetch. It sends one GET https://<host>/robots.txt and follows up to 5 redirects. The row records the HTTP status, the final URL after redirects, the redirect chain, the content type and the size.

2. Rule evaluation for each bot. It evaluates rules per RFC 9309 plus Google's documented matching behaviour:

   • User-agent values are compared case-insensitively on the product token.
   • Every group that names the bot is merged. If none names it, the * group(s) apply. If there is no * group either, everything is allowed.
   • The longest matching pattern wins, and Allow wins a tie. * and $ are supported.
   • The root path / is always checked. Paths you add in checkPaths are checked too. Paths are matched against the rules only; they are never fetched.

3. HTTP status handling (RFC 9309):

   • 2xx: the rules are evaluated.
   • 4xx, including 401, 403 and 404: no restrictions, so every bot is reported as allowed. A 401 or 403 also adds a bot-protection warning.
   • 5xx or 429: treated as a complete disallow, so every bot is reported as blocked, with a warning.
   • More than 5 redirects: treated as unavailable, like a 4xx.

4. Sitemap. Checks whether any Sitemap: line is present and lists up to 50 URLs.

5. llms.txt. Sends one GET for /llms.txt and reads at most 64 KiB. The status is one of:

   • present: 2xx with a non-HTML body
   • html-response: 2xx but the body is HTML, which is often a soft 404
   • absent: 404 or other 4xx, or an empty body
   • unknown: 401, 403, 429, 5xx, a timeout or an error

   This is a factual presence check only. The audit makes no claim about whether any AI system reads or respects llms.txt.

6. Content-Signal. Content-Signal: lines are a non-standard extension. They are listed with their line numbers and are not evaluated.

7. Warnings. A warning is added for:

   • 401/403 responses, which may be bot protection
   • 5xx/429 responses
   • redirects, including host changes
   • files larger than 500 KiB, where the rest is ignored (Google documents the same 500 KiB limit)
   • robots.txt served as HTML
   • a file with no user-agent groups
   • timeouts
   • redirects to private addresses, which are refused

What "status" means for each bot

status	Meaning
blocked	/ is disallowed for this token. The cause can be a group that names the bot, the * fallback, or a 5xx/429 robots.txt.
allowed	A group names this token, and / is allowed.
not-mentioned	No group names this token, and / is not blocked. That covers three cases: the * group allows it, no group applies, or robots.txt returned a 4xx.

Each bot also has these fields:

• mention: specific, wildcard, none or n/a
• verdict: allowed or blocked, for /
• rule: {type, pattern, line, text}, the winning rule (null when no rule matched)
• line: the line number of that rule
• paths[]: one verdict for each checked path

Limits (please read)

• Rules only, no crawling. Only robots.txt and /llms.txt are requested. The actor does not fetch pages, render JavaScript, or read <meta name="robots"> tags or X-Robots-Tag headers.
• Only / and the paths you list are checked. A bot shown as allowed at / can still be blocked from other sections.
• Standard * fallback only. Some vendors document their own fallback rules. Amazon says Amzn-SearchBot may follow rules written for other search bots when it isn't named. Apple says Applebot follows Googlebot rules when Applebot isn't named. These vendor-specific fallbacks are not modelled.
• robots.txt is advisory, not enforcement. It is a request that well-behaved crawlers choose to honour. Several vendors say that user-triggered fetchers may not follow it; these are marked userTriggered: true. This audit reports what the file says, not what any crawler actually does.
• Responses can differ by client. A CDN or bot-protection layer may give this actor a different response (for example 403) than it gives real crawlers. When that seems likely, the row carries a warning.
• Scheme. Bare domains are checked over https://. If you need http://, pass the full URL.
• Bot list is fixed. The list contains 21 tokens, each verified against vendor documentation on 2026-09-25. Claude-Web, anthropic-ai and Bytespider are deliberately left out because current vendor docs for them could not be verified. Other user-agent groups that appear in a file do not change the result for the listed tokens.

Safety and politeness

• Only http:// and https:// on the default ports are allowed. URLs with credentials are refused.
• Private, loopback, link-local, CGNAT, multicast and documentation IP ranges (IPv4 and IPv6) are blocked.
  • The check runs at DNS-resolution time, on the address the socket actually connects to.
  • It runs again on every redirect hop.
• Timeouts are 10 s per request, covering all redirect hops together. Response bodies are capped at 500 KiB for robots.txt and 64 KiB for llms.txt.
• Each domain gets at most 2 requests (robots.txt, then llms.txt), made one after the other. By default 5 domains run in parallel (maxConcurrency, 1–20). Duplicate origins are skipped.
• Requests identify themselves with the user agent AI-Crawler-Readiness-Audit/0.1 (Apify actor; fetches robots.txt and llms.txt only).

Input

Field	Type	Default	Notes
domains	string[]	— (required)	Domains or URLs; each is reduced to its origin. 1–1000 entries.
bots	string[]	all 21	Optional filter; tokens from the table below, case-insensitive.
checkPaths	string[]	[]	Extra paths besides / (max 20), e.g. /blog/.
includeRawRobots	boolean	false	Adds the robots.txt text to each row.
maxConcurrency	integer	5	1–20 domains in parallel.

{
  "domains": ["example.com", "https://www.example.org/some/page"],
  "bots": ["GPTBot", "ClaudeBot", "Google-Extended"],
  "checkPaths": ["/blog/"],
  "includeRawRobots": false,
  "maxConcurrency": 5
}

Output

Each row is one domain. Here is an illustrative item, trimmed to one bot:

{
  "input": "example.com",
  "domain": "example.com",
  "origin": "https://example.com",
  "robotsUrl": "https://example.com/robots.txt",
  "finalUrl": "https://www.example.com/robots.txt",
  "fetchStatus": 200,
  "robotsState": "ok",
  "robotsTxt": { "contentType": "text/plain", "bytes": 412, "truncated": false, "redirects": 1,
                 "redirectChain": ["https://example.com/robots.txt", "https://www.example.com/robots.txt"], "groupCount": 3 },
  "checkedPaths": ["/", "/blog/"],
  "bots": [
    {
      "token": "GPTBot", "vendor": "OpenAI",
      "purpose": "Crawls content that may be used to train OpenAI foundation models",
      "docs": "https://platform.openai.com/docs/bots", "userTriggered": false,
      "status": "blocked", "mention": "specific", "verdict": "blocked",
      "rule": { "type": "disallow", "pattern": "/", "line": 5, "text": "Disallow: /" },
      "line": 5, "groupAgents": ["GPTBot"], "groupLines": [4],
      "paths": [
        { "path": "/", "verdict": "blocked", "rule": { "type": "disallow", "pattern": "/", "line": 5, "text": "Disallow: /" } },
        { "path": "/blog/", "verdict": "blocked", "rule": { "type": "disallow", "pattern": "/", "line": 5, "text": "Disallow: /" } }
      ]
    }
  ],
  "summary": { "botsChecked": 21, "allowed": 1, "blocked": 4, "notMentioned": 16,
               "namedInRobots": 5, "blockedViaWildcard": 0, "blockedOnAnyPath": 4 },
  "sitemap": { "declared": true, "count": 1, "urls": ["https://www.example.com/sitemap.xml"] },
  "llmsTxt": { "url": "https://example.com/llms.txt", "status": "absent", "httpStatus": 404,
               "finalUrl": "https://www.example.com/llms.txt", "contentType": "text/html", "bytes": 1256, "note": null },
  "contentSignals": [{ "line": 2, "value": "search=yes, ai-train=no" }],
  "warnings": [
    "robots.txt redirected 1 time(s) to https://www.example.com/robots.txt (host changed from example.com to www.example.com); rules were read from the final URL.",
    "Found 1 Content-Signal line(s) (a non-standard usage-preference extension). They are listed for reference only and are not evaluated."
  ],
  "error": null,
  "audited": true,
  "checkedAt": "2026-09-25T12:00:00.000Z"
}

Rows can also report failures:

• Failed domains, such as DNS failures, timeouts or refused private addresses: audited: false, error: {code, message}, and empty bots.
• Invalid inputs: error.code: "INVALID_INPUT".

The dataset's Overview view shows one table row per domain with the key columns.

Bot tokens checked

Each token was verified against the vendor's own documentation on 2026-09-25.

Token	Vendor	What the vendor says it controls	Vendor docs
GPTBot	OpenAI	Content that may be used to train foundation models	OpenAI crawlers
OAI-SearchBot	OpenAI	Appearing in ChatGPT search results	OpenAI crawlers
ChatGPT-User	OpenAI	User actions in ChatGPT/GPTs (vendor notes robots.txt rules may not apply)	OpenAI crawlers
ClaudeBot	Anthropic	Content that may go into model training	Anthropic help center
Claude-SearchBot	Anthropic	Search result quality	Anthropic help center
Claude-User	Anthropic	Fetches when a user asks Claude	Anthropic help center
Google-Extended	Google	Gemini training and grounding (control token; no separate crawler; not a Search ranking signal)	Google common crawlers
PerplexityBot	Perplexity	Perplexity search results (not model training)	Perplexity crawlers
Perplexity-User	Perplexity	User-requested fetches (vendor says it generally ignores robots.txt)	Perplexity crawlers
CCBot	Common Crawl	Common Crawl archive	CCBot
Applebot-Extended	Apple	Training Apple foundation models (control token; doesn't crawl)	About Applebot
Meta-ExternalAgent	Meta	Uses such as training AI models or indexing	Meta web crawlers
Meta-WebIndexer	Meta	Meta AI search results/citations	Meta web crawlers
Meta-ExternalFetcher	Meta	User-requested fetches (vendor says it may bypass robots.txt)	Meta web crawlers
Amazonbot	Amazon	Amazon products; may be used to train Amazon AI models	About Amazonbot
Amzn-SearchBot	Amazon	Search experiences such as Alexa (not training)	About Amazonbot
Amzn-User	Amazon	User actions such as Alexa queries (may not follow all robots.txt directives)	About Amazonbot
DuckAssistBot	DuckDuckGo	AI-assisted answers (not training)	DuckAssistBot
MistralAI-Training	Mistral AI	Training datasets	Mistral crawlers
MistralAI-Index	Mistral AI	Indexing for Mistral search	Mistral crawlers
MistralAI-User	Mistral AI	User-requested fetches	Mistral crawlers

Pricing

The actor uses pay-per-event pricing with a single event, domain-audited:

• Charged: once per domain whose robots.txt returned an HTTP response (any status) and was fully evaluated.
• Not charged: invalid inputs, DNS failures, timeouts, and refused private-address targets. These still get a row with an error.

The Actor owner sets the price in Apify Console; see the Actor's pricing tab for the current amount. The run stops starting new domains once your per-run cost limit is reached. Details are in PRICING.md.

Related tools

• aicrawlready.com: free single-site AI crawler and robots.txt checker, plus guides.
• aicrawlready.com/tools: docs for this Actor, the AI Crawler Checker Chrome extension and the public JSON APIs.

Local development

npm install
npm test            # matcher, input normalizer, and mocked-network integration tests
npm run build       # TypeScript -> dist/
npm run local -- nytimes.com vercel.com anthropic.com   # stubbed Actor, writes local-output.json
npx apify-cli run -p                                    # real SDK with local storage (./storage)
npx apify-cli validate-schema

The matcher and bot list were copied from a companion Next.js tool (seo-microtool) and are kept in sync by hand.