AI Search Readiness Audit: AI crawlers, llms.txt and basic SEO

Check any list of websites for AI search readiness in one run. For each domain the Actor reads the site's own robots.txt, llms.txt, first sitemap file and (optionally) a few public pages, and returns one clean row per domain: which AI crawlers are allowed or blocked, whether the site publishes llms.txt, whether a sitemap exists, how the home page looks to a crawler that does not run JavaScript, and a 0-100 readiness score with a list of fix codes.

No login, no browser, no proxy and no API key. Requests are polite: at most one request per second per domain, a clear user agent (AISearchReadinessAudit), and the Actor follows the robots.txt of the site it audits.

What you get per domain

• AI crawler matrix (18 crawlers): for each of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Applebot-Extended, CCBot, Bytespider, Amazonbot, meta-externalagent, cohere-ai, DuckAssistBot and MistralAI-User: allowed, blocked or partial, with the robots.txt rule that decided it and whether it came from a named group or the * group. Rules follow RFC 9309 (longest match wins, Allow wins ties, wildcards * and $).
• AI policy summary: open, blocks_training_only, blocks_some_search_bots, blocks_ai_search, or unknown.
• llms.txt and llms-full.txt: found or not, size, and whether llms.txt follows the format on llmstxt.org (H1 title, summary, sections, link count). HTML pages that answer on that path (soft 404) are not counted as found.
• Sitemap: found or not, type, number of entries in the first file.
• Home page and sampled pages (optional): HTTP status, response time, HTTPS, title and meta description length, canonical, lang, hreflang count, Open Graph tags, H1/H2 count, visible words in the raw HTML (a low number often means the content is built by JavaScript), JSON-LD types, image alt coverage, noindex and AI opt-out meta tags, security headers (HSTS, CSP, X-Content-Type-Options).
• Score and issues: geoScore (0-100) and issues such as SEARCH_BOT_BLOCKED:PerplexityBot, LLMS_TXT_MISSING, JSON_LD_MISSING.

Input

Field	Meaning	Default
domains	Websites to check, e.g. example.com (URLs also work, only the domain is used)	required
checkPages	Also read the sitemap and home page and compute the full score. Off = fast check of robots.txt and llms.txt only (3 requests per domain)	on
pagesPerDomain	Home page plus pages spread evenly over the first sitemap file (1-10)	1
maxConcurrency	Domains checked in parallel (each domain still gets at most 1 request per second)	5
delaySeconds	Pause between requests to one domain (minimum 1)	1
includePageTexts	Also return title and meta description text (default: lengths only)	off

Example:

{
  "domains": ["example.com", "wikipedia.org", "dr.dk"],
  "checkPages": true,
  "pagesPerDomain": 3
}

Output example (shortened)

{
  "domain": "dr.dk",
  "status": "ok",
  "aiPolicy": "blocks_ai_search",
  "geoScore": 62,
  "scoreScope": "robots.txt, llms.txt, sitemap and home page",
  "issues": ["SEARCH_BOT_BLOCKED:OAI-SearchBot", "SEARCH_BOT_BLOCKED:Claude-SearchBot", "SEARCH_BOT_BLOCKED:PerplexityBot", "LLMS_TXT_MISSING"],
  "trainingBotsBlocked": ["GPTBot", "ClaudeBot", "Google-Extended"],
  "crawlers": [{"bot": "GPTBot", "vendor": "OpenAI", "purpose": "training", "status": "blocked", "rule": "Disallow: /", "ruleSource": "named group"}],
  "llmsTxt": {"found": false}
}

How the score works

geoScore is a rule of thumb, not an official ranking factor of any search engine or AI product. Points: search bots not blocked (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) 30, valid llms.txt 10, sitemap 10, and for the home page: title length 10-70 characters 8, meta description 50-170 characters 8, canonical 4, lang 3, exactly one H1 6, JSON-LD 10, at least 3 Open Graph tags 4, at least 150 words in raw HTML 7. The score is the share of the points that were checked, so scoreScope tells you what it is based on. Do not compare scores with different scopes. Blocking training crawlers is a legitimate choice and is not penalised.

Limits and honesty

• The Actor reads robots.txt rules only. A firewall, CDN or bot-protection service can block AI crawlers even when robots.txt allows them. That cannot be seen from robots.txt.
• Some AI products fetch pages on a user's request and may ignore robots.txt (the vendors document this for user-triggered fetchers). The purpose field tells you which crawlers are of that kind.
• The crawler list follows the vendors' public documentation (OpenAI, Anthropic, Perplexity, Google, Apple); other vendors' tokens can change. Unknown bots are not reported.
• If /robots.txt cannot be read (server error), the Actor reports aiPolicy: unknown and fetches no further pages from that domain. A 404 means no rules (everything allowed), as in RFC 9309.
• Pages are read as raw HTML without JavaScript, up to 1 MB each. The Actor skips pages that the site's robots.txt disallows for its user agent.
• Check only websites you are allowed to check. The Actor does not log in, does not solve CAPTCHAs and does not hide itself.
• Output contains site-level facts only (rules, counts, lengths, schema types, headers). No contact details, names of people or user content.

Pricing

Pay per checked domain (one row per domain, about 10 USD per 1,000 domains). Domains that cannot be reached are not charged; they are listed in the key-value store record FAILED_DOMAINS. If you set a maximum cost per run, the Actor stops when the limit is reached and says so in the log and the run status.

Sources and standards

robots.txt: RFC 9309. llms.txt: llmstxt.org. Crawler tokens: documentation from OpenAI, Anthropic, Perplexity, Google and Apple.