llms.txt Checker & AI Crawler Auditor — GEO Readiness Score & Fix List

Use this llms.txt checker and AI crawler auditor to see, per domain, whether AI crawlers — GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Googlebot and more — can reach and cite your content, whether your llms.txt is valid, and exactly what to fix to stay citable in ChatGPT, Claude, Perplexity and Google AI Overviews. You get a GEO-readiness verdict, a 0–100 score, and a prioritized fix list — per domain.

It evaluates 22 AI crawler and opt-out tokens across four tiers — AI search/citation crawlers, training crawlers, on-demand agent fetchers, and the Google-Extended / Applebot-Extended opt-out tokens — and runs 16 checks per domain. It reads only public well-known files (robots.txt, llms.txt, llms-full.txt, sitemap.xml) plus one page's <head>, with a single polite request each: no crawl, no JS rendering, no raw page content stored.

What makes it different

Most "AI crawler checker" tools only tell you a token is present or absent. This one tiers crawler compliance so you learn the truth:

• A Disallow: PerplexityBot or Bytespider in robots.txt won't actually stop them — those operators are documented ignoring robots.txt (only a WAF will) — so we flag an ineffective block.
• It separates AI-search/citation bots (blocking them means you vanish from ChatGPT, Claude, Perplexity and Google AI answers — a P0 error) from training bots (blocking them is a legitimate choice) and from opt-out tokens (Google-Extended, Applebot-Extended — directives, not crawlers).
• It reads the page, not just robots.txt: content-in-raw-HTML (no-JS), noindex, canonical, JSON-LD, Open Graph, headings, freshness and llms.txt validity all roll into one weighted score.

Use cases

• Audit your own sites and subdomains before or after a robots.txt, CDN or Cloudflare change.
• Agencies: batch-audit a client portfolio and hand over a prioritized fix list.
• Catch the silent killer: a noindex, a JS-only page, or an accidental AI-search block that is costing you citations right now.
• Monitor: schedule it weekly and alert when a domain's geo_ready_verdict drops or search_bots_blocked > 0.

How to check if AI crawlers can reach your site (3 steps)

1. Add your domains — { "domains": ["yourbrand.com", "docs.yourbrand.com"] }. Optionally set verifyToken and drop <meta name="geo-audit-verify" content="TOKEN"> on the page to prove ownership.
2. Run the Actor — it fetches the well-known files and one page per domain, then scores every check.
3. Read the verdict — READY / NEEDS_WORK / NOT_READY plus top_fixes in priority order (P0 first).

Input

Field	Required	Description
domains	yes	Domains or full URLs to audit — ideally your own.
auditPath	no	Page path for on-page signals (default /).
verifyToken	no	Token to match against the geo-audit-verify meta tag; marks ownership_verified=true.
timeoutSecs	no	Per-request timeout in seconds (default 15).

{ "domains": ["yourbrand.com", "docs.yourbrand.com"], "auditPath": "/" }

Output example

A clean domain returns a high score with no fixes; a misconfigured one lists exact actions. These are measured results for apify.com and example.com:

{
  "domain": "apify.com",
  "geo_ready_verdict": "READY",
  "geo_score": 96,
  "search_bots_blocked": 0,
  "training_bots_blocked": 0,
  "llms_txt_present": true,
  "llms_txt_validity": "valid",
  "issue_count": 1,
  "top_fixes": ["[WARN] freshness_dates: no freshness dates"]
}

{
  "domain": "example.com",
  "geo_ready_verdict": "NOT_READY",
  "geo_score": 59,
  "llms_txt_present": false,
  "llms_txt_validity": "absent",
  "content_in_raw_html": false,
  "issue_count": 8,
  "top_fixes": [
    "[FAIL] content_in_raw_html: little/no content without JS — AI crawlers may see an empty page",
    "[WARN] sitemap_declared: no valid /sitemap.xml",
    "[WARN] structured_data_jsonld: no schema.org JSON-LD"
  ]
}

Per domain you also get ai_crawler_access (allow/block per token), noncompliant_blocks_flagged, google_extended_optout, applebot_extended_optout, the full checks[] array, title, meta_description, canonical_url, jsonld_types and http_status.

The 16 checks (per domain)

HTTPS/200 + redirect chain · AI-search crawlers not blocked (P0) · ineffective-block flag (gray-zone/non-compliant crawlers) · training opt-out tokens · sitemap present/valid/referenced · content in raw HTML (no-JS, P0) · noindex/nosnippet (P0) · AI opt-out meta (noai) · schema.org JSON-LD · canonical · title + meta description · Open Graph · semantic structure (single H1, <main>/<article>) · freshness dates · llms.txt presence + validity · llms-full.txt. → weighted score + READY / NEEDS_WORK / NOT_READY verdict + top_fixes.

Pricing (pay-per-event) & cost example

Event	Price	When
domain-audited (primary)	$0.04	per domain fully audited
issue-found	$0.01	per fix-worthy issue flagged (capped at 8/domain)

A clean domain costs $0.04; a domain with 4 issues, $0.08; a badly-misconfigured one, at most $0.12. Measured platform compute is <$0.001/domain (<3% of revenue). Compare: brand-tracker SaaS runs $25–99/mo per domain; this is per-audit and pennies.

Integrate

Schedule it (weekly), then use Apify webhooks/API/MCP/Make/n8n to alert on any domain whose geo_ready_verdict drops or whose search_bots_blocked > 0. Results land in a dataset you can pull from the API, export to JSON/CSV/Excel, or wire into a dashboard.

Related Actors

• HTTP Security Headers & OWASP Posture Auditor — audit your CSP, HSTS, XFO, and 20+ security headers alongside your GEO score.
• Email Deliverability & Domain Posture Auditor — check SPF, DKIM, DMARC, BIMI, and MTA-STS across your domains.
• Technical On-Page SEO & Meta Auditor — verify the on-page SEO signals (title, meta, H1, canonical, schema, broken links) that AI answer engines also read.

FAQ

What is an llms.txt checker and why do I need one? An llms.txt checker validates the llms.txt file your site publishes for AI agents — it must contain an H1 and a list of links to be spec-valid. This Actor reports whether your llms.txt is valid, present but malformed, or absent, and includes that in your GEO score.

How do I check if AI crawlers are blocked from my site? Point the Auditor at your domain. It parses your robots.txt and evaluates 22 AI crawler and opt-out tokens, then reports search_bots_blocked and training_bots_blocked, and flags the blocks that won't actually work.

Does GPTBot respect robots.txt? GPTBot and most OpenAI, Anthropic and Google crawlers do honor robots.txt. Some others — ByteDance's Bytespider and Perplexity's agent fetcher — are documented as ignoring it, so a Disallow alone won't stop them; only edge/WAF rules will. The Auditor flags those as ineffective blocks.

Is llms.txt actually used? It is an emerging convention, not a ranking factor today; AI tooling and agents increasingly look for it. Treat it as a low-cost, optional nicety: the Actor reports it as INFO, not a failure.

How is GEO readiness different from traditional SEO? GEO (Generative Engine Optimization) focuses on making your content discoverable and citable by AI answer engines (ChatGPT, Claude, Perplexity, Google AI Overviews). Traditional SEO optimizes for search-engine rankings; GEO adds llms.txt, AI-opt-out meta, semantic structure, and crawler-access compliance.

Does this tool scrape or save any data from my site? No. The Auditor reads only standards-defined public files (robots.txt, llms.txt, llms-full.txt, sitemap.xml) and one page, with a single request each. It never stores raw page content — only the audit results (scores, verdicts, fix lists).

Can I audit domains I don't own? The tool is designed for domains you own or are authorized to audit. It reads only public files with single requests, but ownership verification is recommended via the optional verifyToken parameter.

Limitations

• Audits one page per domain (the root or your auditPath) — not a full crawl.
• robots.txt matching uses the standard longest-prefix rule; unusual wildcard patterns may under-report.
• Intended for domains you own or are authorized to audit. Compliance tiering reflects publicly documented crawler behavior and can change.
• Availability of individual well-known files varies by host; a 404 on llms.txt is reported as absent, not an error.

Found an issue or have a feature request? Open an issue on the GitHub Issues tab — we respond promptly.