AI Search Readiness Audit - llms.txt, AI Bots & Schema

Can ChatGPT, Perplexity and Claude actually reach, read and cite your site? This Actor scores your site 0-100 for AI-search readiness from the outside, tells you the exact fixes in priority order, and generates the files that close the biggest gaps: a ready llms.txt, a robots.txt snippet for AI search crawlers, and recommended JSON-LD for pages that have none. (Formerly "llms.txt Generator & Checker"; it also absorbs the JSON-LD generate/validate modes of the Structured Data Generator.)

What is measured

Area	Points	What is checked
AI crawler access	25	robots.txt rules for 12 AI bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent). Plus a live test: does your server answer those user agents with 200, or does a WAF return 403/429?
llms.txt	20	Exists, title and summary line, links, coverage of your sitemap URLs, stale links
Structured data	20	JSON-LD coverage across audited pages, validity of existing markup against Google's required/recommended properties, unparseable blocks
Extractable text	15	Words present in the raw HTML (most AI crawlers do not run JavaScript), <main>/<h1>, lang, JavaScript app-shell detection
Sitemap	10	sitemap.xml found, lastmod present, referenced in robots.txt
Canonical and meta	10	Self-referencing canonical, title + description, accidental noindex

Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot...) are listed but never penalised: blocking them is a legitimate business choice. Only blocking search/answer crawlers costs points, because those fetch pages to answer questions and cite sources.

What you get

1. A summary row: score, grade, per-area breakdown, top fixes, bot-by-bot allowed/blocked table, existing llms.txt audit.
2. Fix rows, ranked: area, severity, points at stake, the issue, and the concrete fix.
3. Page rows: words in raw HTML, canonical, noindex, JSON-LD types found, existing-markup errors and warnings, and a recommended JSON-LD block where the page has none.
4. Files in the key-value store: llms.txt, optional llms-full.txt, robots-ai-snippet.txt (only if search bots are blocked), recommended-jsonld.json, FIX_LIST.md.

Sample output

{
  "type": "summary", "site": "https://docs.apify.com", "mode": "full",
  "score": 70, "grade": "C",
  "headline": "AI search readiness 70/100 (C). Biggest gap: 20 of 20 audited pages have no JSON-LD structured data.",
  "areas": {
    "llmsTxt": { "score": 16, "max": 20, "exists": true, "coveragePercent": 11 },
    "aiCrawlerAccess": { "score": 25, "max": 25, "retrievalBotsAllowed": "6/6" },
    "structuredData": { "score": 0, "max": 20, "pagesWithJsonLd": "0/20" },
    "sitemap": { "score": 8, "max": 10, "urls": 4053 },
    "extractableText": { "score": 14, "max": 15, "pagesWithEnoughStaticText": "18/20" },
    "canonicalAndMeta": { "score": 7, "max": 10 }
  }
}

{ "type": "fix", "priority": 1, "severity": "high", "pointsAtStake": 12, "area": "Structured data",
  "issue": "20 of 20 audited pages have no JSON-LD structured data.",
  "fix": "Add the recommended JSON-LD (key-value store record \"recommended-jsonld.json\" ...) to those pages." }

Input

Field	Default	What it does
siteUrl	https://docs.apify.com	Site (or a single page; the typed URL is always audited)
mode	full	full score + fixes + files, llms-txt only llms.txt, json-ld only structured data
maxPages	20	Pages sampled for page-level checks. Site-level checks cover the whole site
useSitemap, includePatterns, excludePatterns	true, empty	Page discovery filters
testBotAccess	true	Request the homepage as 5 AI user agents to detect WAF blocks
generateFullText, maxFullTextChars	false, 500000	Also build llms-full.txt
siteName, summary	empty	Override the generated llms.txt title and summary line

Run with {} to see the report on the default docs site.

Pricing

Pay per event, no subscription:

Event	Price	When
Site analyzed	$0.002	per run
Page processed	$0.002	per page fetched
Audit run	$0.002	per scored audit (full mode)
File generated	$0.01	per generated file (llms.txt, llms-full.txt, recommended-jsonld.json)

A default 20-page audit costs about $0.07. Failed pages are not charged.

Use cases

• Marketing and SEO teams: a checklist for "are we visible in ChatGPT and Perplexity answers?" with the fixes already written.
• Developers shipping docs: generate llms.txt / llms-full.txt and check JSON-LD in CI.
• Agencies: audit a prospect's site in one run for a sales call.
• Before/after: run again after the fixes and compare the score.

Honest limits

• This measures readiness, not results. It cannot tell you whether any AI product currently cites your site.
• llms.txt is a proposed convention. Major AI providers have not confirmed they use it, so it is weighted at 20% of the score. Cheap to publish, but do not expect it to move rankings on its own.
• The bot test changes only the User-Agent header. Real crawlers come from published IP ranges, so a WAF may still treat them differently. Confirm in your CDN logs.
• Page-level scores are measured on the maxPages sample, not the entire site.
• It reads raw HTML and does not run JavaScript, which is the point of the extractable-text check.
• JSON-LD recommendations are drafts inferred from page content (type guesses below 40% confidence are not offered). Review them before publishing, and validate with Google's Rich Results Test.

FAQ

Should I allow GPTBot and ClaudeBot? That is your call. They collect training data. The score only rewards allowing the search and answer crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and the user-fetch bots).

Which JSON-LD types can it generate? Product, Article, LocalBusiness, Organization, FAQPage, HowTo, Recipe, Event, BreadcrumbList, JobPosting and VideoObject, validated against Google's required/recommended properties (ruleset 2025-06).

Does it work for JavaScript-only sites? It will tell you they are a problem: pages with little text in the raw HTML are flagged as app shells.

Integrations

Works with Make, Zapier, n8n, the Apify API and scheduler, and Apify's MCP server so an AI agent can run the audit. Schedule it monthly and watch the score.

If it helped, a review on the Store page helps others find it. Questions or a site it handles badly? Open an issue on the Actor page.