AI Search Readiness Audit: llms.txt, AI Crawlers & GEO Score

Find out in seconds whether ChatGPT, Claude, Perplexity, Google AI Overviews, Gemini and Microsoft Copilot can actually reach, read and cite a website. Paste a list of domains or page URLs and get one row per URL with a 0-100 AI search readiness score, a letter grade, a pass/fail list of 27 checks and the top fixes in plain language.

This is a technical GEO (generative engine optimization) / AEO (answer engine optimization) audit that runs in bulk. It reads only the site's own public files (robots.txt, llms.txt, sitemap, the page HTML) with a plain HTTP client, so it is fast and cheap: $6 per 1,000 URLs, and URLs that do not answer at all are free.

What does it check?

Crawler access (robots.txt and firewall)

• The robots.txt verdict for 30 AI crawlers, split by purpose: AI search bots that fetch pages to answer and cite (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot, DuckAssistBot, Amazonbot, YouBot...) and training bots that collect data for models (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, cohere-ai...).
• Whether each bot is named in robots.txt or only covered by a wildcard rule.
• Cloudflare Content-Signal lines (search=yes, ai-train=no) when present.
• Whether the page answers a plain HTTP client at all, or returns a block, a challenge page or an error (AI fetchers usually see the same thing).
• noindex, nosnippet, max-snippet:0, noai and noimageai in meta robots and the X-Robots-Tag header.

AI files

• llms.txt: present, well-formed (H1 title + Markdown links), number of sections and links.
• llms-full.txt: present and its size.
• XML sitemap (from robots.txt or the usual locations).

Content and structure

• Content visible without JavaScript: the number of words in the server-rendered HTML, and whether the page is an empty JavaScript app shell. Most AI crawlers do not run JavaScript, so a client-side-only page is invisible to them.
• Title and meta description length, canonical URL, lang.
• One H1, subheadings, lists and tables (the parts AI answers quote most easily).
• schema.org JSON-LD types (Organization, WebSite, Article, Product, FAQPage, LocalBusiness...), FAQ content.
• Author / publisher, published and modified dates, Open Graph tags, image alt-text coverage.
• Response time and HTML size.

Who is it for?

• SEO and GEO agencies auditing a client list or a prospect list before a pitch.
• Marketing teams checking that a robots.txt change or a CDN bot setting did not lock out ChatGPT search or Perplexity.
• Publishers deciding which AI bots to block for training while staying visible in AI search.
• Developers verifying that a JavaScript site renders its content on the server.
• Analysts measuring how many sites in a sector block AI crawlers.

Input

Field	What it does
urls	domains or page URLs, one per line (stripe.com, https://example.com/pricing)
includeCheckDetails	add the full checks list and per-crawler verdicts aiCrawlers (default on)
maxConcurrency	URLs audited in parallel, default 8
maxResults	cap on the number of URLs

{
  "urls": ["stripe.com", "https://www.nytimes.com", "https://vercel.com/pricing"],
  "includeCheckDetails": true
}

robots.txt, llms.txt and the sitemap are read once per site, so auditing 50 pages of one site costs no extra requests for those files.

Output

One row per URL (shortened):

{
  "url": "https://www.nytimes.com",
  "domain": "nytimes.com",
  "score": 47,
  "grade": "F",
  "aiSearchBotsAllowed": 2,
  "aiSearchBotsChecked": 6,
  "blockedSearchBots": ["OAI-SearchBot", "ChatGPT-User", "Claude-SearchBot", "PerplexityBot"],
  "blockedTrainingBots": ["GPTBot", "ClaudeBot", "Google-Extended", "Applebot-Extended"],
  "llmsTxt": false,
  "sitemap": "https://www.nytimes.com/sitemaps/new/news.xml.gz",
  "httpStatus": 403,
  "pageContentChecked": false,
  "topFixes": [
    "The page answers plain HTTP clients with a block, a challenge page or an error...",
    "robots.txt blocks OAI-SearchBot, so ChatGPT search cannot read or cite your pages..."
  ]
}

A healthy site looks like this:

{
  "domain": "vercel.com", "score": 98, "grade": "A",
  "aiSearchBotsAllowed": 6, "llmsTxt": true, "llmsTxtValid": true,
  "contentVisibleWithoutJs": true, "wordsWithoutJs": 546,
  "schemaTypes": ["Organization", "PostalAddress", "ContactPoint"],
  "topFixes": ["Your meta description is 45 characters; make it 50-170 characters..."]
}

Other fields: robotsTxtFound, contentSignals, llmsTxtUrl, llmsTxtSections, llmsTxtLinks, llmsFullTxt, finalUrl, responseMs, htmlKb, title, metaDescription, canonical, lang, robotsDirectives, noindex, nosnippet, noai, wordsWithoutJs, javascriptAppShell, h1, h1Count, h2Count, listsCount, tablesCount, hasOrganizationSchema, hasFaqSchema, author, datePublished, dateModified, ogImage, imagesWithAltPct, checksPassed, checksTotal, fixes (each with priority and points) and, with details on, checks and aiCrawlers.

When the page itself is blocked, only the crawler-access and AI-file checks are scored (pageContentChecked: false), so a firewall does not count as dozens of content failures.

How is the score calculated?

Each check has a weight: crawler access weighs most (Googlebot 10, a readable page 10, content without JavaScript 10, OAI-SearchBot 8, Bingbot 8, no noindex 8, PerplexityBot 6), then structured data, AI files, metadata and structure. The score is the share of weighted points passed; grades are A 90+, B 80+, C 70+, D 55+, F below. Blocking training bots does not lower the score: that is a legitimate choice, and it is reported separately in blockedTrainingBots.

How much does it cost?

You pay $0.006 per URL audited ($6 per 1,000) plus a one-time $0.001 actor start. URLs that give no response are pushed for free. Platform usage is small: a 256 MB run.

• 25 client homepages: about $0.15.
• 500 prospect domains for an agency pitch: about $3.00.
• 5,000 domains for a sector study of AI-crawler blocking: about $30.

Higher Apify plans get automatic volume discounts.

FAQ

Is this legal? Yes. The actor reads publicly served files (robots.txt, llms.txt, sitemaps, page HTML) the same way a browser does, respects that the files are public by design, and never logs in. It identifies itself with its own user agent and never pretends to be an AI crawler.

Does it test whether my firewall blocks GPTBot? It reports robots.txt rules and whether the page answers a plain client. It does not send fake crawler user agents. If your CDN blocks verified AI bots by IP, check your bot-management settings as well.

Does it run JavaScript? No, on purpose. The point is to see the page the way most AI crawlers see it.

Can I audit many pages of one site? Yes: put the page URLs in the list. Site files are read once and reused.

What is llms.txt? A proposed standard (llmstxt.org): a Markdown file at the site root with a title, a short summary and links to the pages that matter, so AI tools can find the right content quickly.

Integrations

Export results as JSON, CSV or Excel, schedule weekly re-audits, call the actor from the Apify API, or connect it to Make, Zapier, n8n, Google Sheets and webhooks to get alerted when a site's score drops.

Changelog

• 0.1 (Nov 2026): first release with 30 AI crawlers, llms.txt / llms-full.txt, Content-Signal, JS-free content check, schema and 27 weighted checks.