AI Search Readiness Audit

Find out in seconds whether ChatGPT, Claude, Perplexity, Google AI Overviews and Bing Copilot can reach, read and cite a website, with a 0-100 score and a prioritized fix list for every domain.

What does AI Search Readiness Audit do?

AI Search Readiness Audit checks the technical side of GEO (generative engine optimization): whether a website can show up as a cited source in AI answers. For each domain or URL it parses robots.txt the way crawlers do and gives a verdict for 18 AI and search bots. It also validates llms.txt and checks how much content is visible without JavaScript. Then it inspects indexing and snippet directives, structured data and sitemap freshness. Everything is rolled into one row per website: an AI readiness score, a letter grade, a transparent score breakdown and plain-English fixes that you can hand to a developer or a client.

Why use it

• Answers the question clients actually ask. "Can ChatGPT and Perplexity see us?" gets a direct answer per crawler, with the exact robots.txt line responsible.
• Separates training from search. Blocking AI training (GPTBot, Google-Extended, CCBot...) is a legitimate business choice and is reported neutrally. Only blocks that stop you from being found and cited cost points.
• Real robots.txt logic, not keyword matching. It handles user-agent groups, merged groups, * fallback, longest-match precedence, Allow vs Disallow ties, * and $ wildcards, Apple's documented Googlebot fallback, and Cloudflare-style Content-Signal lines.
• Catches the silent killers. It finds client-rendered single-page apps that AI crawlers see as empty pages, noindex/nosnippet in meta tags or X-Robots-Tag headers, robots.txt files that return 5xx (crawlers then treat the whole site as blocked), and firewalls that refuse automated requests.
• One row per domain, ready for reporting. It is built for agency audits, prospect lists and monitoring portfolios of sites. It replaces separate robots.txt, sitemap and structured-data checkers.
• Prioritized fixes with copy-paste snippets, for example the exact User-agent group to add so ChatGPT search can cite you while GPTBot stays blocked.

GEO and AI search readiness, explained for marketers

AI assistants answer questions by searching the web, reading pages and citing sources. To be cited, a page has to pass three gates:

1. Access: the assistant's crawler must be allowed by robots.txt and not blocked by a firewall.
2. Readability: the crawler must find real text in the HTML. Published crawler analyses show that most AI crawlers, including those from OpenAI, Anthropic and Perplexity, fetch HTML without running JavaScript, so content injected client-side is invisible to them. Googlebot does render JavaScript.
3. Eligibility and understanding: the page must be indexable and allowed to be quoted (no noindex, nosnippet or tiny max-snippet). Structured data, a sitemap with honest lastmod dates and, optionally, an llms.txt file help machines understand and prioritise it. Google states that a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews and AI Mode.

Training vs. search vs. user-triggered crawlers

AI companies run different bots for different jobs, and robots.txt lets you treat them independently:

Purpose	Bots checked	What blocking does	Affects score?
Training	GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot	Opts your content out of future model training. Google-Extended and Applebot-Extended are control tokens and do not remove you from Google Search or Siri.	No, reported as policy
Search / answer	OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot, DuckAssistBot	Removes you from ChatGPT search, Claude search, Perplexity, Google Search and AI Overviews, Bing/Copilot, Siri/Spotlight and DuckAssist results and citations.	Yes, weighted by reach
User-triggered	ChatGPT-User, Claude-User, Perplexity-User	Stops the assistant from opening your page when a user asks about it or pastes the link. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores" it, so treat these rules as a statement of policy.	Yes, small weight

A common, sensible setup is to block training crawlers and allow search and user-triggered crawlers. This Actor grades that setup as fully ready.

What data you get

Field	Description
aiReadinessScore, grade	0-100 score and A-F grade (A ≥ 90, B ≥ 80, C ≥ 70, D ≥ 60)
summary	One-sentence verdict: policy, llms.txt, number of high-priority fixes
scoreBreakdown	Points per category with every individual check listed (see scoring below)
scoreCaps, scoreBeforeCaps	Critical blockers that cap the score, e.g. a client-only SPA or every search crawler blocked
fixes[]	Prioritized {priority, category, issue, recommendation, pointsLost} items in plain English
blockedSearchBots, blockedTrainingBots, blockedUserFetchers	Bots blocked site-wide, by purpose
aiCrawlers[]	Per bot: operator, purpose, matched robots.txt group, allowed / partially-blocked / blocked for / and for the input URL path, blocking rule and line number, crawl-delay
aiPolicy	Neutral description of the AI policy, including Content-Signal and noai notes
robotsTxt	Status, groups, user agents, AI bots with their own rules, sitemap declarations, crawl-delays, Content-Signal
llmsTxt, llmsFullTxt	Presence, H1 title, summary, sections, link count, broken-looking links, size
pages.homepage, pages.inputPage	HTTP status, redirect chain, response time, title/description length, lang, canonical, meta robots + X-Robots-Tag (noindex, nofollow, nosnippet, max-snippet, noai, noimageai), Open Graph, JSON-LD types and errors, words visible without JavaScript, SPA signals
sitemap	Where it was found, index or urlset, child sitemaps sampled, URL count (sampled and estimated), lastmod coverage, share older than 180 days, newest lastmod, gzip support
partialAudit, status, error	Whether some categories could not be evaluated; failure reason for uncharged items

How the score works

Category	Points	What is checked
Crawler access	30	robots.txt verdicts for search/answer bots (Googlebot 7, Bingbot 5, OAI-SearchBot 4, Claude-SearchBot 3, PerplexityBot 3, Applebot 2, DuckAssistBot 1) and user-triggered fetchers (5 total). Training bots: 0.
Content accessibility	25	HTTP 200, words visible without JavaScript, response time, redirects, title, meta description, lang
Structured data	15	JSON-LD present and valid, @type on every entity, Organization and WebSite, page-type schema, Open Graph
Discoverability	15	Sitemap found and declared in robots.txt, lastmod coverage and freshness, llms.txt, canonical
Snippet controls	15	No noindex, nosnippet, restrictive max-snippet or page-level nofollow; Content-Signal does not opt out of AI answers

Critical blockers cap the total: an unreachable robots.txt or every search crawler blocked (max 40), no page returning 200 (max 50), Googlebot or Bingbot blocked or a noindexed page (max 60), a client-rendered SPA (max 65). If a site refuses our page requests (for example HTTP 403 from bot protection), the page-level categories are marked "not evaluated" and the score is rescaled to the categories that could be checked.

How to use

1. Open the Actor and paste your domains or URLs into Websites or URLs to audit. Bare domains (example.com) and deep URLs (https://example.com/pricing) both work.
2. Keep Check llms.txt on, and adjust Max child sitemaps to sample if you want more precise sitemap statistics for large sites.
3. Click Start. Each website typically takes 2-8 seconds.
4. Open the Overview tab for scores and top fixes, Fixes for the full prioritized list, or AI crawlers for the per-bot robots.txt verdicts. Export as CSV, Excel or JSON.

{
  "urls": [
    "example.com",
    "https://www.example.org/blog/how-we-work",
    "https://shop.example.net"
  ],
  "checkLlmsTxt": true,
  "maxSitemapsToSample": 3,
  "maxConcurrency": 5
}

Output example

Trimmed result from a test run in October 2026:

{
  "input": "https://www.theguardian.com",
  "domain": "theguardian.com",
  "finalUrl": "https://www.theguardian.com/international",
  "status": "ok",
  "aiReadinessScore": 74,
  "grade": "C",
  "summary": "74/100 (C). Opts out of 6 of 8 AI training crawlers/tokens (ClaudeBot, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot). Blocks search/answer crawlers: Claude-SearchBot, PerplexityBot, DuckAssistBot. Disallows user-triggered fetchers: Claude-User. No llms.txt. 1 high-priority fix.",
  "blockedSearchBots": ["Claude-SearchBot", "PerplexityBot", "DuckAssistBot"],
  "blockedTrainingBots": ["ClaudeBot", "Applebot-Extended", "CCBot", "Bytespider", "Meta-ExternalAgent", "Amazonbot"],
  "blockedUserFetchers": ["Claude-User"],
  "hasLlmsTxt": false,
  "serverRenderedWords": 3301,
  "looksLikeClientOnlySpa": false,
  "sitemapUrlCount": 504,
  "scoreBreakdown": {
    "crawlerAccess": { "score": 22, "max": 30 },
    "contentAccessibility": { "score": 25, "max": 25 },
    "structuredData": { "score": 0, "max": 15 },
    "discoverability": { "score": 12, "max": 15 },
    "snippetControls": { "score": 15, "max": 15 }
  },
  "fixes": [
    {
      "priority": "high",
      "category": "crawlerAccess",
      "issue": "Claude-SearchBot, Claude-User, PerplexityBot and DuckAssistBot are blocked by `Disallow: /` under the group that names them (robots.txt line 102). Claude web search, Perplexity and DuckAssist cannot crawl or cite your pages; Claude cannot open your pages when a user asks about them.",
      "recommendation": "Remove these user agents from the group that names them (or give them their own group with `Allow: /`) if you want to appear in Claude web search, Claude, Perplexity and DuckAssist. Blocking the matching training crawler is a separate, independent setting.",
      "pointsLost": 8.5
    },
    {
      "priority": "medium",
      "category": "structuredData",
      "issue": "No structured data (JSON-LD or microdata) found.",
      "recommendation": "Add JSON-LD to the homepage: an Organization (name, url, logo, sameAs links) and a WebSite entity. On content pages add the matching type (Article, Product, FAQPage, LocalBusiness...). It gives AI systems unambiguous facts about who you are and what each page is.",
      "pointsLost": 5
    }
  ],
  "aiCrawlers": [
    {
      "bot": "OAI-SearchBot",
      "operator": "OpenAI",
      "purpose": "search",
      "matchedGroup": "*",
      "rootAccess": "partially-blocked",
      "typicalPageAllowed": true,
      "blockingRule": null,
      "note": "Allowed under `User-agent: *`, except 48 disallowed path pattern(s) such as `/sendarticle/`, `/Users/`, `/users/`."
    },
    {
      "bot": "Claude-SearchBot",
      "operator": "Anthropic",
      "purpose": "search",
      "matchedGroup": "Claude-SearchBot",
      "rootAccess": "blocked",
      "blockingRule": "Disallow: /",
      "blockingLine": 102,
      "note": "Blocked site-wide by `Disallow: /` under `User-agent: Claude-SearchBot`."
    }
  ],
  "sitemap": { "found": true, "type": "urlset", "urlCountSampled": 504, "lastmodCoveragePct": 100, "newestLastmod": "2026-10-16T08:34:01.000Z" },
  "checkedAt": "2026-10-16T08:35:04.588Z"
}

partially-blocked means the homepage is allowed but some paths are disallowed (normal housekeeping such as /admin/) or, the other way round, the homepage is blocked but some paths are allowlisted. typicalPageAllowed: false flags the second case: only a few allowlisted paths are open, and that costs points.

Pricing

This Actor uses pay-per-event pricing: $0.01 per audited website or URL.

• 100 websites: $1
• 1,000 websites: $10
• 10,000 websites: $100

Not charged: invalid inputs, domains that do not resolve or cannot be reached, sites that refuse every request (bot protection, status blocked), and duplicate inputs. Each unique URL is one audit, so example.com and example.com/pricing are two audits. You can set a maximum cost per run, and the Actor stops cleanly when it is reached.

Integrations and scheduling

• Schedules: run weekly or monthly on a client portfolio to catch robots.txt changes, for example a CMS update or a CDN "block AI bots" toggle that silently blocks ChatGPT search.
• Webhooks and integrations: send results to Slack, Google Sheets, Make, Zapier or n8n when a run finishes, or alert when a grade drops.
• API: start runs and fetch the dataset (?view=overview for the compact table) from any language via the Apify API or client libraries.
• AI agents and MCP: the Actor works as a tool through the Apify MCP server, so an AI agent can audit a prospect's site and draft recommendations from the fixes list.

FAQ and limitations

Does blocking GPTBot lower my score? No. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot) are reported under blockedTrainingBots as a policy choice. Only search/answer crawlers and user-triggered fetchers count.

Is llms.txt required? No. It is an emerging, proposed convention (llmstxt.org) and AI providers' support for it is limited and largely unconfirmed. It is worth 3 of 100 points, and a missing file is always a low-priority fix.

Why does my site get a 403 / "partial audit"? Our requests look like a normal browser. If a firewall or bot-protection layer refuses them, AI crawlers may be refused too unless they are allowlisted. Page-level checks are then skipped and the score is rescaled. If every request is refused, nothing is charged.

Why does a JavaScript site score low? The Actor reads the HTML as delivered, without running JavaScript, which is how most AI crawlers read pages. If the HTML is an empty app shell, those crawlers have nothing to read or quote.

Limitations:

• The Actor reads robots.txt and HTML from the audit server. It cannot see IP-based firewall rules, CDN bot settings, or a robots.txt served differently to verified crawlers.
• Some sites serve different content to bots and browsers (dynamic rendering). The SPA check reflects what a browser-like request receives.
• Sitemap statistics come from up to 20 sampled child sitemaps (3 by default), each read up to 10 MB. estimatedTotalUrls is an extrapolation.
• Only the homepage and the URL you enter are inspected; it is not a full-site crawl.
• Response times are measured from the audit server (plus proxy, if used) and are indicative.

Responsible use: The Actor only requests public files that crawlers are expected to read (robots.txt, sitemaps, llms.txt) and at most two public pages per input. That is typically 6-12 lightweight requests per website, with at most 3 in parallel per site. It collects no personal data and does not bypass logins, paywalls or bot protection. Audit websites you own, manage, or have a legitimate reason to evaluate, and follow the websites' terms.