SiteVerdict — AI Search Readiness & GEO Audit

Find out whether ChatGPT, Claude, Perplexity, Copilot and Google AI Overviews can reach, read and cite your site, and get the exact fixes in one run.

Paste one URL or 10,000. Each site gets:

• a 0–100 AI search readiness score with five sub-scores
• a 35-agent AI crawler matrix checked against each vendor's documentation (11 Oct 2026)
• an honest CDN/WAF AI-blocking verdict
• a raw-vs-JavaScript gap measurement
• entity and structured-data checks and citation-readiness checks
• copy-paste fixes: robots.txt, a spec-valid llms.txt, Organization JSON-LD and FAQ schema

Each site also gets a shareable HTML report, and optionally a white-label PDF.

Tested on 25 public sites, our robots.txt verdicts matched an independent RFC 9309 parser 804 of 804 times. A simple "is the bot named?" check got 45 wrong. See BENCHMARK.md.

Who it's for

1. SEO and GEO agencies. Audit a client portfolio and send a white-label PDF with your logo, then put the sites on weekly monitoring with Slack alerts for when someone ships a robots.txt that blocks OAI-SearchBot.
2. In-house marketing and web teams. Find out why you're missing from ChatGPT or Perplexity answers. Common causes are a Cloudflare default, a JavaScript-only page, nosnippet, or a 2019 robots.txt that blocks PerplexityBot. Each comes with the fix your developer can paste.
3. Developers and AI agents. Call it from the API or the SiteVerdict MCP server and get typed JSON with a one-line summary, blockers and topFixes for each site.

Before / after (real output from 11 Oct 2026, shortened)

www.theverge.com: AI search readiness 77/100 (B). 7/11 AI search crawlers allowed by robots.txt.
  BLOCKER  Unblock AI search / assistant crawlers in robots.txt                  ≈ +4.8 pts
           ChatGPT-User      → "Disallow: /" (robots.txt line 97, own group)
           Claude-SearchBot  → "Disallow: /" (line 101)
           PerplexityBot     → "Disallow: /" (line 293)
  FIX      Open with a direct, self-contained answer                             ≈ +3.7 pts
  FIX      Server-side render the main content (79% of rendered text is in raw HTML)
  INFO     Training bots blocked: 5/8 (a policy choice, never penalised)

The generated "search-only" robots.txt fix tells you which existing groups to edit, and copies your private paths so a new named group doesn't accidentally open them:

# NOTE: your robots.txt already has a group for PerplexityBot (line 291); edit or remove it rather than adding a second one.
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
# (copied from your "User-agent: *" group, because a named group replaces it)
Disallow: /wp-admin/
Disallow: /login
Disallow: /account
…
Allow: /

# Model-training crawlers / opt-out tokens: blocked
User-agent: GPTBot
User-agent: Amazonbot
User-agent: MistralAI-Training
Disallow: /

After: turn on monitoring and the next scheduled run reports [info] PerplexityBot is now ALLOWED by robots.txt (was blocked) in the dataset and in Slack.

What makes it different

	SiteVerdict	Typical AI-SEO checker
AI agents checked	35, each with vendor, role (search / user-triggered / training / other), docs link and the exact robots line that matched	10–19
New 2026 agents (Google-Agent, Meta-WebIndexer, MistralAI-Index, Claude-User, Perplexity-User…)	✅	rarely
robots.txt correctness	RFC 9309, cross-checked 804/804 against an independent parser	substring checks are common
Blocking training bots	never penalised (it's your choice)	often counted as "bad"
CDN / WAF AI blocking	honest inference from what our real client receives, plus confirmation from your own access logs with IPs checked against OpenAI, Anthropic, Perplexity, Google, Bing, Apple, DuckDuckGo and Mistral published ranges	none, or spoofed bot user agents
Content-Signal, Cloudflare managed robots.txt, TDMRep	✅ read and explained	✗
JavaScript dependence	measured: share of rendered text in raw HTML, structured data and headings that exist only after JS, noindex injected by JS	"JS shell" guess
Entity quality	@id graph, publisher links, dangling references, knowledge-graph sameAs, phone (NAP) consistency, required properties	schema present yes/no
llms.txt	validator, link checker, and a generator built from your real pages	presence only
Monitoring and alerts	named store, severity-ranked diff, only-changed output, Slack and webhook	✗
Client reports	HTML and white-label PDF, request log and screenshot as evidence	HTML at best
AI fix (optional)	your own Anthropic or OpenAI key drafts llms.txt, meta description, an answer-first paragraph and FAQ JSON-LD. Every FAQ answer must quote a sentence on your page or it is dropped.	✗

Benchmark summary (25 public sites, 11 Oct 2026)

	Result
robots.txt decisions vs independent parser	804 / 804 agree (naive method: 45 wrong)
Sites blocking at least one AI search agent	6 / 25 (BBC, Guardian, Reuters, The Verge, NYT, Notion)
Sites serving a bot challenge to non-browser clients	4 / 25 (DataDome, Akamai ×2, Imperva). Not charged
Sites with llms.txt	8 / 25, all valid, 63/64 links OK
Speed	median 13 s per site with headless rendering; 25 sites in 131 s
False positives found and fixed during the benchmark	6, each now a regression test

Method, per-site table, competitor comparison and caveats are in BENCHMARK.md. We read competitors' documentation only and did not run their Actors.

Pricing (pay per event: you only pay for delivered results)

Event	Price	When it is charged
site-checked	$0.03	Site analysed (status ok or blocked-by-robots). Errors, timeouts and bot-challenge results are free.
page-audited	$0.004	Each extra page in multi-page mode
render-comparison	$0.01	Raw vs JavaScript-rendered comparison completed
pdf-report	$0.02	PDF generated
ai-fix-generated	$0.05	AI drafts produced and validated (LLM usage is billed by your own provider)
site-rechecked-unchanged	$0.01	Monitoring re-check with "only changed" on and nothing changed
change-alert-sent	$0.01	Per site included in a delivered Slack or webhook alert
access-log-analysed	$0.10	Your log file parsed and verified against official IP ranges

Examples:

• 1 site: $0.03
• 100 homepages: $3.00
• 100 homepages with JS measurement and PDF: $6.00
• One client site, 25 pages, JS measurement, PDF and AI fix: ≈ $0.21
• Weekly monitoring of 50 sites where nothing changed: $0.50 a week

Input (most-used fields)

{
  "startUrls": [{ "url": "https://www.example.com" }],
  "maxPagesPerSite": 10,
  "compareRendered": true,
  "generatePdfReport": true,
  "reportPreparedBy": "Your Agency Ltd",
  "monitoringMode": true,
  "onlyChanged": true,
  "slackWebhookUrl": "https://hooks.slack.com/services/…",
  "aiFixProvider": "anthropic",
  "aiApiKey": "sk-ant-…"
}

Other options:

• recommendedPolicy (search-only / allow-all / block-all): the policy the generated robots.txt snippet implements
• captureScreenshots
• checkLlmsTxtLinks
• accessLogUrl / accessLogText: your own logs
• alertMinSeverity, scoreChangeThreshold
• webhookUrl
• aiModel
• reportLogoUrl, reportAccentColor
• politeness and limit settings

Secrets (webhooks, API key, logs) are encrypted inputs and are never logged or output.

Output

One dataset row per site, with these views:

• Overview
• Prioritised fixes
• AI crawler matrix
• Pages
• Monitoring changes

Each view exports as JSON, CSV or Excel.

Fields in each row:

• url, inputUrl: the site you entered, without any user name or password. A query string or fragment is replaced by ?[redacted] / #[redacted] and paths longer than 120 characters are shortened, so tokens or session ids in a URL do not end up in the output. A plain URL such as https://www.example.com/ is unchanged.
• urlHash: the first 16 hex characters of the SHA-256 hash of the full URL that was checked, to match rows to your input when url was shortened.
• score, grade, subScores (access, extractability, structure, trust, citability)
• summary: a one-line verdict for agents
• botMatrix[]: agent, vendor, role, allowed, matched rule, confidence, Content-Signal, docs link
• cdnAiBlocking: level (confirmed / likely / possible / unlikely), evidence, how to confirm
• aiPreferences: Content-Signal, Content-Usage, Cloudflare managed robots, TDMRep
• renderComparison: raw vs rendered words, text coverage, JS-only schema and headings, JS-made changes
• entity, citability, checks[], topFixes[] (blocker flag, estimated points, snippet, where, pages affected)
• siteAudit and pages[] (multi-page mode)
• accessLogs: per-agent requests, IP-verified count, impostors, status codes, refusal share
• monitoring: previous score, changes with severity
• aiFix (optional): llms.txt, meta description, answer-first paragraph, FAQ JSON-LD, dropped items
• snippets: robots.txt for 3 policies, llms.txt, Organization JSON-LD
• evidence: timestamped request log and screenshot link
• reportUrl, pdfUrl, llmsTxtUrl

Key-value store:

• an HTML report per site, plus PDF, screenshot and llms.txt files
• a SUMMARY page
• OUTPUT: an agent-ready JSON summary of the run, including privateNetworkProtection (start URLs refused, browser requests blocked)

How the score works

Every point is a visible check with a reason.

Category weights:

Category	Weight
Access	30%
Extractability	20%
Structure	20%
Trust	15%
Citability	15%

Eligibility gates cap the score when AI systems can't use the page:

Condition	Score capped at
Page unreachable	20
noindex	35
Most AI search agents blocked, or CDN blocking confirmed from your logs	45

Training-bot opt-outs, Content-Signal preferences and unconfirmed CDN signals are reported but not scored. Citability checks (answer-first opening, specific facts, lists and tables, cited sources) are marked heuristic.

FAQ

Does blocking GPTBot hurt my ChatGPT visibility? No. ChatGPT search uses OAI-SearchBot, and live visits use ChatGPT-User. GPTBot is training only, and the same split applies to Claude and Google (Google-Extended doesn't affect AI Overviews). We show each separately and never penalise training opt-outs.

Can you tell whether Cloudflare is blocking AI bots? From the outside, only partly. We report possible when Cloudflare is in front of the site, because it blocks known AI crawlers by default on many zones. We report likely when our own honest client gets challenged. To get a confirmed answer, give us your access logs: we check every AI-bot request's source IP against the vendor's published ranges and show the status codes your server or CDN returned. We never fake other bots' user agents, because CDNs verify bots by IP and spoofed tests are unreliable and unfair to site owners.

Do I need llms.txt? It's an emerging proposal, and no major AI vendor has confirmed using it for search or citations, so it carries little weight in the score. If you want one, we generate a spec-valid file from your real pages and check the links in any existing one.

Will this guarantee citations in AI answers? No tool can. We measure the technical and content signals that make a page reachable, readable and quotable, and we tell you which checks are heuristic.

Is the AI fix safe? It is off by default and runs on your own key, which is stored encrypted and never logged. Page text is treated as untrusted. Links in the llms.txt come only from pages we actually fetched. FAQ answers must quote a real sentence from your page or they are dropped. Review every draft before publishing.

Can I monitor client sites? Yes. Turn on monitoringMode and schedule the Actor. Each run compares the site against the previous snapshot in a named store. You can output only the sites that changed and get Slack or webhook alerts for changes such as "AI search bot newly blocked", "noindex appeared", "bot challenge started", "structured data removed" or a score drop.

Limits

• Pages and JS rendering: one URL per input row, up to 50 pages per site in multi-page mode. JS rendering, screenshots and PDF use headless Chromium; allocate 2 GB of memory or more, and we cap concurrency at 3.
• Bot protection: sites behind it may challenge our datacenter IP. We report that honestly, don't charge for it, and never use proxies or unblockers to get around it.
• Access logs: the access-log check applies to the first site in the run (your own site), with a 50 MB limit.
• Accuracy: TTFB is a single sample from our server. Citability checks are English-centric heuristics.
• Private networks: only public websites are checked. A start URL whose host is private, loopback, link-local, cloud-metadata or an internal name, or whose DNS answer is a private address, is not contacted: its row has status error with "refused: private-network protection; not charged" and it is not charged. Page, robots.txt, sitemap, llms.txt and access-log requests connect only to the address that was checked and re-check every redirect; JavaScript rendering, screenshots and PDF logos go through a local checking proxy that applies the same rule. webhookUrl, slackWebhookUrl, reportLogoUrl and accessLogUrl must be public https:// addresses, otherwise they are ignored with a warning.

Compliance and sources

• How we crawl:

  • We identify honestly as SiteVerdictBot and respect robots.txt for our own requests by default.
  • We send polite, sequential requests to each site.
  • We read only public pages.
  • We never bypass logins, CAPTCHAs, paywalls or bot protection, and never impersonate other crawlers.

• Data protection:

  • Personal data is not collected. Contact details shown are only those the site itself publishes as its business contacts.
  • Access-log IP addresses are processed in memory and never output.

• Sources: the crawler list was checked on 11 Oct 2026 against vendor documentation:

  • OpenAI: developers.openai.com/api/docs/bots
  • Anthropic: support.claude.com article 8896518
  • Perplexity: docs.perplexity.ai/guides/bots
  • Google: common crawlers and user-triggered fetchers
  • Apple: support.apple.com/119829
  • Meta: developers.facebook.com web crawlers
  • Amazon: developer.amazon.com/amazonbot
  • Mistral: docs.mistral.ai/robots
  • DuckDuckGo, You.com and Common Crawl: their own documentation pages

  Agents with no official documentation (Bytespider, cohere-ai) are labelled unofficial.

• Standards: RFC 9309 (robots.txt), llmstxt.org, Cloudflare Content Signals Policy, W3C TDMRep, schema.org.

• Disclaimer: this is not legal advice. Whether to allow AI training is a business and legal decision for the site owner.

• Where results are stored: the default dataset and key-value store of the run (reports, robots.txt suggestions, optional PDF), inside your own Apify account. These default storages are unnamed. How long Apify keeps them is set by Apify and your plan, not by SiteVerdict (Apify data retention); you can delete them sooner in Apify Console.

• Monitoring (off by default): when you turn it on, the previous result for each site is kept in a named key-value store in your account (siteverdict-ai-search-monitor, or the name you set). Apify keeps named stores until you delete them.

• Access logs (optional): a log you supply by URL or text is read during the run only; IP addresses are not output (see above). Sample request paths in the output are reported without query strings, and e-mail addresses, IP addresses, UUIDs and token-like strings inside a path are masked.

• URLs in results and logs: the dataset, reports, OUTPUT, monitoring snapshots, alerts and the run log show URLs without query strings, fragments or credentials (see url above); the request evidence and final URLs are shown as scheme, host and path only. The checker itself still follows the full URL you entered.

• Third parties you choose: the optional AI fix sends the audited page's public text, title, headings and the site's page titles to Anthropic or OpenAI under your own API key and their terms; alerts go only to the Slack/webhook URLs you enter.

• What SiteVerdict keeps: nothing. The Actor runs in your Apify account and sends no data to SiteVerdict.