AI Search Visibility Audit

Audits any website for Generative Engine Optimisation (GEO) — how likely your pages are to be cited, referenced, or recommended by ChatGPT, Perplexity, Gemini, Claude, Copilot, Grok, and Google AI Overviews / AI Mode.

This is not a traditional SEO audit. It checks whether AI systems can crawl, understand, extract from, and trust your content enough to surface it in their answers.

Why it matters

AI-powered search is replacing link-based results. Google AI Overviews and AI Mode, ChatGPT web search, Perplexity, Copilot and Grok now answer questions directly — and they choose which sources to cite. If your site isn't structured for AI extraction, you're invisible to a growing share of search traffic.

This actor gives you a concrete, actionable score so you know exactly what to fix.

What you get

• GEO Score (0–100) per page and site-wide, graded A+ to F
• 6 category scores with weighted rollup: AI Crawlability (30%), Entity & Brand Clarity (20%), Content Structure (25%), Schema & Structured Data (10%), Authority & Trust (10%), AI-Specific Optimisation (5%)
• Live AI crawler probe — real requests using real AI user-agents, to catch CDN-level blocking that robots.txt checks cannot see
• Eligibility gate — a site AI systems cannot reach cannot score well, no matter how good its markup is
• Prioritised issues — every issue includes severity, estimated impact, the exact points deducted, and a plain-English fix hint
• Quick wins — the top 5 highest-impact fixes you can make today
• Competitor comparison — optional side-by-side scoring against up to 3 competitor URLs
• Deterministic output — no LLM calls for data extraction; same input produces the same result

How AI crawler access is measured

Access is the precondition for everything else, so it is assessed two ways and scored by tier.

The three tiers

Not all AI crawlers matter equally, and the actor no longer pretends they do.

Tier	Agents	Penalty per blocked agent	Why
Retrieval	OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, DuckAssistBot, Amazonbot	−15, critical	These build the index AI answers are drawn from. Blocking them removes you from AI answers.
User-triggered	ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, Meta-ExternalFetcher	−15, critical	These fetch your page when a user asks an assistant about you. Blocking them kills live citation.
Training	GPTBot, ClaudeBot, anthropic-ai, Google-Extended, CCBot, Bytespider, Meta-ExternalAgent, Applebot-Extended	0 — reported, not penalised	Blocking these is a legitimate rights decision that costs little present-day visibility.

One caveat on the training tier: CCBot feeds Common Crawl, the dataset behind most open models (Llama, Mistral and similar). Blocking it reduces your long-term presence in those models. The actor says so rather than scoring it.

The live probe

robots.txt only records what a site says. Cloudflare and similar CDNs block AI crawlers by default on many plans, regardless of robots.txt. So once per domain the actor issues real requests using genuine AI crawler user-agent strings, plus one ordinary browser request as a baseline:

• ≤ 7 requests per domain (6 AI agents + 1 baseline), sequential, 10s timeout, at most one retry.
• Detects hard blocks (401/403/429/503) and bot-challenge interstitials (cf-challenge, "Just a moment", Turnstile).
• Fires CDN_BLOCKS_AI_RETRIEVAL / CDN_BLOCKS_AI_USER only when an AI agent is blocked while the baseline browser request succeeds — a differential result, so a site that is simply down is not misreported as AI-blocked.

Competitor URLs on other domains are probed separately, so budget ≤7 requests per distinct domain.

Tri-state, never fail-open

Every access verdict is allowed, blocked, or unavailable. unavailable is never scored as allowed. If robots.txt itself returns a 403 or a challenge page, that is reported as ROBOTS_UNREACHABLE (−10) rather than silently treated as permission granted. A 404 is different and is treated as genuinely open — no robots.txt means nothing is disallowed.

The eligibility gate

Being reachable is not one signal among six; it is the precondition for the other five to matter. So the weighted blend is capped, never quietly averaged away:

Condition	Cap	Resulting grade
All retrieval + user-tier agents blocked	25	F
Half or more blocked	55	D

The cap is always visible in the output as gated: true plus a human-readable gateReason, and the pre-cap number is retained as blendedScore.

Who it's for

• SEO teams adding GEO to their toolkit
• Content strategists optimising for AI citation
• Developers validating schema markup and crawlability
• Agencies running AI-readiness audits for clients
• Founders who want to know if AI can find them

Input

Parameter	Type	Default	Description
startUrls	array	required	URLs to audit
crawlPages	boolean	false	Follow internal links to audit multiple pages
maxPages	integer	50	Maximum pages to crawl (1–500)
maxConcurrency	integer	5	Concurrent page audits (1–20)
brandName	string	null	Your brand name for consistency checks
competitors	array	null	Up to 3 competitor URLs for comparison
auditCrawlability	boolean	true	Toggle crawlability audit
auditEntity	boolean	true	Toggle entity audit
auditContent	boolean	true	Toggle content audit
auditSchema	boolean	true	Toggle schema audit
auditAuthority	boolean	true	Toggle authority audit
auditAIOptimisation	boolean	true	Toggle AI optimisation audit

Output

Two record types are pushed to the default dataset:

Page records (type: "page")

One per audited URL with overall score, 6 category scores, full audit details, and all issues grouped by severity.

Site summary (type: "site-summary")

One per run with average scores, score distribution, top 10 issues ranked by severity and frequency, top 5 quick wins, and optional competitor comparison.

Issue format

Every issue includes:

• id — unique identifier (e.g., NO_ORG_SCHEMA, THIN_CONTENT)
• message — human-readable description
• severity — critical, warning, or info
• priority — 1 (highest) to 5 (lowest)
• estimatedImpact — high, medium, or low
• deduction — exact points this issue removed from its category score
• fixHint — actionable recommendation
• pagesAffected — URLs or ["site-wide"]

Every point deducted anywhere maps to a visible issue — there are no silent deductions, so a category score can always be reconciled against the issues that produced it.

Why we crawl without JavaScript

AI crawlers do not execute JavaScript. They fetch your raw HTML and read what is in it.

So this actor uses CheerioCrawler and deliberately does not render JS: it sees your pages exactly as an AI system sees them. That is the point of the design, not a shortcut. If your content only appears after hydration, an AI crawler gets the empty shell — and so does this audit, which is why JS_RENDERING_DEPENDENT is scored as a real problem rather than worked around.

Front-loading

Roughly 44% of LLM citations come from the first 30% of a document, so on pages over 600 words the actor measures where your substance actually sits. Sentences carrying a number, a comparison, a definition, or the page's own topic word are located within the document; if under 25% of them fall in the opening third, NO_ANSWER_FIRST fires. Fully deterministic — no LLM calls.

Pricing

Free to run. Pay only for Apify platform compute (CheerioCrawler is lightweight — a 50-page audit typically costs < $0.01).

Limitations

• Readability scoring uses Flesch-Kincaid, designed for English text; the content and authority pattern lists are English-only
• robots.txt, llms.txt, sitemap and the AI probe are cached per domain, fetched once per run
• The probe reports what a site does at audit time; bot-management rules can vary by geography and by request volume
• No LLM-based content analysis — all checks are rule-based and deterministic

Validated against

• cogcogcog.com — Shopify e-commerce (Cognitive Performance Drinks)
• goodrays.com — Shopify e-commerce (CBD drinks)
• example.com — minimal static HTML (edge case baseline)

Scoring is deterministic for identical responses: the same HTML and the same crawler-access results always produce the same score. The two page-speed checks are the exception, since they reflect timing measured at run time and can legitimately vary between runs near the 1000 ms and 3000 ms thresholds.