AI Search Readiness Audit (AEO/GEO)

Audit URLs in bulk for AI search readiness: can ChatGPT, Perplexity, Claude and other answer engines actually crawl, parse and cite your pages? This Actor turns raw robots.txt, HTML and metadata into a deterministic PASS / WARN / FAIL verdict with a 0–100 score per URL — plus the exact evidence an agency can paste straight into a client audit.

What it checks per URL

1. AI crawler access matrix (with evidence). robots.txt is evaluated for 13 AI crawlers — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot (incl. legacy anthropic-ai), Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent — per RFC 9309 (longest match, Allow beats Disallow on ties, wildcard groups). Every allowed/blocked/no-rule verdict quotes the exact robots.txt line that produced it. Meta robots, noai and X-Robots-Tag are checked too.
2. llms.txt. Presence per domain, structural validity (H1 title, sections, markdown links) and a consistency probe of a few advertised links. Honestly labeled: llms.txt is an emerging convention (llmstxt.org) — Google does not use it.
3. Structured data. JSON-LD extraction: detected @type values (Organization, WebSite, Article, FAQPage, Product, BreadcrumbList…) and syntax errors that make crawlers silently drop a block.
4. Renderability without JavaScript (heuristic). How much visible text the raw HTML exposes, empty SPA root shells (<div id="root">…), text-to-markup ratio. Several AI crawlers do not execute JavaScript — a client-side-only page is invisible to them.
5. Citation signals. Title, meta description, H1 and heading hierarchy, canonical, lang, machine-readable dates, Open Graph.

Each URL gets a verdict, a score, per-check statuses and an issues[] list with severity, plain-English message, recommendation and evidence.

Input — three chainable sources

• startUrls — a plain list of URLs.
• sitemapUrl — an XML sitemap or sitemap index; URLs are pulled automatically.
• datasetId + urlField — Dataset chaining: point it at the output of any Apify crawler run and audit what it found, no glue code needed.

Sources combine, URLs are deduplicated, and maxUrls caps the run (default 100, max 5000).

{
  "startUrls": [{ "url": "https://www.example.com/" }],
  "sitemapUrl": "https://www.example.com/sitemap.xml",
  "maxUrls": 200
}

Output

• Dataset — one record per URL with flat fields plus the full per-bot evidence matrix.
• REPORT.html — self-contained, client-ready HTML report (key-value store).
• SUMMARY.md — portable Markdown summary for tickets and docs.
• RESULTS.csv — spreadsheet-ready export with formula-injection-safe escaping.
• OUTPUT — JSON run summary: counts, billed URLs, blocked-bot ranking, issue frequencies.

Pricing — you never pay for guesses

Pay per event: $0.009 per audited URL (that is $9.00 per 1,000 URLs), plus a near-zero $0.00005 Actor start event.

• A URL is charged only when the audit reaches a definitive PASS, WARN or FAIL verdict — and only after its record is written to the dataset.
• Unresolved URLs (DNS failures, timeouts, 429 rate limits, 5xx after an automatic retry, invalid or unsafe URLs) are delivered in the dataset for free and never charged.
• WAF/bot-protection blocks (403, Cloudflare challenges) get the dedicated status blocked_by_waf — useful information ("an AI crawler will not get in here either"), delivered for free and never charged.
• Duplicate URLs are deduplicated before work, so nothing is billed twice.
• Set maxTotalChargeUsd on the run to cap your spend. The Actor trims the input before doing any work, so your limit is never exceeded and skippedByChargeLimit reports what was left out.
• Platform usage is covered by the Actor creator; the event price is what you pay.

Honest limits (read before you buy)

• llms.txt is an emerging convention. Google has stated it does not use it; support among AI vendors varies. It is reported as informational, not as a ranking requirement.
• Renderability is a heuristic on raw HTML, not a full browser render. A client_side_only verdict means non-JS crawlers get an empty shell; it does not measure what Googlebot's renderer eventually sees.
• This is a technical readiness audit — not a guarantee that AI assistants will cite your pages. Content quality and authority are out of scope by design.
• The Actor does not log in, bypass bot protection, execute JavaScript or use any LLM. That keeps runs deterministic, reproducible and cheap — and makes the Actor safe to call from AI agents and MCP pipelines.

Security and responsible use

• Audit only sites you own or are authorized to test.
• Localhost, private/link-local/reserved IP ranges, cloud metadata hosts, URL credentials and non-standard ports are blocked, with DNS re-validated at connection time (no DNS-rebinding gap). Every redirect hop is checked again.
• Response bodies are capped, requests time out, retries are bounded (one per URL) and concurrency is limited, so audited sites are not hammered.
• No personal data is collected; no proxy or external API is used.

Works well with

Migrating a site? Validate your redirect map with Site Migration Redirect Map Validator & Audit — same evidence-first reporting, built by the same author.

Recommended workflow

1. Run the audit on your key pages (or the full sitemap).
2. Fix every FAIL (blocked AI crawlers, noindex, client-side-only rendering), then review WARN items (structured data, canonicals, thin HTML).
3. Re-run and archive REPORT.html as the before/after artifact for your client.
4. Schedule a periodic re-audit with Apify Schedules to catch regressions.

Sources

• RFC 9309 — Robots Exclusion Protocol
• OpenAI crawlers
• Anthropic crawlers
• Perplexity crawlers
• Google-Extended
• llms.txt proposal
• Google Search Central: JSON-LD structured data