AI Search Readiness & Technical SEO Audit (GEO)

Can ChatGPT, Claude, Perplexity, Gemini and Google actually crawl, read and cite your website? This Actor checks in seconds and returns one scored report per site with a prioritised list of fixes.

What it checks

Area	What you get
AI crawler access	A matrix of 17 crawlers (GPTBot, OAI-AdsBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Amazonbot, meta-externalagent, CCBot, Bytespider, Googlebot, Bingbot) showing whether your robots.txt allows each one. The parser follows RFC 9309: group matching, longest match, * and $ wildcards.
Search vs training split	Blocking training bots is a legitimate choice and is flagged as info only. Blocking search/answer bots (which drive citations and referral traffic) is flagged as critical.
CDN / firewall test (opt-in)	robots.txt can say "allowed" while Cloudflare or another WAF quietly returns 403 to GPTBot or ClaudeBot. Turn this on for your own sites and the Actor requests the page with the real AI-crawler user agents, then flags any that get blocked.
llms.txt	Whether /llms.txt exists and is well formed. Soft-404 HTML pages are detected.
Sitemaps	Finds sitemaps via robots.txt or /sitemap.xml, follows sitemap indexes, and counts URLs.
Per page	HTTP status, response time, title, meta description, canonical, noindex (meta and X-Robots-Tag), lang, H1 count, word count, JavaScript-only rendering (many AI crawlers don't run JS), Open Graph, JSON-LD structured data types and parse errors, image alt coverage, and internal/external links.
Score	0–100, weighted by severity (critical, high, medium, low, info).

Who it's for

• SEO and marketing agencies auditing client sites for AI-search visibility ("GEO" / "AEO").
• Site owners checking that a CDN or bot-protection change hasn't silently blocked AI search crawlers.
• Monitoring: schedule it weekly across a portfolio of sites and alert on score drops via Apify integrations (Slack, email, webhooks).

Input

{
  "startUrls": [{ "url": "https://www.example.com/" }],
  "maxPagesPerSite": 5,
  "includePageDetails": true
}

Output (one item per site, abridged)

{
  "site": "https://www.bbc.co.uk",
  "score": 72,
  "robotsTxtFound": true,
  "aiBotAccess": [{ "bot": "OAI-SearchBot", "purpose": "ChatGPT search index", "allowed": false, "rule": "specific" }],
  "llmsTxt": { "present": false, "valid_heading": false, "links": 0, "bytes": 0 },
  "sitemapUrlCount": 18681,
  "pagesAudited": 3,
  "issueCounts": { "critical": 1, "high": 0, "medium": 0, "low": 3, "info": 1 },
  "issues": [{ "severity": "critical", "code": "search_bots_blocked", "message": "robots.txt blocks AI/search crawlers…" }],
  "pages": [{ "url": "https://www.bbc.co.uk/", "status": 200, "jsonLdTypes": ["WebSite"], "likelyJsRendered": false }]
}

(Real output from a test run on 21 Oct 2026.)

Pricing (pay per event)

• $0.02 per site report
• $0.004 per page audited

A 5-page audit costs $0.04, and 100 sites at 5 pages each cost about $4. You pay nothing else; platform usage is included. Set a maximum charge per run and the Actor stops cleanly when it's reached.

Responsible crawling

• Identifies itself honestly as AIReadinessAudit/1.0, and obeys robots.txt for its own requests (pages disallowed for it are skipped and listed in pagesSkippedByRobots).
• Polite: one request per second per site, at most 50 pages per site, no login, no form submission, no bypassing bot protection.
• Collects no personal data. It reads only technical page metadata.
• Intended for sites you own or manage, or for benchmarking public pages.

Limits

• It doesn't run JavaScript. That's deliberate, because it mirrors what most AI crawlers see. Pages flagged likelyJsRendered need server-side rendering or pre-rendering to be visible to them.
• The firewall test is a spot check of one URL with four user agents. Some CDNs verify real crawler IPs, so a pass from our IP doesn't prove the real bot gets through. Check your server logs as well.
• llms.txt is an emerging convention, not a standard, so a missing file is scored as low severity only.

Use from AI agents (MCP)

This Actor is pay-per-event, charges only for events, and needs no special permissions, so AI agents can call it as a tool through the Apify MCP server (mcp.apify.com) and pay per call. Example agent prompt: "Audit example.com for AI search readiness and list the critical fixes."

Changelog

• 1.0 (Oct 2026): first release.