AI Search Readiness Audit: GEO/AEO, llms.txt & AI Crawler Checker

Find out if ChatGPT, Claude, Perplexity, Gemini and Google AI Overviews can find, read and cite your website, then get a prioritized fix list. Bulk-audit 1 or 10,000 sites in one run. You pay per site audited, and failed sites are free.

What does AI Search Readiness Audit do?

For every website you give it, this Actor checks the signals that decide whether AI answer engines can use your content (Generative Engine Optimization / Answer Engine Optimization):

Category (weight)	What is checked
AI crawler access (30%)	robots.txt rules for 19 AI and search bots: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Bingbot, Applebot-Extended, DuckAssistBot, Meta-ExternalAgent, Amazonbot, CCBot, Bytespider, cohere-ai, MistralAI-User. Also flags homepage noindex and WAF/bot-protection blocks (403s)
Discovery (20%)	/llms.txt and /llms-full.txt (llmstxt.org format check), XML sitemap, sitemap declared in robots.txt, markdown alternates
Structured data (20%)	schema.org JSON-LD types per page, Organization/Person entity, FAQPage/HowTo, invalid JSON-LD
Content extractability (20%)	Server-rendered text vs JavaScript-only shells (most AI crawlers don't run JS), H1/heading outline, visible dates
Metadata & trust (10%)	title, meta description, canonical, lang, Open Graph, author signals, HTTP errors

Each site gets a 0-100 score, an A-F grade, category scores, and prioritized issues (high/medium/low), each with a concrete fix. When something is missing you also get a ready-to-paste llms.txt and a robots.txt snippet that unblocks the AI search bots.

Who is it for?

• SEO and GEO agencies running AI-visibility audits for clients and prospects at scale
• Lead generation: score a list of prospect domains, then pitch the ones graded D/F
• Marketing and content teams tracking AI-readiness across their own properties (schedule it weekly)
• AI agents and MCP clients that need a fast, structured website-readiness check

How to use it

1. Enter one or more websites (domains or URLs).
2. Optionally raise Pages to sample per site (the homepage plus diverse pages from the sitemap; default 5).
3. Click Start, then download results as JSON, CSV, Excel, or through the API.

Input example

{
  "urls": ["https://apify.com", "python.org", "example.com"],
  "maxPagesPerSite": 5
}

Output example (trimmed)

{
  "site": "https://example.com",
  "score": 44,
  "grade": "F",
  "categoryScores": { "aiCrawlerAccess": 95, "discovery": 0, "structuredData": 0, "contentExtractability": 52, "metadataAndTrust": 55 },
  "issues": [
    { "priority": "high", "category": "discovery", "issue": "No XML sitemap discovered (robots.txt Sitemap: line or /sitemap.xml).", "fix": "Publish /sitemap.xml and reference it from robots.txt with a Sitemap: line." },
    { "priority": "high", "category": "structured-data", "issue": "No JSON-LD structured data on any sampled page.", "fix": "Add schema.org JSON-LD (WebSite, Organization, Article/Product as relevant)." }
  ],
  "aiCrawlerAccess": [ { "token": "OAI-SearchBot", "operator": "OpenAI", "purpose": "search", "allowed": true, "rule": "wildcard" } ],
  "llmsTxt": { "present": false },
  "sitemap": { "urlsFound": 0, "sitemapsChecked": 1, "declaredInRobots": false },
  "pagesAudited": 1,
  "recommendedLlmsTxt": "# Example Domain\n\n## Key pages\n\n- [Example Domain](https://example.com/)\n"
}

Each result also has a pages array with per-page signals: word count, JSON-LD types, H1 count, canonical, Open Graph, noindex, response time, and more.

Pricing (pay per event)

• $0.05 per site audited (covers the homepage plus up to 4 more pages)
• $0.005 per extra page when you set maxPagesPerSite above 5
• Sites that fail to load are not charged.

100 sites cost about $5. You can cap your spend with the run's maximum-charge setting, and the Actor stops cleanly when the cap is reached.

Responsible crawling

The Actor reads only public pages: robots.txt, llms.txt, sitemaps, and a small sample of pages per site. It identifies itself as AIReadinessAuditBot and skips sampled pages that robots.txt disallows for it (configurable). It never logs in and never bypasses bot protection.

FAQ

Is a high score a guarantee of AI citations? No. It measures technical readiness (can AI systems access, parse and trust your pages). Citations also depend on content quality and authority.

Should I allow GPTBot / ClaudeBot (training bots)? That's a business decision, so blocking them does not lower your score. The score only penalizes blocking search and user-triggered bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User…), because blocking those removes you from AI answers.

Can I run it on a schedule? Yes. Use Apify Schedules to re-audit weekly and track score changes.

Found a bug or need a check added? Open an issue on the Actor's Issues tab.