AI Search Readiness & SEO Audit — AI bots, llms.txt, schema

Find out in one run whether AI search engines and assistants can find, read and cite a website, and fix the SEO basics at the same time.

ChatGPT search, Perplexity, Claude and Google's AI features now send real traffic. Many sites block them by accident in robots.txt, have no llms.txt, or lack the structured data AI answers rely on. This Actor checks all of that, page by page, and returns a 0–100 AI-readiness score, page scores, and a plain-English fix list.

What it checks

Per site

• AI crawler access in robots.txt, bot by bot: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot-Extended, meta-externalagent, Amazonbot, CCBot, Bytespider. Each is reported as allowed or blocked.
• llms.txt present or not.
• XML sitemap found (via robots.txt or the default locations) and how many URLs it lists.
• AI-readiness score and recommendations, e.g. "Allow AI answer engines in robots.txt (PerplexityBot is blocked)".

Per page (crawled breadth-first from your start URL, same site only)

• Title and meta description (presence and length), <h1>, canonical, lang, Open Graph.
• schema.org structured data types (JSON-LD and microdata).
• Indexability (meta robots and X-Robots-Tag noindex), HTTP status, redirects, response time.
• Images without alt text, word count, internal and external link counts.
• A page score and a list of issues.

Who is it for?

• SEO and marketing agencies: run it across every client site each month and send the fix list.
• Site owners and content teams: see whether your robots.txt is hiding you from AI search.
• Developers: add it to CI or a weekly schedule to catch accidental noindex or blocked bots after a deploy.

How to use

1. Enter one or more sites (example.com or a start URL like https://example.com/blog).
2. Choose Max pages per site (10 is a good first audit).
3. Run it. The Site summaries view gives the scores and recommendations; Page audits gives the per-page detail.

Example output: site summary

{
  "type": "site",
  "site": "https://example.com",
  "aiReadinessScore": 64,
  "averagePageScore": 78,
  "pagesAudited": 10,
  "aiCrawlers": { "GPTBot": "blocked", "OAI-SearchBot": "allowed", "ClaudeBot": "allowed", "PerplexityBot": "blocked", "Google-Extended": "allowed" },
  "aiCrawlersBlocked": ["GPTBot", "PerplexityBot"],
  "llmsTxt": false,
  "sitemap": "https://example.com/sitemap.xml",
  "sitemapUrlCount": 214,
  "recommendations": [
    "Add /llms.txt: a short Markdown guide to your key pages for AI assistants",
    "Allow AI answer engines in robots.txt (PerplexityBot are blocked) if you want to be cited in AI search",
    "Fix on 6 page(s): No schema.org structured data"
  ]
}

Example output: page audit

{
  "type": "page",
  "url": "https://example.com/pricing",
  "status": 200,
  "score": 82,
  "issues": ["Meta description length 34 (aim for 50–170)", "No schema.org structured data"],
  "title": "Pricing — Example",
  "canonical": "https://example.com/pricing",
  "schemaTypes": [],
  "indexable": true,
  "imagesWithoutAlt": 0,
  "wordCount": 612,
  "responseTimeMs": 184
}

How the AI-readiness score works

It is 60% the average page score, plus 10 points each for: an llms.txt, an XML sitemap, schema.org data on the homepage, and access for the AI answer engines (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot). Blocking training crawlers like GPTBot or CCBot is a legitimate choice and doesn't lower the score. It is reported so you can see exactly what you're blocking.

Pricing

Pay per page audited. Site summaries are free. Robots.txt, llms.txt and sitemap checks are included.

Free Apify plan: up to 50 pages per run. Any paid Apify plan removes the cap.

Good to know

• Polite crawling: the Actor reads robots.txt and respects it for its own user agent (locaihost-site-audit). It fetches one page at a time per site with a short pause, and only follows same-site links (no PDFs or images).
• No JavaScript rendering in v1: pages are analysed as served. That's what most AI crawlers see too.
• Audit sites you own or manage. It's built for site owners and their agencies.

Feedback

Want another check (Core Web Vitals, hreflang, broken-link checking)? Open an issue on the Issues tab.