Website Audit & AI Readiness Checker

Find out whether search engines and AI assistants (ChatGPT, Claude, Perplexity, Google AI Overviews, Copilot) can find, read and quote a website. The Actor crawls a site and returns:

• an AI readiness score (0-100, grade A-F) per site with a breakdown and the most important fixes,
• an SEO score (0-100) per page with a prioritized list of issues and a concrete fix for each one,
• a client-ready HTML report you can send as it is.

What it checks

AI crawler access (robots.txt) for 20 agents, grouped by purpose:

Purpose	Agents
AI search and assistants	OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot, Amazonbot, DuckAssistBot, MistralAI-User, Meta-ExternalFetcher
Search engines that feed AI answers	Googlebot, Bingbot
Model training only	GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider

Each agent is classified for the home page as allowed, partial (home page open, some sections closed) or blocked. Rules are matched like real crawlers do (RFC 9309): groups for the same agent are merged, * and $ wildcards are supported, the most specific rule wins. Blocking only training crawlers does not lower the score: that is a legitimate way to opt out of model training while staying visible in AI answers.

Other site checks (once per site)

• robots.txt reachable. A server error on robots.txt can make crawlers skip the whole site.
• XML sitemap: actually loaded and validated, number of entries
• llms.txt and llms-full.txt: present and well-formed (title, summary, links)
• HTTP redirects to HTTPS

Page checks

• Content visible without JavaScript. Many AI crawlers do not run JavaScript; client-side rendered pages look empty to them.
• Structured data: JSON-LD and microdata, invalid JSON, all schema.org types found, and missing recommended properties for Organization, LocalBusiness, WebSite, Article, NewsArticle, BlogPosting, Product, FAQPage, BreadcrumbList, Event, Recipe, HowTo and VideoObject
• noindex and nosnippet / max-snippet:0 (meta robots, googlebot meta, X-Robots-Tag). nosnippet also keeps a page out of AI-generated answers.
• Title and meta description (missing, too short, too long), duplicates across the site
• H1 headings, canonical, lang, viewport, Open Graph, images without alt text, thin content
• HTTPS, redirects, response time, HTTP errors (4xx/5xx), last modified date

AI readiness score

Part	Points
AI search and assistant crawlers can access the site (weighted by reach: OpenAI 3, Anthropic 2, Perplexity 2, others 1)	30
Content readable without JavaScript	20
Google and Bing can crawl the site	15
Valid and complete structured data	15
Pages may be indexed and quoted (no noindex / nosnippet)	10
llms.txt present and well-formed	5
Valid XML sitemap	5

Grades: A 90+, B 80+, C 65+, D 50+, F below 50. If site checks are switched off, the score is based on the page checks only.

Why use it

• Agencies: audit many client sites in one run, send the HTML report, export the data to CSV/Excel/JSON.
• Developers: run it in CI or on a schedule after each release.
• AI agents: callable via the Apify MCP server to check a site on request.

Input

Field	Description	Default
startUrls	Sites to audit, usually the home page. example.com and www.example.com count as one site.	crawlee.dev
maxPagesPerSite	Max pages analyzed per site	10
includeSiteChecks	robots.txt, AI crawlers, sitemap, llms.txt, HTTPS	true
maxConcurrency	Parallel requests	5

The crawler stays on the site, follows the redirect of a start URL (e.g. to www. or another domain), skips links to files (PDF, images, ...) and respects robots.txt.

Output

Dataset, one item per:

• type: "summary": per site, AI readiness score and breakdown, average SEO score, broken pages, duplicates, top issues (use the view "Summary")
• type: "site": site checks including the access of every AI crawler
• type: "page": page checks with score, issues and fixes
• type: "skipped": a start URL that robots.txt does not allow to crawl

Example page item:

{
  "type": "page",
  "url": "https://example.com/blog",
  "statusCode": 200,
  "score": 78,
  "issues": [
    { "code": "structured-data-incomplete", "severity": "medium", "message": "Structured data is missing recommended properties: Article (author, image).", "fix": "Add the missing properties so rich results and AI systems can use the data." }
  ],
  "jsDependent": false,
  "nosnippet": false,
  "structuredData": { "jsonLdBlocks": 1, "jsonLdInvalid": 0, "types": ["Article", "Person"], "microdata": false, "incomplete": [{ "type": "Article", "missing": ["author", "image"] }] }
}

Key-value store

• OUTPUT: the summaries of all sites as JSON
• REPORT: the HTML report (open it from the run's Storage tab)

Pricing

Pay per event, no subscription: one event per analyzed page and one per site check. Error pages count as analyzed pages. Pages blocked by robots.txt, links that redirect to other sites, the summary and the HTML report are free.

Data and privacy

The Actor only reads technical website data (HTML structure, meta tags, structured data types, robots.txt). It does not extract e-mail addresses, phone numbers, names or any other personal data.

Limits

• HTML is analyzed without executing JavaScript, the same way most AI crawlers see a page. Content that only appears in the browser is reported as js-rendered-content.
• Core Web Vitals / Lighthouse are not part of this Actor.
• robots.txt tells well-behaved crawlers what to do. Some user-triggered AI fetchers may not follow it.