AI Readiness Checker

What does AI Readiness Checker do?

AI Readiness Checker audits how accessible and understandable a website is to AI search engines, retrieval systems, and browser agents. It analyzes crawl access, machine-readable content, structured semantics, source attribution, page organization, and interaction clarity.

Each website receives a deterministic AI Readiness score from 0 to 100, an A-F grade, category scores, a confidence score, evidence-backed issues, and prioritized technical recommendations. Run it from Apify Console or integrate it through the API, schedules, webhooks, datasets, and automation tools.

The Actor measures technical readiness only. It does not predict rankings, citations, inclusion in AI answers, AI-generated content, or hallucination probability.

Why use AI Readiness Checker?

Use this Actor to:

• verify whether important public content is available to AI search and retrieval crawlers;
• distinguish search access from optional AI model-training permissions;
• find content that depends heavily on client-side rendering;
• evaluate Schema.org coverage and validation quality;
• identify weak headings, missing canonical URLs, incomplete author or date signals, and unclear page structure;
• detect controls and form fields that are difficult for browser agents to understand;
• turn technical findings into a prioritized implementation backlog;
• monitor readiness across multiple domains with consistent, repeatable scoring.

How to use AI Readiness Checker

1. Open the Actor in Apify Console.
2. Add one or more public website URLs under Websites.
3. Choose the maximum number of representative pages to analyze per website.
4. Enable or disable optional AI crawler, llms.txt, and browser-agent checks.
5. Configure a proxy only when your use case requires one.
6. Start the run and open the default dataset to review the reports.

Equivalent www and non-www inputs are assessed once. Every domain receives its own page budget, and a failure on one website does not stop the remaining websites.

Input

Field	Description	Default
startUrls	One or more public HTTP(S) website URLs.	https://apify.com
maxPagesPerSite	Maximum representative pages analyzed per domain.	1
checkLlmsTxt	Checks the optional /llms.txt file and a bounded set of its links.	true
checkAiCrawlers	Compares declared crawler rules with bounded effective-access requests.	true
checkAgentUsability	Evaluates semantic controls, labels, landmarks, and interaction barriers.	true
maxConcurrency	Maximum websites assessed concurrently.	3
timeoutSecs	Timeout for website and resource requests.	45
proxyConfiguration	Optional Apify Proxy or custom proxy configuration.	Disabled

Example input:

{
    "startUrls": [{ "url": "https://apify.com" }, { "url": "https://crawlee.dev" }],
    "maxPagesPerSite": 25,
    "checkLlmsTxt": true,
    "checkAiCrawlers": true,
    "checkAgentUsability": true,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
}

Output

The Actor stores one report per deduplicated domain in the default dataset. Reports contain concise evidence and summaries rather than raw page HTML.

{
    "url": "https://apify.com/",
    "domain": "apify.com",
    "status": "success",
    "score": 84,
    "grade": "B",
    "gradeLabel": "Good",
    "scoreConfidence": 0.94,
    "scores": {
        "aiDiscoveryAndAccess": { "score": 23, "maxScore": 25 },
        "machineReadability": { "score": 17, "maxScore": 20 },
        "structuredSemantics": { "score": 17, "maxScore": 20 },
        "contentRetrievability": { "score": 16, "maxScore": 20 },
        "agentUsability": { "score": 8, "maxScore": 10 },
        "aiSpecificHints": { "score": 3, "maxScore": 5 }
    },
    "aiCrawlerAccess": {
        "OAI-SearchBot": {
            "provider": "OpenAI",
            "purpose": "search",
            "declaredAccess": "allowed",
            "effectiveAccess": "allowed"
        }
    },
    "trainingPolicy": {
        "GPTBot": {
            "provider": "OpenAI",
            "purpose": "training",
            "declaredAccess": "blocked",
            "effectiveAccess": "not_checked"
        }
    },
    "issues": [],
    "recommendations": [],
    "pagesAnalyzed": 25
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

Data table

Field	Description
status	Whether the assessment succeeded, partially succeeded, or failed.
score, grade	Deterministic 0-100 readiness score and A-F grade.
scoreConfidence	Completeness of the collected evidence from 0 to 1.
scores	Scores for the six readiness categories.
aiCrawlerAccess	Declared and effective access for search and user-initiated crawlers.
trainingPolicy	Informational model-training and extended-use crawler policies.
llmsTxt	Optional file status, structure, links, redirects, and link checks.
content	Primary-content availability, indexability, and rendering dependency.
structuredSemantics	Schema.org types, coverage, warnings, and errors.
agentUsability	Unnamed controls, unlabeled inputs, and interaction barriers.
issues	Findings with stable codes, severity, evidence, and affected URLs.
recommendations	Deduplicated technical actions ordered by priority.
analysis	Assessment timing information.

AI Readiness score

Category	Maximum points
AI Discovery & Crawl Access	25
Machine Readability	20
Structured Semantics	20
Content Retrievability & Citability	20
Agent Usability	10
AI-specific Hints	5

Grades:

• A, 90-100: Excellent
• B, 75-89: Good
• C, 60-74: Moderate
• D, 40-59: Poor
• F, 0-39: Critical

Scoring is deterministic and bounded. The confidence score represents evidence completeness and does not increase or decrease the readiness score.

AI search access and training permissions

Search and user-initiated retrieval crawlers include OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, and Googlebot. Their access can affect technical discovery and retrieval.

GPTBot, ClaudeBot, and Google-Extended are classified as training or extended-use crawlers. Blocking these crawlers is reported as an informational publisher policy and does not reduce the AI Readiness score.

Declared robots access and effective access are reported separately. A crawler can be allowed by robots.txt while receiving HTTP 401, 403, or 429 responses, a CAPTCHA, a WAF challenge, a login wall, or another blocking interstitial. The Actor is diagnostic and does not bypass security controls.

Severity levels

• CRITICAL: site-wide retrieval blocking or no crawlable primary content;
• HIGH: major rendering dependency, indexing restrictions, structured-data errors, or interaction barriers;
• MEDIUM: weak hierarchy, incomplete attribution, or ambiguous controls;
• LOW: optional improvements such as a missing or malformed llms.txt;
• INFO: publisher policy choices that do not require a readiness fix.

Severity and score impact are separate. An informational training-policy finding always has zero score impact.

How much does an AI readiness audit cost?

Cost depends on the number of domains, page budget, page loading time, retries, and proxy usage. Start with one website and a smaller page budget to estimate consumption for your targets, then increase coverage for production audits.

Large websites cannot consume the page budget allocated to other domains. Current Apify plan limits and platform pricing apply.

Tips and advanced options

• Start with one page for a quick homepage assessment.
• Increase the budget to 25 pages for a broader representative audit.
• Compare declaredAccess and effectiveAccess before changing crawler policies.
• Resolve access failures before interpreting missing content or semantic evidence.
• Prioritize CRITICAL and HIGH recommendations, then rerun the audit.
• Treat a good score with low confidence as a signal to collect more evidence.
• Keep crawler rules limited to public content and retain restrictions for private or sensitive paths.

FAQ, disclaimers, and support

Does this guarantee that a website appears in AI answers?

No. The Actor does not predict proprietary rankings, citations, answer selection, or placement in any AI product.

Does blocking AI training reduce the score?

No. Training and extended-use crawler restrictions are informational and have zero score impact.

Is llms.txt required?

No. llms.txt is an optional and evolving convention. A missing file produces only a LOW-severity finding and has a small score impact.

Is this a complete SEO or accessibility audit?

No. It evaluates selected technical signals relevant to machine retrieval and browser-agent understanding. It is not a complete SEO review or WCAG conformance audit.

Is website analysis legal?

Only analyze websites you are authorized to assess. Respect applicable laws, website terms, robots policies, rate limits, and personal-data obligations.

Authenticated content, regional responses, temporary outages, anti-bot systems, and highly dynamic interfaces can affect results. Use the Actor's Issues tab for support or feature requests. Custom audit rules and integrations are available as tailored solutions.

Local development and deployment

npm install
npm run build
npm test
npm run lint
apify run

Local storage remains on your machine and is not uploaded automatically to Apify Console.

Deploy after reviewing the Actor configuration and target account:

apify login
apify push