# AI Readiness Checker (`ib4ngz/ai-readiness-checker`) Actor

Assess technical AI crawl access, machine readability, semantics, retrievability, and browser-agent usability.

- **URL**: https://apify.com/ib4ngz/ai-readiness-checker.md
- **Developed by:** [Iqbal R](https://apify.com/ib4ngz) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $6.00 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### What does AI Readiness Checker do?

**AI Readiness Checker audits how accessible and understandable a website is to AI search engines, retrieval systems, and browser agents.** It analyzes crawl access, machine-readable content, structured semantics, source attribution, page organization, and interaction clarity.

Each website receives a deterministic **AI Readiness score from 0 to 100**, an A-F grade, category scores, a confidence score, evidence-backed issues, and prioritized technical recommendations. Run it from Apify Console or integrate it through the API, schedules, webhooks, datasets, and automation tools.

The Actor measures technical readiness only. It does not predict rankings, citations, inclusion in AI answers, AI-generated content, or hallucination probability.

### Why use AI Readiness Checker?

Use this Actor to:

- verify whether important public content is available to AI search and retrieval crawlers;
- distinguish search access from optional AI model-training permissions;
- find content that depends heavily on client-side rendering;
- evaluate Schema.org coverage and validation quality;
- identify weak headings, missing canonical URLs, incomplete author or date signals, and unclear page structure;
- detect controls and form fields that are difficult for browser agents to understand;
- turn technical findings into a prioritized implementation backlog;
- monitor readiness across multiple domains with consistent, repeatable scoring.

### How to use AI Readiness Checker

1. Open the Actor in Apify Console.
2. Add one or more public website URLs under **Websites**.
3. Choose the maximum number of representative pages to analyze per website.
4. Enable or disable optional AI crawler, `llms.txt`, and browser-agent checks.
5. Configure a proxy only when your use case requires one.
6. Start the run and open the default dataset to review the reports.

Equivalent www and non-www inputs are assessed once. Every domain receives its own page budget, and a failure on one website does not stop the remaining websites.

### Input

| Field                 | Description                                                               | Default             |
| --------------------- | ------------------------------------------------------------------------- | ------------------- |
| `startUrls`           | One or more public HTTP(S) website URLs.                                  | `https://apify.com` |
| `maxPagesPerSite`     | Maximum representative pages analyzed per domain.                         | `1`                 |
| `checkLlmsTxt`        | Checks the optional `/llms.txt` file and a bounded set of its links.      | `true`              |
| `checkAiCrawlers`     | Compares declared crawler rules with bounded effective-access requests.   | `true`              |
| `checkAgentUsability` | Evaluates semantic controls, labels, landmarks, and interaction barriers. | `true`              |
| `maxConcurrency`      | Maximum websites assessed concurrently.                                   | `3`                 |
| `timeoutSecs`         | Timeout for website and resource requests.                                | `45`                |
| `proxyConfiguration`  | Optional Apify Proxy or custom proxy configuration.                       | Disabled            |

Example input:

```json
{
    "startUrls": [{ "url": "https://apify.com" }, { "url": "https://crawlee.dev" }],
    "maxPagesPerSite": 25,
    "checkLlmsTxt": true,
    "checkAiCrawlers": true,
    "checkAgentUsability": true,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
}
```

### Output

The Actor stores one report per deduplicated domain in the default dataset. Reports contain concise evidence and summaries rather than raw page HTML.

```json
{
    "url": "https://apify.com/",
    "domain": "apify.com",
    "status": "success",
    "score": 84,
    "grade": "B",
    "gradeLabel": "Good",
    "scoreConfidence": 0.94,
    "scores": {
        "aiDiscoveryAndAccess": { "score": 23, "maxScore": 25 },
        "machineReadability": { "score": 17, "maxScore": 20 },
        "structuredSemantics": { "score": 17, "maxScore": 20 },
        "contentRetrievability": { "score": 16, "maxScore": 20 },
        "agentUsability": { "score": 8, "maxScore": 10 },
        "aiSpecificHints": { "score": 3, "maxScore": 5 }
    },
    "aiCrawlerAccess": {
        "OAI-SearchBot": {
            "provider": "OpenAI",
            "purpose": "search",
            "declaredAccess": "allowed",
            "effectiveAccess": "allowed"
        }
    },
    "trainingPolicy": {
        "GPTBot": {
            "provider": "OpenAI",
            "purpose": "training",
            "declaredAccess": "blocked",
            "effectiveAccess": "not_checked"
        }
    },
    "issues": [],
    "recommendations": [],
    "pagesAnalyzed": 25
}
```

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

### Data table

| Field                 | Description                                                           |
| --------------------- | --------------------------------------------------------------------- |
| `status`              | Whether the assessment succeeded, partially succeeded, or failed.     |
| `score`, `grade`      | Deterministic 0-100 readiness score and A-F grade.                    |
| `scoreConfidence`     | Completeness of the collected evidence from 0 to 1.                   |
| `scores`              | Scores for the six readiness categories.                              |
| `aiCrawlerAccess`     | Declared and effective access for search and user-initiated crawlers. |
| `trainingPolicy`      | Informational model-training and extended-use crawler policies.       |
| `llmsTxt`             | Optional file status, structure, links, redirects, and link checks.   |
| `content`             | Primary-content availability, indexability, and rendering dependency. |
| `structuredSemantics` | Schema.org types, coverage, warnings, and errors.                     |
| `agentUsability`      | Unnamed controls, unlabeled inputs, and interaction barriers.         |
| `issues`              | Findings with stable codes, severity, evidence, and affected URLs.    |
| `recommendations`     | Deduplicated technical actions ordered by priority.                   |
| `analysis`            | Assessment timing information.                                        |

### AI Readiness score

| Category                            | Maximum points |
| ----------------------------------- | -------------: |
| AI Discovery & Crawl Access         |             25 |
| Machine Readability                 |             20 |
| Structured Semantics                |             20 |
| Content Retrievability & Citability |             20 |
| Agent Usability                     |             10 |
| AI-specific Hints                   |              5 |

Grades:

- **A, 90-100:** Excellent
- **B, 75-89:** Good
- **C, 60-74:** Moderate
- **D, 40-59:** Poor
- **F, 0-39:** Critical

Scoring is deterministic and bounded. The confidence score represents evidence completeness and does not increase or decrease the readiness score.

### AI search access and training permissions

Search and user-initiated retrieval crawlers include OAI-SearchBot, Claude-SearchBot, Claude-User, PerplexityBot, and Googlebot. Their access can affect technical discovery and retrieval.

GPTBot, ClaudeBot, and Google-Extended are classified as training or extended-use crawlers. Blocking these crawlers is reported as an informational publisher policy and does **not** reduce the AI Readiness score.

Declared robots access and effective access are reported separately. A crawler can be allowed by `robots.txt` while receiving HTTP 401, 403, or 429 responses, a CAPTCHA, a WAF challenge, a login wall, or another blocking interstitial. The Actor is diagnostic and does not bypass security controls.

### Severity levels

- **CRITICAL:** site-wide retrieval blocking or no crawlable primary content;
- **HIGH:** major rendering dependency, indexing restrictions, structured-data errors, or interaction barriers;
- **MEDIUM:** weak hierarchy, incomplete attribution, or ambiguous controls;
- **LOW:** optional improvements such as a missing or malformed `llms.txt`;
- **INFO:** publisher policy choices that do not require a readiness fix.

Severity and score impact are separate. An informational training-policy finding always has zero score impact.

### How much does an AI readiness audit cost?

Cost depends on the number of domains, page budget, page loading time, retries, and proxy usage. Start with one website and a smaller page budget to estimate consumption for your targets, then increase coverage for production audits.

Large websites cannot consume the page budget allocated to other domains. Current Apify plan limits and platform pricing apply.

### Tips and advanced options

- Start with one page for a quick homepage assessment.
- Increase the budget to 25 pages for a broader representative audit.
- Compare `declaredAccess` and `effectiveAccess` before changing crawler policies.
- Resolve access failures before interpreting missing content or semantic evidence.
- Prioritize CRITICAL and HIGH recommendations, then rerun the audit.
- Treat a good score with low confidence as a signal to collect more evidence.
- Keep crawler rules limited to public content and retain restrictions for private or sensitive paths.

### FAQ, disclaimers, and support

#### Does this guarantee that a website appears in AI answers?

No. The Actor does not predict proprietary rankings, citations, answer selection, or placement in any AI product.

#### Does blocking AI training reduce the score?

No. Training and extended-use crawler restrictions are informational and have zero score impact.

#### Is llms.txt required?

No. `llms.txt` is an optional and evolving convention. A missing file produces only a LOW-severity finding and has a small score impact.

#### Is this a complete SEO or accessibility audit?

No. It evaluates selected technical signals relevant to machine retrieval and browser-agent understanding. It is not a complete SEO review or WCAG conformance audit.

#### Is website analysis legal?

Only analyze websites you are authorized to assess. Respect applicable laws, website terms, robots policies, rate limits, and personal-data obligations.

Authenticated content, regional responses, temporary outages, anti-bot systems, and highly dynamic interfaces can affect results. Use the Actor's **Issues** tab for support or feature requests. Custom audit rules and integrations are available as tailored solutions.

### Local development and deployment

```bash
npm install
npm run build
npm test
npm run lint
apify run
```

Local storage remains on your machine and is not uploaded automatically to Apify Console.

Deploy after reviewing the Actor configuration and target account:

```bash
apify login
apify push
```

# Actor input Schema

## `startUrls` (type: `array`):

One or more HTTP(S) website URLs. Equivalent www/non-www hostnames are assessed once.

## `maxPagesPerSite` (type: `integer`):

Maximum number of representative pages analyzed separately for each domain.

## `checkLlmsTxt` (type: `boolean`):

Inspect the optional /llms.txt convention. Missing files have only a small score impact.

## `checkAiCrawlers` (type: `boolean`):

Send bounded diagnostic requests using search and user-initiated crawler user agents. Security controls are never bypassed.

## `checkAgentUsability` (type: `boolean`):

Analyze semantic labels, landmarks, controls, and blocking interstitial signals relevant to browser agents.

## `maxConcurrency` (type: `integer`):

Maximum number of independent website assessments running concurrently.

## `timeoutSecs` (type: `integer`):

Timeout for each shared-intelligence page/resource request and AI-specific diagnostic request.

## `proxyConfiguration` (type: `object`):

Optional Apify Proxy or custom proxy settings used for website requests.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ],
  "maxPagesPerSite": 1,
  "checkLlmsTxt": true,
  "checkAiCrawlers": true,
  "checkAgentUsability": true,
  "maxConcurrency": 3,
  "timeoutSecs": 45,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `reports` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("ib4ngz/ai-readiness-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://apify.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("ib4ngz/ai-readiness-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://apify.com"
    }
  ]
}' |
apify call ib4ngz/ai-readiness-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ib4ngz/ai-readiness-checker"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/kdqsXVmfzWJzEgYf8/builds/RamngDuKGdF1fn2gr/openapi.json