# AI Readiness Checker for Website AEO & GEO (`muhammadafzal/ai-readiness-checker`) Actor

Audit websites for AI crawler access, llms.txt, structured data, answer-ready content, trust signals, scores, and prioritized fixes.

- **URL**: https://apify.com/muhammadafzal/ai-readiness-checker.md
- **Developed by:** [Muhammad Afzal](https://apify.com/muhammadafzal) (community)
- **Categories:** AI, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $6.00 / 1,000 page readiness audits

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Readiness Checker for Websites

Audit public website pages for AI crawler access, machine-readable structure, answer-ready content, trust signals, and technical discoverability. The Actor returns one transparent page report per dataset item, with a 0–100 score, letter grade, category breakdown, and prioritized recommendations.

The checker uses direct public HTML and site-owned files. It does not send content to an LLM, invent rankings, claim that a page will be cited, or require an external AI API key.

### What the AI readiness checker audits

| Area | Signals checked |
|---|---|
| AI crawler access | `robots.txt` policies for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, and Google-Extended |
| Discovery | Page `noindex`, sitemap presence, canonical URLs, and `llms.txt` presence |
| Machine readability | JSON-LD types, content-specific schema, organization schema, semantic `main` or `article` markup, and page language |
| Answer-ready content | Titles, descriptions, H1 usage, content depth, heading and paragraph structure, question-style headings, lists, and tables |
| Trust and citation | Author and date signals, links to About/Contact/Privacy/Terms/editorial pages, and external supporting links |
| Technical delivery | HTTP status, content type, initial HTML response time, redirects, and indexability |

Each result includes these useful fields:

- `score` and `grade`: the overall 0–100 assessment and A–F grade.
- `scoreBreakdown`: access (25), machine readability (20), content (25), trust (15), and technical delivery (15).
- `aiBotAccess`: the effective root-path robots policy for six named AI crawlers.
- `structuredDataTypes`: the unique schema.org `@type` values found in JSON-LD.
- `issues`: stable issue codes, severity, plain-language evidence, remediation, and score impact.
- Content and link counts for comparison across a site or over time.

### When to use it

Use this Actor for a fast baseline before an AEO/GEO program, technical content audit, site migration, structured-data project, or AI visibility investigation. Agencies can export the dataset for client reports. Product, content, and SEO teams can schedule the Actor and compare scores over time. AI agents can use the stable output fields to route remediation tasks.

This Actor is not a substitute for analytics, server-log analysis, Search Console, a live AI citation monitor, JavaScript rendering, accessibility testing, or legal advice. It audits public initial HTML. Content rendered only after client-side JavaScript may appear missing. A high score indicates that the checked signals are present; it does not guarantee indexing, retrieval, ranking, or citation by any AI system.

### Input

| Field | Default | Description |
|---|---:|---|
| `startUrls` | `https://example.com/` | One or more public HTTP(S) pages or sites to audit |
| `maxPages` | `5` | Hard cap from 1–50 delivered page audits and result events |
| `crawlLinks` | `true` | Follow same-site HTML links until the page cap |
| `respectRobotsTxt` | `true` | Skip discovered links disallowed for the wildcard crawler; starting URLs are still audited |
| `maxConcurrency` | `3` | Simultaneous page requests from 1–10 |
| `requestTimeoutSecs` | `30` | Per-request timeout from 5–60 seconds |
| `proxyConfiguration` | direct connection | Optional Apify or authorized proxy configuration |

Default input:

```json
{
  "startUrls": [{ "url": "https://example.com/" }],
  "maxPages": 5,
  "crawlLinks": true,
  "respectRobotsTxt": true,
  "maxConcurrency": 3,
  "requestTimeoutSecs": 30
}
```

For a single-page audit, set `crawlLinks` to `false` and `maxPages` to `1`. For a small site sample, keep discovery enabled and raise `maxPages`. The Actor deduplicates normalized URLs and only follows HTTP(S) links on the same hostname.

#### API example

```bash
curl -X POST "https://api.apify.com/v2/acts/YOUR_ACTOR_ID/runs?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://www.example.com/guide"}],
    "maxPages": 10,
    "crawlLinks": true
  }'
```

Do not put real tokens in shared scripts, logs, or public repositories. Prefer an authorization header or an environment secret in production.

### Output

The default dataset contains one homogeneous record per successfully audited page. Diagnostics and aggregates live in the `OUTPUT` and `SUMMARY` key-value records, so warnings never pollute the page dataset.

Compact result example:

```json
{
  "finalUrl": "https://example.com/guide",
  "indexable": true,
  "aiCrawlable": true,
  "robotsTxtFound": true,
  "sitemapFound": true,
  "llmsTxtFound": false,
  "aiBotAccess": {
    "gptBot": "allowed",
    "oaiSearchBot": "allowed",
    "chatGptUser": "allowed",
    "claudeBot": "allowed",
    "perplexityBot": "allowed",
    "googleExtended": "allowed"
  },
  "structuredDataTypes": ["Article", "Organization"],
  "score": 86,
  "grade": "B",
  "scoreBreakdown": {
    "access": 25,
    "machineReadability": 18,
    "content": 23,
    "trust": 12,
    "technical": 8
  },
  "issues": [
    {
      "code": "LLMS_TXT_MISSING",
      "category": "machineReadability",
      "severity": "info",
      "message": "No /llms.txt file was found.",
      "recommendation": "Consider publishing /llms.txt with a concise site summary and links to canonical documentation.",
      "scoreImpact": 2
    }
  ]
}
```

`OUTPUT` adds the aggregate `averageScore`, overall `grade`, repeated `topIssues`, delivered record count, failed request count, robots skips, warnings, and estimated event charges. Invalid input produces a truthful `REJECTED` summary and zero dataset items. If all page requests fail, the run fails without fabricating results.

### Pay-per-event pricing

| Event | Price | Trigger |
|---|---:|---|
| Actor start | $0.00005 | One automatic start event for up to 1 GB memory |
| Page readiness audit | $0.006 | One automatic event for each page record written to the default dataset |

A five-page run produces five page-audit events, estimated at **$0.030**, plus the start event. A one-page run produces one page-audit event, estimated at **$0.006**, plus the start event. Platform usage is intended to be included in event prices. The Actor never manually charges either synthetic event, and it does not charge a result event for a failed, skipped, or rejected page.

Your Apify maximum-charge setting remains the final spending control. The Actor also caps work with `maxPages`, bounds retries, limits concurrency, and reports the number of delivered dataset items in its terminal status and summary.

### Scoring and interpretation

The score is a deterministic checklist, not an opaque model judgment. Issues deduct from their category only, and category scores never fall below zero. Grades are A (90–100), B (80–89), C (70–79), D (60–69), and F (below 60).

Treat recommendations as a prioritized engineering backlog. Some signals are contextual: a deliberate crawler block is not inherently wrong, an author is not required on every transactional page, and `llms.txt` is an emerging convention rather than a universal requirement. The Actor therefore labels contextual findings as `info` and applies smaller deductions than hard blockers such as `noindex`, HTTP errors, or blocking every checked AI crawler.

### Reliability and limits

The Actor starts with low-cost direct HTML requests, retries transient page failures twice, honors a per-request timeout, and can use an authorized proxy when supplied. It does not defeat CAPTCHAs, authentication, paywalls, or other access controls. A site that serves challenge HTML or requires browser rendering may receive an incomplete audit; inspect the response fields and warnings before acting on the score.

`robots.txt`, `/sitemap.xml`, and `/llms.txt` are checked per origin and cached during the run. The crawler follows only same-host links, filters common document and media extensions, and respects wildcard robots rules for discovered links by default. Starting URLs are fetched because examining the requested public page is the core audit operation.

Use the results only for sites you are permitted to access. Keep crawl limits conservative, respect site policies, avoid personal or sensitive pages, and independently review changes before deployment.

# Actor input Schema

## `startUrls` (type: `array`):

Use this when you know the public pages or sites to audit. Enter absolute HTTP(S) URLs, for example https://example.com/. The default audits example.com; authenticated, private, and non-web URLs are not supported.

## `maxPages` (type: `integer`):

Use this to cap delivered page audits and dataset-item charges. Accepts 1–50; default 5. This is a hard page-result limit, not a depth setting.

## `crawlLinks` (type: `boolean`):

Use this to follow same-site HTML links from each starting page until the page limit. Default true. Disable it to audit only the supplied URLs.

## `respectRobotsTxt` (type: `boolean`):

Use this to skip discovered paths disallowed for the wildcard crawler. Default true. Starting URLs are still fetched so their readiness can be audited; this does not bypass authentication or access controls.

## `maxConcurrency` (type: `integer`):

Use this to control simultaneous page requests. Accepts 1–10; default 3. Lower it for fragile sites or strict rate limits.

## `requestTimeoutSecs` (type: `integer`):

Use this to bound each page, robots.txt, sitemap, and llms.txt request. Accepts 5–60 seconds; default 30. It is not the overall Actor timeout.

## `proxyConfiguration` (type: `object`):

Use this only when a public site limits datacenter traffic. Configure an Apify Proxy group or your own authorized proxy. Default uses the Actor's direct connection; do not provide stolen sessions or access credentials.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ],
  "maxPages": 5,
  "crawlLinks": true,
  "respectRobotsTxt": true,
  "maxConcurrency": 3,
  "requestTimeoutSecs": 30
}
```

# Actor output Schema

## `results` (type: `string`):

Schema-validated page-level AI readiness audits in the default dataset.

## `summary` (type: `string`):

Aggregate score, grade, top issues, warnings, delivery count, and estimated result charges.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("muhammadafzal/ai-readiness-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com/" }] }

# Run the Actor and wait for it to finish
run = client.actor("muhammadafzal/ai-readiness-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ]
}' |
apify call muhammadafzal/ai-readiness-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,muhammadafzal/ai-readiness-checker"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/RciSEc0wQf8olbHcS/builds/f21eUErYTgqBg9YmD/openapi.json