# AI Readiness Auditor (`muhammadafzal/ai-readiness-auditor`) Actor

Audit website crawlability, AI bot access, llms.txt, sitemaps, metadata, JSON-LD, content, and accessibility with prioritized page-level fixes.

- **URL**: https://apify.com/muhammadafzal/ai-readiness-auditor.md
- **Developed by:** [Muhammad Afzal](https://apify.com/muhammadafzal) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 ai readiness page audits

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Readiness Auditor

Audit public website pages for AI discoverability, crawlability, structured data, metadata, content quality, accessibility signals, and technical delivery. The Actor gives growth, SEO, content, and engineering teams a prioritized, machine-readable readiness report for every successfully audited page.

### What the Actor checks

The auditor uses direct HTTP retrieval first and performs deterministic checks—no API key or LLM subscription is required. Each page receives a score from 0 to 100, a letter grade, seven category scores, and stable-code findings with evidence and recommended fixes.

| Output | What it means |
|---|---|
| `overallScore`, `grade` | Weighted AI readiness score and A–F grade |
| `categoryScores` | Crawlability, AI discoverability, metadata, structured data, content, accessibility, and performance |
| `robotsAllowed`, `indexable` | Generic robots.txt access and robots-meta indexability |
| `aiBotAccess` | Explicit root policy for GPTBot, ChatGPT-User, Google-Extended, ClaudeBot, and PerplexityBot |
| `llmsTxtStatus`, `sitemapStatus` | Site-wide discovery-file availability |
| `structuredDataTypes` | schema.org `@type` values parsed from valid JSON-LD |
| `findings` | Prioritized critical, high, medium, low, and informational actions |
| `discoveredUrls` | Bounded same-origin links available for crawl continuation |

The `OUTPUT` key-value-store record contains the run outcome, delivered and charged counts, warnings, charge-cap status, and a concise summary of robots.txt, sitemap, llms.txt, and named AI-crawler policy for every origin.

### When to use it

Use this Actor to establish a website baseline before an AI visibility program, check whether important landing pages expose clear machine-readable entities, generate an implementation backlog, compare templates across a site, or monitor readiness after a redesign.

Do not use it as proof that a specific AI model will cite or rank a page. The score measures observable technical and content signals, not proprietary model behavior. It is also not a browser-rendering, accessibility-conformance, Core Web Vitals, security, or legal-compliance test. JavaScript-only content may appear thin because the Actor intentionally audits the server-delivered HTML that crawlers can retrieve cheaply and consistently.

### Input

| Field | Default | Description |
|---|---:|---|
| `startUrls` | Apify homepage | One or more public HTTP/HTTPS pages; private, local, reserved-IP, and authenticated URLs are rejected |
| `maxPages` | `5` | Maximum billable page reports across all origins, from 1 to 100 |
| `maxDepth` | `1` | Same-origin link depth from 0 to 3 |
| `respectRobotsTxt` | `true` | Skip generically disallowed URLs without producing billable page records |
| `includeSitemapUrls` | `true` | Fill the bounded queue with same-origin URLs found in XML sitemaps |
| `requestTimeoutSecs` | `20` | Per-request timeout for pages and discovery files |
| `proxyConfiguration` | direct | Optional Apify or custom proxy settings for permitted public access |

Minimal one-page audit:

```json
{
  "startUrls": [{ "url": "https://example.com/" }],
  "maxPages": 1,
  "maxDepth": 0,
  "includeSitemapUrls": false
}
```

Bounded site sample:

```json
{
  "startUrls": [{ "url": "https://example.com/" }],
  "maxPages": 25,
  "maxDepth": 2,
  "respectRobotsTxt": true,
  "includeSitemapUrls": true,
  "requestTimeoutSecs": 20,
  "proxyConfiguration": { "useApifyProxy": false }
}
```

API clients can start the Actor with the same JSON input, then read page records from the default dataset and the aggregate diagnostic from `OUTPUT` in the default key-value store. The output schema exposes direct links to both resources on the run page.

### Example result

```json
{
  "url": "https://example.com/guide",
  "finalUrl": "https://example.com/guide",
  "outcome": "data",
  "statusCode": 200,
  "overallScore": 82,
  "grade": "B",
  "categoryScores": {
    "crawlability": 100,
    "aiDiscoverability": 80,
    "metadata": 90,
    "structuredData": 80,
    "content": 75,
    "accessibility": 85,
    "performance": 90
  },
  "llmsTxtStatus": "present",
  "sitemapStatus": "present",
  "structuredDataTypes": ["Article", "BreadcrumbList"],
  "findings": [
    {
      "code": "FRESHNESS_SIGNAL_MISSING",
      "category": "content",
      "severity": "info",
      "message": "No machine-readable published or modified date was detected.",
      "recommendation": "Add time[datetime] or article date metadata when the content is time-sensitive.",
      "evidence": null
    }
  ]
}
```

### Pricing

The Actor uses transparent pay per event pricing. Platform usage is intended to be included rather than passed through separately.

| Event | Price |
|---|---:|
| Audit run start | $0.0001 per run |
| AI readiness page audit | $0.004 per delivered page report |

A one-page run costs about **$0.0041** in Actor events. A successful 10-page audit costs about **$0.0401**. Blocked, robots-disallowed, unreachable, non-HTML, and rejected pages do not create dataset records and therefore do not trigger page-audit events. Apify's maximum-charge setting is checked before page work, and the Actor stops cleanly when no further page event fits the budget.

### Reliability and limitations

The Actor retrieves server-delivered HTML plus `/robots.txt`, declared or conventional XML sitemaps, and `/llms.txt`. Requests use realistic headers, safe redirect validation, bounded retries for rate limits and transient server errors, a 5 MB response ceiling, and a per-request timeout. Public redirects are revalidated to prevent access to local or private infrastructure. One failed site does not discard valid reports from other URLs; diagnostics remain in `OUTPUT`.

Scoring is intentionally transparent and deterministic, but it is opinionated. Missing llms.txt is a low-severity opportunity, not a guarantee of poor AI visibility. Named crawler rules are reported as allowed, blocked, or unspecified without asserting that every vendor still uses the same crawler identity. Response time is origin-observation latency, not a field measurement. The Actor does not solve CAPTCHAs, bypass authentication or paywalls, or defeat access controls.

### Responsible use

Audit only websites you are authorized to assess and follow their terms, robots policy, and applicable law. Page content and metadata can contain personal or sensitive information; configure Apify dataset retention and access controls appropriately. The Actor does not publish, republish, or train on retrieved content—it extracts a limited set of technical signals and short evidence values for the requested audit.

For support, include the run ID, a non-sensitive example URL, and the unexpected finding code. Never include passwords, tokens, cookies, or private proxy URLs in an issue.

# Actor input Schema

## `startUrls` (type: `array`):

Use this for public HTTP or HTTPS pages to audit. Enter one or more absolute URLs, for example https://www.apify.com/; default is a live public homepage, not a private network or authenticated page.

## `maxPages` (type: `integer`):

Use this to cap billable page-audit records across all websites. Accepts 1–100; default 5. This is not a request retry limit.

## `maxDepth` (type: `integer`):

Use this to follow same-origin HTML links from each start page. Accepts 0–3; default 1. Set 0 to audit only supplied and sitemap-selected pages.

## `respectRobotsTxt` (type: `boolean`):

Use this to skip URLs disallowed for generic crawlers. Enabled by default; disabling it does not bypass authentication, CAPTCHAs, paywalls, or access controls.

## `includeSitemapUrls` (type: `boolean`):

Use this to fill the bounded audit queue with same-origin URLs from a declared or conventional XML sitemap. Enabled by default; it never exceeds Maximum pages.

## `requestTimeoutSecs` (type: `integer`):

Use this to bound each page, robots.txt, sitemap, and llms.txt request. Accepts 5–60 seconds; default 20. This is per request, not the whole run.

## `proxyConfiguration` (type: `object`):

Use this only when a public website blocks direct requests or requires consistent geography. Direct access is the default; a proxy does not authorize bypassing access controls.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ],
  "maxPages": 5,
  "maxDepth": 1,
  "respectRobotsTxt": true,
  "includeSitemapUrls": true,
  "requestTimeoutSecs": 20,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `audits` (type: `string`):

Schema-validated AI readiness reports, one billable item per successfully audited HTML page.

## `summary` (type: `string`):

Outcome, site-wide robots/sitemap/llms.txt signals, billing counts, and warnings.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("muhammadafzal/ai-readiness-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com/" }] }

# Run the Actor and wait for it to finish
run = client.actor("muhammadafzal/ai-readiness-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com/"
    }
  ]
}' |
apify call muhammadafzal/ai-readiness-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,muhammadafzal/ai-readiness-auditor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4MkAfdW90iwb78QRw/builds/mtJnzNfi6PpeBNPV9/openapi.json