# AI Readiness Audit · llms.txt, AI Crawlers, Structured Data (`agentsgearbox/ai-readiness-audit`) Actor

Audit any website for how well AI search engines and agents can use it: AI crawler access in robots.txt, llms.txt validity, structured data, content readable without JavaScript, Markdown delivery and agent discovery files. Score 0-100 with a fix list, 1,000 sites per run. First audit a month free.

- **URL**: https://apify.com/agentsgearbox/ai-readiness-audit.md
- **Developed by:** [AgentsGearbox](https://apify.com/agentsgearbox) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $12.00 / 1,000 site auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Readiness Audit · llms.txt, AI crawler access, structured data, agent files

Can AI search engines and agents actually use your website? This actor checks what they see: whether ChatGPT, Claude, Perplexity, Gemini and Copilot crawlers are allowed in, whether an `llms.txt` exists and follows the spec, whether the page carries structured data and a real title and description, whether the content is there **without running JavaScript**, whether a Markdown version is served, and whether agent discovery files (A2A agent card, MCP server card, OpenAPI) are published. Every site gets a 0–100 score, category scores and a prioritised fix list with the evidence behind each finding. Up to 1,000 sites per run, about ten small requests per site, no browser. The first audit each month is free for every account.

Built by [agentsgearbox](https://agentsgearbox.com). Questions: support@agentsgearbox.com.

### Why this matters now

AI answers send readers before Google does for new sites, and they can only cite what they can read. A page that renders only in JavaScript is invisible to most AI fetchers; a robots.txt that blocks `OAI-SearchBot` or `PerplexityBot` removes you from AI answers, not from training; `llms.txt` is now checked by Chrome's Lighthouse and published by OpenAI, Anthropic and Google for their own docs. Agencies sell "GEO readiness" work on exactly these points; this actor is the measurement.

### What you get

One `audit` item per site:

```json
{
  "site": "docs.apify.com",
  "status": "OK",
  "score": 83,
  "grade": "B",
  "accessScore": 100,
  "guidanceScore": 100,
  "structureScore": 17,
  "readabilityScore": 100,
  "performanceScore": 100,
  "agentScore": null,
  "llmsTxt": "valid",
  "aiSearchBotsAllowed": "11/11",
  "jsOnly": false,
  "topFix": "Add JSON-LD for Organization or WebSite on the homepage and Product, Article or FAQPage where they apply; it is the most machine-readable description of what the page is.",
  "fixes": [
    { "check": "structured-data", "severity": "high", "fix": "Add JSON-LD for Organization or WebSite on the homepage…" },
    { "check": "canonical", "severity": "medium", "fix": "Add <link rel=\"canonical\"> so agents and engines keep one URL per page." }
  ],
  "checks": [
    { "id": "ai-search-access", "category": "access", "label": "AI answer engines allowed to read the homepage", "status": "pass", "value": "11/11", "evidence": "All evaluated answer-engine crawlers may fetch the homepage.", "fix": null, "weight": 6 },
    { "id": "llms-txt", "category": "guidance", "label": "llms.txt present and valid", "status": "pass", "value": "valid, 12 section(s), 467 link(s)", "evidence": "\"Apify documentation\" with 467 links, 5/5 sampled links resolve", "fix": null, "weight": 6 },
    { "id": "content-without-js", "category": "readability", "label": "Main content present without JavaScript", "status": "pass", "value": "10726 words in raw HTML", "evidence": "Text-to-HTML ratio 31.4%", "fix": null, "weight": 8 },
    { "id": "markdown", "category": "readability", "label": "Markdown version available", "status": "pass", "value": "content negotiation (Accept: text/markdown)", "evidence": "text/markdown; charset=utf-8", "fix": null, "weight": 2 }
  ],
  "facts": {
    "homepage": { "finalUrl": "https://docs.apify.com/", "status": 200, "redirects": 0, "ttfbMs": 250, "bytes": 184233, "botProtected": false },
    "robots": { "present": true, "sitemaps": ["https://docs.apify.com/sitemap.xml"], "contentSignals": [], "bots": { "GPTBot": { "allowed": true, "rule": null, "matchedGroup": "*" } } },
    "llmsTxt": { "present": true, "valid": true, "title": "Apify documentation", "sections": 12, "links": 467, "linksChecked": 5, "linksOk": 5, "fullPresent": true, "problems": [] },
    "sitemap": { "present": true, "kind": "urlset", "urls": 710, "declaredInRobots": true },
    "structure": { "title": "Apify Documentation", "description": "…", "canonical": null, "lang": "en", "h1Count": 1, "jsonLdTypes": [], "metaRobots": null },
    "readability": { "words": 10726, "htmlBytes": 184233, "textRatio": 31.4, "jsOnly": false, "spaShell": false },
    "markdown": { "negotiated": true, "indexMd": false },
    "agentFiles": { "/.well-known/agent-card.json": { "present": false, "status": 404 }, "/.well-known/mcp/server-card.json": { "present": false, "status": 404 } }
  },
  "auditedAt": "2026-10-17T15:40:12.118Z"
}
```

Plus one free `summary` item per run (sites audited, average score, grade distribution, the five most common failures) and a free `error` item for any site that could not be reached at all.

### The checks

| Category (weight) | Checks |
|---|---|
| **Access** (25) | robots.txt reachable and plain text; which AI answer-engine crawlers may read the homepage (`OAI-SearchBot`, `ChatGPT-User`, `Claude-SearchBot`, `Claude-User`, `PerplexityBot`, `Perplexity-User`, `Google-Extended`, `Googlebot`, `Bingbot`, `Applebot-Extended`, `DuckAssistBot`); homepage not `noindex`; `Content-Signal` declarations; `noai` tags (reported, not scored). Training crawlers (`GPTBot`, `ClaudeBot`, `CCBot`, `Bytespider`, `Amazonbot`, `meta-externalagent`) are reported as the owner's policy and never scored. |
| **Guidance** (20) | `llms.txt` present and valid to the v2 spec (H1, blockquote summary, H2 sections with `- [name](url): note` lists), up to five of its links verified; `llms-full.txt`; XML sitemap (index or urlset, URL count) and whether robots.txt declares it. |
| **Structure** (20) | title, meta description, canonical, `html lang`, exactly one H1, Open Graph, JSON-LD structured data (types listed, invalid blocks counted). |
| **Readability** (20) | main content present in the raw HTML without JavaScript (word count, text ratio, empty app-shell detection, "enable JavaScript" messages); Markdown version via `Accept: text/markdown` or `/index.md`. |
| **Performance** (10) | homepage reachable without a bot challenge, HTTPS, time to first byte, HTML weight. |
| **Agent files** (5) | A2A agent card (`/.well-known/agent-card.json`), MCP server card, agent-skills manifest, OpenAPI description, `security.txt`, TDM reservation file, `ai.txt`. Their absence is never scored against a site (fewer than 0.01% have them); their presence earns the bonus. |

Scoring: pass = full weight, warn = half, fail = none; categories with no scorable result are left out instead of counted as zero. Grades: A ≥ 90, B ≥ 75, C ≥ 60, D ≥ 40, F below.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | string\[] | required | Homepages or bare domains, up to 1,000. A URL with a path (e.g. `/docs/`) also looks for a path-level `llms.txt` first, as the spec allows. |
| `aiBots` | string\[] | 17 crawlers | User agents evaluated against robots.txt. |
| `checkLlmsLinks` | boolean | `true` | Fetch up to five links from llms.txt and count how many resolve. |
| `includeFixes` | boolean | `true` | Add the prioritised fix list. |
| `concurrency` | integer | `6` | Sites audited in parallel. |
| `timeoutSecs` | integer | `20` | Per-request timeout. |

### Pricing

**Try it free.** The first 1 audit each calendar month are free for every Apify account, with nothing to enter: the actor keeps count in a small key-value store in your account (`free-slice-ai-readiness-audit`) and charges only from the next one on. The free results are real results you keep; the counter resets on the first of each month.

Pay per event: **$0.02 per site audited** (`site-audit`). Summary and error records are free; a site that cannot be reached costs nothing. Apify plans above Free get an automatic discount on every event (10% to 40%).

| Run | Cost |
|---|---|
| Audit one site | $0.02 |
| Audit a 50-site client portfolio | $1.00 |
| Weekly audit of 1,000 domains for a prospect list | $20.00 a week |

A single site takes 2 to 5 seconds; 1,000 sites finish in a few minutes.

### Example tasks

Ready-made configurations you can open, copy and run as they are:

- [Audit a portfolio of websites for AI search and agent readiness](https://apify.com/agentsgearbox/ai-readiness-audit/examples/audit-a-client-portfolio-for-ai-readiness): six well-known sites scored side by side, from a JavaScript-only homepage graded D to fully readable sites graded A, each with its fix list.

### Run it from your code

**Node.js**

```js
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('agentsgearbox/ai-readiness-audit').call({ urls: ['https://example.com', 'competitor.com'] });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const a of items.filter((i) => i.type === 'audit')) console.log(a.site, a.score, a.grade, a.topFix);
```

**Python**

```python
from apify_client import ApifyClient
client = ApifyClient(token=os.environ["APIFY_TOKEN"])
run = client.actor("agentsgearbox/ai-readiness-audit").call(run_input={"urls": ["https://example.com"]})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item.get("type") == "audit":
        print(item["site"], item["score"], [f["fix"] for f in item["fixes"][:3]])
```

**cURL**

```bash
curl -s -X POST "https://api.apify.com/v2/acts/agentsgearbox~ai-readiness-audit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' -d '{"urls":["https://example.com"]}'
```

### For AI agents

Give your agent hands. Instead of fetching a dozen files and reasoning about robots syntax, an agent calls this actor and gets a verified audit back as JSON, priced per site. Auditing one site by hand means reading robots.txt rules for seventeen crawlers, validating llms.txt against the spec, parsing JSON-LD and measuring the page without JavaScript: twenty minutes for a specialist. This actor returns it in about three seconds for $0.02.

- **MCP, this actor only:** add `https://mcp.apify.com?tools=agentsgearbox/ai-readiness-audit` to Claude Desktop, Claude Code, Cursor or any MCP client (with your Apify token), then ask for "an AI readiness audit of example.com with the top three fixes".

* **MCP, all agentsgearbox actors:** `https://mcp.apify.com?tools=agentsgearbox/telegram-channel-scraper,agentsgearbox/google-play-reviews-scraper,agentsgearbox/substack-scraper,agentsgearbox/google-news-scraper,agentsgearbox/brand-name-generator,agentsgearbox/ai-readiness-audit,agentsgearbox/company-lookup,agentsgearbox/app-intelligence,agentsgearbox/app-store-reviews-scraper`

- **No Apify account?** This actor is eligible for Apify's agentic payments: an agent can buy a spend-capped token through [agi.apify.com](https://agi.apify.com) (x402, MPP or Skyfire) and call the API directly.
- Prices are per result and known in advance; unreachable sites come back as free error records instead of failed runs.

### Reading the results

- `jsOnly: true` is the single most important finding: the homepage HTML carries almost no text, so AI fetchers that do not run JavaScript see an empty page. Server-render or pre-render the pages you want cited.
- `aiSearchBotsAllowed` counts the answer-engine crawlers allowed to fetch the homepage. Blocking them keeps you out of AI answers; blocking training crawlers is a separate decision and does not affect the score.
- `llmsTxt` is `valid`, `present, with issues` (the problems are listed in `facts.llmsTxt.problems`) or `missing`.
- `status` is `OK`, `BOT_PROTECTED` (the homepage answered with a bot challenge; everything a plain client cannot read is marked `unknown`) or `HTTP_<code>`. Sites that cannot be audited are free `error` rows instead: `UNREACHABLE` (DNS, connection or timeout), `NOT_HTML` (the URL serves a PDF, text or JSON file, not a page) or `ERROR` (invalid URL, private or local host). Different spellings of the same page (`http://`, trailing slash, upper case, `www.`) are audited and charged once.
- Each check carries its `evidence` so you can verify a finding before acting on it.

### Limitations, stated plainly

- Measurements are taken from a European datacenter with a plain HTTPS client. Sites that serve different content to bots, or that pass bot challenges only for browsers, are reported as `BOT_PROTECTED` rather than guessed at.
- "Content without JavaScript" is measured on the raw HTML; pages that load text through JavaScript are treated as invisible, because that is how most AI fetchers treat them.
- Only the homepage (or the URL you give) is analysed for structure and readability; the audit is a site-level readiness check, not a crawl of every page.
- Discovery-file conventions for agents (A2A, MCP cards) are young and still changing; the paths checked are the current ones and are listed in the output so you can see exactly what was probed.

### Troubleshooting

| Symptom | Cause | Fix |
|---|---|---|
| `BOT_PROTECTED` | Cloudflare or similar challenged the request | Allow verified AI crawlers through, or audit a staging URL without protection |
| `robots.txt reachable: warn, HTML returned` | The server returns a web page instead of a text file at /robots.txt | Serve a plain-text robots.txt with HTTP 200 |
| `llms.txt present, with issues` | File exists but misses the H1, the link lists or has broken links | Follow the problems listed in `facts.llmsTxt.problems` |
| `agentScore: null` | No agent discovery files found | Expected for almost every site today; nothing to fix unless you expose an API or agent |

### More from agentsgearbox

- [Telegram Channel Scraper](https://apify.com/agentsgearbox/telegram-channel-scraper): public Telegram channels as JSON, no API key.
- [Google Play Reviews Scraper](https://apify.com/agentsgearbox/google-play-reviews-scraper): Google Play reviews with replies and versions, any market.
- [Substack Scraper](https://apify.com/agentsgearbox/substack-scraper): Substack posts with full text and comments, custom domains too.
- [Google News Scraper](https://apify.com/agentsgearbox/google-news-scraper): Google News with real publisher URLs and full text.
- [Brand & Business Name Generator](https://apify.com/agentsgearbox/brand-name-generator): brand names verified live on domains, handles and app stores.
- [Company Lookup](https://apify.com/agentsgearbox/company-lookup): one row per domain with legal identity and VAT validation, contacts, socials, tech stack, open jobs, apps, news and Wikidata facts.
- [App Intelligence](https://apify.com/agentsgearbox/app-intelligence): App Store and Google Play in one run: search, top charts, full app details and each app's twin on the other store.
- [App Store Reviews Scraper · iOS + Google Play, Any Country](https://apify.com/agentsgearbox/app-store-reviews-scraper): Reviews of any iOS or Android app from both stores in one run: stars, text, version, votes, developer replies, since-last-run checkpoint, rating trend per app..

All of them are pay-per-event, run without a browser, and are compared with their alternatives on [agentsgearbox.com](https://agentsgearbox.com/actors/).

### Support

Email support@agentsgearbox.com or use the issues tab. Compare this actor with alternatives on [agentsgearbox.com](https://agentsgearbox.com).

# Actor input Schema

## `urls` (type: `array`):

Homepage URLs or bare domains, one site per line (up to 1,000). Each site gets about ten small requests: robots.txt, llms.txt, sitemap, the homepage, a Markdown negotiation and the agent discovery files.

## `aiBots` (type: `array`):

User agents whose access to the homepage is evaluated against robots.txt. The default covers the answer engines (ChatGPT, Claude, Perplexity, Gemini, Copilot, Apple, DuckDuckGo) and the training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, Amazonbot, Meta). Only answer-engine access affects the score; training access is reported as the site owner's choice.

## `checkLlmsLinks` (type: `boolean`):

Fetches up to five of the links listed in llms.txt and reports how many resolve (adds up to five requests per site).

## `includeFixes` (type: `boolean`):

Adds a prioritised list of concrete fixes to every audit.

## `concurrency` (type: `integer`):

Lower it if you audit many pages of the same host.

## `timeoutSecs` (type: `integer`):

Maximum time to wait for each request (robots.txt, llms.txt, homepage, sitemap, discovery files).

## Actor input object example

```json
{
  "urls": [
    "https://agentsgearbox.com",
    "https://docs.apify.com",
    "https://llmstxt.org"
  ],
  "aiBots": [
    "OAI-SearchBot",
    "ChatGPT-User",
    "GPTBot",
    "Claude-SearchBot",
    "Claude-User",
    "ClaudeBot",
    "PerplexityBot",
    "Perplexity-User",
    "Google-Extended",
    "Googlebot",
    "Bingbot",
    "Applebot-Extended",
    "DuckAssistBot",
    "CCBot",
    "Bytespider",
    "Amazonbot",
    "meta-externalagent"
  ],
  "checkLlmsLinks": true,
  "includeFixes": true,
  "concurrency": 6,
  "timeoutSecs": 20
}
```

# Actor output Schema

## `results` (type: `string`):

One item per audited site (score, category scores, every check with evidence and fix), plus one free summary item.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://agentsgearbox.com",
        "https://docs.apify.com",
        "https://llmstxt.org"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("agentsgearbox/ai-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://agentsgearbox.com",
        "https://docs.apify.com",
        "https://llmstxt.org",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("agentsgearbox/ai-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://agentsgearbox.com",
    "https://docs.apify.com",
    "https://llmstxt.org"
  ]
}' |
apify call agentsgearbox/ai-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,agentsgearbox/ai-readiness-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ymMCRXrfOKJowYOzn/builds/L3NDeG2G08TtBUWlz/openapi.json