# AI Search Readiness Audit - llms.txt, AI Bots & Schema (`that_red_bird/ai-search-readiness-audit`) Actor

Is your site visible to ChatGPT, Perplexity and Claude? Score it 0-100: robots.txt rules for GPTBot, ClaudeBot, PerplexityBot; llms.txt; JSON-LD; sitemap; text in raw HTML. Get a prioritized fix list plus a ready llms.txt, robots snippet and recommended JSON-LD.

- **URL**: https://apify.com/that_red_bird/ai-search-readiness-audit.md
- **Developed by:** [mohamed alaya](https://apify.com/that_red_bird) (community)
- **Categories:** AI, SEO tools, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 site analyzeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Search Readiness Audit - llms.txt, AI Bots & Schema

Can ChatGPT, Perplexity and Claude actually reach, read and cite your site? This Actor scores your site
**0-100 for AI-search readiness** from the outside, tells you the **exact fixes in priority order**, and
generates the files that close the biggest gaps: a ready **llms.txt**, a **robots.txt snippet** for AI search
crawlers, and **recommended JSON-LD** for pages that have none.
(Formerly "llms.txt Generator & Checker"; it also absorbs the JSON-LD generate/validate modes of the Structured Data Generator.)

### What is measured

| Area | Points | What is checked |
|---|---|---|
| AI crawler access | 25 | `robots.txt` rules for 12 AI bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent). Plus a live test: does your server answer those user agents with 200, or does a WAF return 403/429? |
| llms.txt | 20 | Exists, title and summary line, links, coverage of your sitemap URLs, stale links |
| Structured data | 20 | JSON-LD coverage across audited pages, validity of existing markup against Google's required/recommended properties, unparseable blocks |
| Extractable text | 15 | Words present in the **raw HTML** (most AI crawlers do not run JavaScript), `<main>`/`<h1>`, `lang`, JavaScript app-shell detection |
| Sitemap | 10 | sitemap.xml found, `lastmod` present, referenced in robots.txt |
| Canonical and meta | 10 | Self-referencing canonical, title + description, accidental noindex |

Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot...) are **listed but never penalised**: blocking them is a
legitimate business choice. Only blocking search/answer crawlers costs points, because those fetch pages to answer questions and cite sources.

### What you get

1. A **summary row**: score, grade, per-area breakdown, top fixes, bot-by-bot allowed/blocked table, existing llms.txt audit.
2. **Fix rows**, ranked: area, severity, points at stake, the issue, and the concrete fix.
3. **Page rows**: words in raw HTML, canonical, noindex, JSON-LD types found, existing-markup errors and warnings, and a recommended JSON-LD block where the page has none.
4. Files in the key-value store: `llms.txt`, optional `llms-full.txt`, `robots-ai-snippet.txt` (only if search bots are blocked), `recommended-jsonld.json`, `FIX_LIST.md`.

#### Sample output

```json
{
  "type": "summary", "site": "https://docs.apify.com", "mode": "full",
  "score": 70, "grade": "C",
  "headline": "AI search readiness 70/100 (C). Biggest gap: 20 of 20 audited pages have no JSON-LD structured data.",
  "areas": {
    "llmsTxt": { "score": 16, "max": 20, "exists": true, "coveragePercent": 11 },
    "aiCrawlerAccess": { "score": 25, "max": 25, "retrievalBotsAllowed": "6/6" },
    "structuredData": { "score": 0, "max": 20, "pagesWithJsonLd": "0/20" },
    "sitemap": { "score": 8, "max": 10, "urls": 4053 },
    "extractableText": { "score": 14, "max": 15, "pagesWithEnoughStaticText": "18/20" },
    "canonicalAndMeta": { "score": 7, "max": 10 }
  }
}
```

```json
{ "type": "fix", "priority": 1, "severity": "high", "pointsAtStake": 12, "area": "Structured data",
  "issue": "20 of 20 audited pages have no JSON-LD structured data.",
  "fix": "Add the recommended JSON-LD (key-value store record \"recommended-jsonld.json\" ...) to those pages." }
```

### Input

| Field | Default | What it does |
|---|---|---|
| `siteUrl` | `https://docs.apify.com` | Site (or a single page; the typed URL is always audited) |
| `mode` | `full` | `full` score + fixes + files, `llms-txt` only llms.txt, `json-ld` only structured data |
| `maxPages` | 20 | Pages sampled for page-level checks. Site-level checks cover the whole site |
| `useSitemap`, `includePatterns`, `excludePatterns` | true, empty | Page discovery filters |
| `testBotAccess` | true | Request the homepage as 5 AI user agents to detect WAF blocks |
| `generateFullText`, `maxFullTextChars` | false, 500000 | Also build `llms-full.txt` |
| `siteName`, `summary` | empty | Override the generated llms.txt title and summary line |

Run with `{}` to see the report on the default docs site.

### Pricing

Pay per event, no subscription:

| Event | Price | When |
|---|---|---|
| Site analyzed | $0.002 | per run |
| Page processed | $0.002 | per page fetched |
| Audit run | $0.002 | per scored audit (`full` mode) |
| File generated | $0.01 | per generated file (llms.txt, llms-full.txt, recommended-jsonld.json) |

A default 20-page audit costs about $0.07. Failed pages are not charged.

### Use cases

- **Marketing and SEO teams:** a checklist for "are we visible in ChatGPT and Perplexity answers?" with the fixes already written.
- **Developers shipping docs:** generate llms.txt / llms-full.txt and check JSON-LD in CI.
- **Agencies:** audit a prospect's site in one run for a sales call.
- **Before/after:** run again after the fixes and compare the score.

### Honest limits

- This measures **readiness**, not results. It cannot tell you whether any AI product currently cites your site.
- **llms.txt is a proposed convention.** Major AI providers have not confirmed they use it, so it is weighted at 20% of the score. Cheap to publish, but do not expect it to move rankings on its own.
- The bot test changes only the `User-Agent` header. Real crawlers come from published IP ranges, so a WAF may still treat them differently. Confirm in your CDN logs.
- Page-level scores are measured on the `maxPages` sample, not the entire site.
- It reads raw HTML and does not run JavaScript, which is the point of the extractable-text check.
- JSON-LD recommendations are drafts inferred from page content (type guesses below 40% confidence are not offered). Review them before publishing, and validate with Google's Rich Results Test.

### FAQ

**Should I allow GPTBot and ClaudeBot?** That is your call. They collect training data. The score only rewards allowing the search and answer crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot and the user-fetch bots).

**Which JSON-LD types can it generate?** Product, Article, LocalBusiness, Organization, FAQPage, HowTo, Recipe, Event, BreadcrumbList, JobPosting and VideoObject, validated against Google's required/recommended properties (ruleset 2025-06).

**Does it work for JavaScript-only sites?** It will tell you they are a problem: pages with little text in the raw HTML are flagged as app shells.

### Integrations

Works with Make, Zapier, n8n, the Apify API and scheduler, and Apify's MCP server so an AI agent can run the audit. Schedule it monthly and watch the score.

If it helped, a review on the Store page helps others find it. Questions or a site it handles badly? Open an issue on the Actor page.

# Actor input Schema

## `siteUrl` (type: `string`):

The site to score, e.g. https://example.com. Pages come from its sitemap.xml (also those named in robots.txt) or from following same-origin links. The exact URL you type is always audited, so you can also paste a single page.

## `mode` (type: `string`):

full: score 0-100 across six areas with a prioritized fix list, plus all generated files. llms-txt: only generate and check llms.txt. json-ld: only structured data, per-page JSON-LD check and a recommended block for pages that have none.

## `maxPages` (type: `integer`):

How many pages are fetched and checked. Page-level scores are measured on this sample, so raise it for large sites. Site-level checks (robots.txt, sitemap, llms.txt) always cover the whole site.

## `useSitemap` (type: `boolean`):

Pick pages from the sitemap first. Falls back to following links from the start URL when there is none.

## `includePatterns` (type: `array`):

Audit only URLs containing one of these substrings, e.g. /docs/, /blog/. Empty means all.

## `excludePatterns` (type: `array`):

Skip URLs containing any of these substrings, e.g. /tag/, /author/.

## `testBotAccess` (type: `boolean`):

Request your homepage with the User-Agent of OAI-SearchBot, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot and compare with a normal browser, to catch WAF or CDN rules that return 403/429 to them. It only changes the User-Agent header; real crawlers come from published IP ranges, so treat a pass as likely, not proof.

## `generateFullText` (type: `boolean`):

Besides llms.txt, build llms-full.txt: the markdown of every audited page in one file, capped by the size limit below. Needs more time on big sites.

## `maxFullTextChars` (type: `integer`):

Stop adding pages to llms-full.txt after this many characters.

## `siteName` (type: `string`):

Overrides the name used as the H1 of the generated llms.txt. Empty means it is taken from the first page title.

## `summary` (type: `string`):

Overrides the one-sentence blockquote summary of the generated llms.txt. Empty means the meta description of the first page that has one.

## `concurrency` (type: `integer`):

How many pages are fetched in parallel.

## `proxyConfiguration` (type: `object`):

Optional Apify proxy for sites that block datacenter IPs. Off by default.

## Actor input object example

```json
{
  "siteUrl": "https://docs.apify.com",
  "mode": "full",
  "maxPages": 20,
  "useSitemap": true,
  "includePatterns": [],
  "excludePatterns": [],
  "testBotAccess": true,
  "generateFullText": false,
  "maxFullTextChars": 500000,
  "siteName": "",
  "summary": "",
  "concurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `downloadCsv` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `count` (type: `string`):

No description

## `llmsTxt` (type: `string`):

No description

## `fixList` (type: `string`):

No description

## `recommendedJsonLd` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteUrl": "https://docs.apify.com",
    "mode": "full",
    "includePatterns": [],
    "excludePatterns": [],
    "siteName": "",
    "summary": "",
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("that_red_bird/ai-search-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "siteUrl": "https://docs.apify.com",
    "mode": "full",
    "includePatterns": [],
    "excludePatterns": [],
    "siteName": "",
    "summary": "",
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("that_red_bird/ai-search-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteUrl": "https://docs.apify.com",
  "mode": "full",
  "includePatterns": [],
  "excludePatterns": [],
  "siteName": "",
  "summary": "",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call that_red_bird/ai-search-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,that_red_bird/ai-search-readiness-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/C2iKcpGZuhFQk4lg9/builds/3IeHKzfU1mbYiKShp/openapi.json