# AI Readiness Checker: GPTBot, ClaudeBot, llms.txt & AI SEO (`arnaud437/ai-readiness-checker`) Actor

Check if AI search can read and cite a website: robots.txt rules for 17 AI bots (GPTBot, ClaudeBot, PerplexityBot...), llms.txt, schema.org, a score and fixes.

- **URL**: https://apify.com/arnaud437/ai-readiness-checker.md
- **Developed by:** [Open Data Tools](https://apify.com/arnaud437) (community)
- **Categories:** AI, SEO tools, Developer tools
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.40 / 1,000 website checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Readiness Checker: GPTBot, ClaudeBot, llms.txt & AI SEO

Give a list of **websites**, see if **ChatGPT, Claude, Perplexity, Gemini** and other AI assistants can **read and cite** each one. The checker reads the **robots.txt rules for 17 AI bots** (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended…), looks for **llms.txt** and **llms-full.txt**, a **sitemap**, **schema.org** structured data and the **text readable without JavaScript**, then gives a **score out of 100**, a **grade** and the **fixes in plain words**. Fast and cheap: **$2.00 per 1,000 websites**.

### What you get

One row per website:

- **Each AI bot**: company, purpose (**training**, **search** or **user** request), **allowed or blocked**, and whether robots.txt names it.
- **Blocked bots**, split into **AI search and assistants** (the ones that cite you) and **training** crawlers.
- **llms.txt**: present or not, its title and number of links; **llms-full.txt**.
- **Sitemap**, **schema.org types**, **title**, **meta description**, **words readable without JavaScript**, **noai** meta tags.
- **Score**, **grade** and **fixes**, worst first.

### Use cases

- **AI SEO (GEO)**: find why a site is never cited by ChatGPT, Claude or Perplexity.
- **Agencies**: audit clients and prospects in bulk and sell the fixes.
- **Publishers and brands**: check that robots.txt blocks AI training but still lets AI search cite you.
- **Market research**: measure how many sites of a sector block AI bots or publish a llms.txt.

### How to use

- **Without changing anything**: apify.com, nytimes.com and stripe.com.
- **Websites**: one per line, up to 10,000 per run. Without `http://` or `https://`, the check starts from `https://`.

### Input example

```json
{
  "websites": ["apify.com", "nytimes.com", "stripe.com"]
}
```

### Output example

```json
{
  "recordType": "website",
  "id": "https://www.anthropic.com/",
  "url": "https://www.anthropic.com/",
  "input": "www.anthropic.com",
  "site": "www.anthropic.com",
  "robotsTxtFound": true,
  "allowedBotCount": 17,
  "blockedBots": [],
  "searchBotsBlocked": [],
  "trainingBotsBlocked": [],
  "blocksAllAi": false,
  "aiBots": [
    {
      "bot": "GPTBot",
      "company": "OpenAI",
      "purpose": "training",
      "allowed": true,
      "namedInRobots": false
    }
  ],
  "hasLlmsTxt": false,
  "llmsTxtTitle": null,
  "llmsTxtLinkCount": 0,
  "hasLlmsFullTxt": false,
  "hasSitemap": true,
  "homeReadable": true,
  "title": "Home \\ Anthropic",
  "metaDescription": "Anthropic is an AI safety and research company that's working to build reliable, interpretable, and steerable AI systems.",
  "structuredDataTypes": [],
  "serverRenderedWords": 310,
  "noAiMeta": false,
  "score": 75,
  "grade": "B",
  "issues": [
    "No /llms.txt: AI assistants get no summary or map of the site",
    "No structured data (schema.org JSON-LD) to describe the business to AI"
  ],
  "status": "ok",
  "sourceName": "robots.txt, llms.txt and home page of www.anthropic.com",
  "sourceUrl": "https://www.anthropic.com/robots.txt",
  "licence": "Public files a site publishes for robots; page text is not kept",
  "sourceUpdatedAt": null,
  "actorVersion": "0.1.1",
  "retrievedAt": "2026-11-19T12:41:05.512Z"
}
```

`aiBots` lists the 17 bots; one is shown here. On nytimes.com, 15 of the 17 bots are blocked, AI search included, and the score is 30 (F).

### How the score works

The score starts at 100 and each issue removes points: home page not readable 30; AI search or assistant bots blocked 25; fewer than 150 words without JavaScript 20; no llms.txt 15; no structured data, no sitemap or no title 10; no meta description or a noai tag 5. Blocking **training** bots only is a choice, not a fault: it is reported but costs no points. Grades: A from 90, B from 75, C from 60, D from 40, F below.

### Pricing

Pay per event: **$2.00 per 1,000 websites checked** (`website-checked`, $0.002 each), plus Apify's start fee of $0.00005 per run. A website that does not answer and text that is not a website give a **free** error row.

| Websites        | Price |
| --------------- | ----- |
| 100 websites    | $0.20 |
| 1,000 websites  | $2.00 |
| 10,000 websites | $20   |

### Data source

Files each site publishes for robots: robots.txt, llms.txt, llms-full.txt, sitemap.xml, and the HTML of the home page. Robots.txt rules are read the way each AI bot reads them (its own group, else the `*` group). Five requests per site at most, robots.txt respected.

### Is it legal?

These files are published so that robots read them. The checker sends a few ordinary requests per site, waits between them, follows robots.txt, never visits private addresses and keeps no page text.

### FAQ

- **What is llms.txt?** A Markdown file at the root of a site that tells AI assistants what the site is and where its key pages are, like a sitemap written for language models.
- **Why is "the home page could not be read"?** The site blocks robots (robots.txt or a bot protection) or answers with an error: AI crawlers likely meet the same wall.
- **Does it run JavaScript?** No, on purpose: most AI crawlers do not, so the words count shows what they actually see.

### Related Actors

- [Website to Markdown](https://apify.com/arnaud437/website-to-markdown): Fast Crawler for LLM & RAG (No Browser)
- [AI Model Prices](https://apify.com/arnaud437/ai-model-prices): Compare 400+ LLM Prices and Features
- [llms.txt Generator](https://apify.com/arnaud437/llms-txt-generator): AI-Ready Site Map for Any Website
- [Structured Data Extractor](https://apify.com/arnaud437/structured-data-extractor): Schema.org, JSON-LD & Open Graph
- [Sitemap Extractor](https://apify.com/arnaud437/sitemap-extractor): All URLs of a Site, Lastmod & Priority

Every Actor here also works as a tool for AI agents (Claude, ChatGPT, Cursor…) through [Apify’s MCP server](https://mcp.apify.com).

### Changelog

See [CHANGELOG.md](CHANGELOG.md).

### En français

#### Votre site est-il lisible par ChatGPT, Claude et Perplexity ?

Donnez des sites, obtenez pour chacun l'accès de 17 robots d'IA d'après robots.txt, la présence d'un llms.txt, du sitemap et des données structurées, le texte lisible sans JavaScript, une note sur 100 et les corrections en clair.

#### Prix

2,00 $ les 1 000 sites. Les lignes d'erreur sont gratuites.

#### Source et légalité

Les fichiers que chaque site publie pour les robots et sa page d'accueil. Quelques requêtes par site, robots.txt respecté, aucun texte gardé.

# Changelog

This Actor's version history is a separate document: https://apify.com/arnaud437/ai-readiness-checker/changelog.md

# Actor input Schema

## `websites` (type: `array`):

One website per line (apify.com or https://www.apify.com). Its robots.txt, llms.txt, sitemap and home page are checked. Up to 10,000 per run; each site once.

## Actor input object example

```json
{
  "websites": [
    "apify.com",
    "nytimes.com",
    "stripe.com"
  ]
}
```

# Actor output Schema

## `results` (type: `string`):

One row per website: access of 17 AI bots, llms.txt, sitemap, structured data, text readable without JavaScript, score and fixes.

## `summary` (type: `string`):

Websites checked, error rows, and dates of the run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websites": [
        "apify.com",
        "nytimes.com",
        "stripe.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("arnaud437/ai-readiness-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "websites": [
        "apify.com",
        "nytimes.com",
        "stripe.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("arnaud437/ai-readiness-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websites": [
    "apify.com",
    "nytimes.com",
    "stripe.com"
  ]
}' |
apify call arnaud437/ai-readiness-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,arnaud437/ai-readiness-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/dDGTQKFb5RcDeW8ef/builds/c2SVdglgz2oMF4WFG/openapi.json
