# AI Search Readiness Checker - AI Bots, llms.txt & Schema (`wrenfield-data/ai-search-readiness-checker`) Actor

Check if ChatGPT, Perplexity, Claude and Google's AI can read and cite your site: robots.txt rules for 17 AI bots, server-level blocking, llms.txt, sitemap, structured data and a 0-100 score with fixes. From $0.005 per site.

- **URL**: https://apify.com/wrenfield-data/ai-search-readiness-checker.md
- **Developed by:** [Wrenfield Data](https://apify.com/wrenfield-data) (community)
- **Categories:** SEO tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.50 / 1,000 site checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Search Readiness Checker: can ChatGPT, Perplexity, Claude and Google's AI read your site?

**AI Search Readiness Checker** audits a list of websites for the technical basics that decide whether AI search engines can fetch and cite them: **robots.txt rules for 17 AI and search bots**, **server-level blocking** of AI crawlers, **llms.txt**, sitemap, structured data and whether your content is in the raw HTML. Each site gets a **0-100 readiness score** and a ranked fix list. No browser, no login, no cookies.

- 🤖 **17 bots tested, split into two kinds.** Search and answer bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot and more) decide whether you can be cited. Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot and more) are reported but never scored, because blocking them is your choice.
- 🧱 **Catches blocks robots.txt cannot show.** Optionally requests the page with the published user agents of OAI-SearchBot, PerplexityBot and ClaudeBot and flags a 403 or 429 while a normal browser gets the page (typical of CDN bot rules).
- 🧾 **One row per site, plain-English fixes.** Every issue says what is wrong and how to fix it.
- 💸 **$5 per 1,000 sites** (from $0.0035 on higher Apify plans). Sites that cannot be fetched are free.
- 🆓 **Free Apify plan:** runs are limited to 20 sites per run. Any paid Apify plan removes the limit and adds volume discounts.

#### Wrenfield Search Data: keyword and AI-search visibility data, priced per result

- **Keyword data:** [Keyword Search Volume](https://apify.com/wrenfield-data/keyword-search-volume) · [Keyword Difficulty](https://apify.com/wrenfield-data/keyword-difficulty-checker) · [Keyword Research](https://apify.com/wrenfield-data/keyword-research-tool) · [Autocomplete Expander](https://apify.com/wrenfield-data/google-autocomplete-keyword-expander) · [Google Trends](https://apify.com/wrenfield-data/google-trends-scraper) · [People Also Ask](https://apify.com/wrenfield-data/people-also-ask-scraper) · [Competitor Keywords](https://apify.com/wrenfield-data/competitor-keywords) · [Keyword Gap](https://apify.com/wrenfield-data/keyword-gap-report)
- **AI search visibility:** [AI Search Visibility Tracker](https://apify.com/wrenfield-data/ai-search-visibility-tracker) · [Google AI Overview Scraper](https://apify.com/wrenfield-data/google-ai-overview-scraper) · [AI Answer Scraper](https://apify.com/wrenfield-data/ai-answer-scraper) · **AI Search Readiness Checker** · [Agency AI Visibility Reports](https://apify.com/wrenfield-data/agency-ai-visibility-reports)
- **Site and domain SEO:** [Domain Authority & Traffic](https://apify.com/wrenfield-data/domain-authority-traffic-checker) · [On-Page SEO Checker](https://apify.com/wrenfield-data/on-page-seo-checker) · [Bulk PageSpeed](https://apify.com/wrenfield-data/bulk-pagespeed-checker) · [SEO Toolkit](https://apify.com/wrenfield-data/seo-toolkit) · [SEO Agency Report](https://apify.com/wrenfield-data/seo-agency-report)

#### Sample output

| Site | Score | AI search bots blocked in robots.txt | llms.txt | Sitemap | Words in HTML |
|---|---|---|---|---|---|
| apify.com | 100 | none | yes | yes | 1,590 |
| stripe.com | 100 | none | yes | yes | 1,992 |
| hubspot.com | 98 | none | no | yes | 2,190 |
| github.com | 94 | none | yes | yes | 1,126 |
| bbc.com | 68 | OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User | no | yes | 2,700 |
| example.com | 66 | none | no | no | 25 |

<sub>Real rows from a test run on Oct 27, 2026, 6 of 41 fields shown. Sites change their robots.txt often; your results will differ. Two sites in the same test (nytimes.com, wikipedia.org) refused the request with HTTP 403, so they were reported as errors and not charged.</sub>

### What does it check?

| Check | What it looks at | Counts toward the score |
| --- | --- | --- |
| AI bot access | robots.txt rules for 17 user agents, for the homepage and the page you gave | Yes (search and answer bots only) |
| Whole-site block | `User-agent: *` with `Disallow: /` | Yes, critical |
| Server blocking | HTTP status when requesting the page as OAI-SearchBot, PerplexityBot and ClaudeBot | Yes |
| Indexability | `noindex` in the robots meta tag or the `X-Robots-Tag` header | Yes, critical |
| Content in raw HTML | Words of visible text in the first HTML response, script-heavy shells (very large pages, such as news homepages over 7 MB, are read in full up to 16 MB) | Yes |
| Structured data | JSON-LD types, Organization / LocalBusiness / WebSite, `sameAs` links | Yes |
| Sitemap | `Sitemap:` line in robots.txt or `/sitemap.xml` | Yes |
| Page basics | Title, meta description, canonical, H1, language, Open Graph | Yes |
| llms.txt | `/llms.txt` and `/llms-full.txt` exist and look valid | Low weight: a proposed convention, no major engine has confirmed it affects citations |

#### Bots tested

| User agent | Vendor | Kind | What it does |
| --- | --- | --- | --- |
| `OAI-SearchBot` | OpenAI | Search and answers | Builds the ChatGPT search index |
| `ChatGPT-User` | OpenAI | Search and answers | Fetches a page when a ChatGPT user asks about it |
| `PerplexityBot` | Perplexity | Search and answers | Builds the Perplexity answer index |
| `Perplexity-User` | Perplexity | Search and answers | Fetches a page when a Perplexity user asks about it |
| `Claude-SearchBot` | Anthropic | Search and answers | Builds the Claude search index |
| `Claude-User` | Anthropic | Search and answers | Fetches a page when a Claude user asks about it |
| `Googlebot` | Google | Search and answers | Google Search crawler; also feeds Google AI Overviews and AI Mode |
| `Bingbot` | Microsoft | Search and answers | Bing crawler; feeds Bing and Copilot answers |
| `DuckAssistBot` | DuckDuckGo | Search and answers | Fetches pages for DuckDuckGo AI answers |
| `GPTBot` | OpenAI | Model training | Collects content to train OpenAI models |
| `ClaudeBot` | Anthropic | Model training | Collects content to train Anthropic models |
| `Google-Extended` | Google | Model training | Control token for Gemini training and grounding; does not affect Google Search |
| `Applebot-Extended` | Apple | Model training | Control token for Apple AI training |
| `CCBot` | Common Crawl | Model training | Common Crawl dataset used to train many models |
| `Bytespider` | ByteDance | Model training | ByteDance crawler |
| `Meta-ExternalAgent` | Meta | Model training | Meta AI training crawler |
| `cohere-ai` | Cohere | Model training | Cohere crawler |

Vendors add and rename bots. The list was reviewed on Oct 27, 2026 against each vendor's published documentation.

#### How the score works

Every site starts at 100. Each critical issue costs 30 points, high 12, medium 6, low 2. Grades: A 90+, B 80+, C 70+, D 60+, F below. The score is a checklist of fixable technical blockers, not a prediction of rankings or citations.

### How to use it

1. Paste websites or pages, one per line, into **Websites or pages to check**. A domain checks the homepage; a full URL checks that page. robots.txt, llms.txt and the sitemap are always read from the site root.
2. Leave **Test server-level blocking** on to catch CDN and firewall bot rules.
3. Click **Start**. About 2-4 seconds per site; 1,000 sites finish in a few minutes.
4. Open the **Readiness results** table, or read the `issues` array on each row for the fixes. The run summary lists the most common issues across your list.

**For agencies:** run a client book or a list of prospects in one go, sort by score, and lead the pitch with the sites that block AI search bots. Pair it with the [AI Search Visibility Tracker](https://apify.com/wrenfield-data/ai-search-visibility-tracker) to show what the engines actually say today.

### Input example

```json
{
    "sites": ["apify.com", "https://www.bbc.com/news", "example.com"],
    "testServerBlocking": true
}
```

| Field | Description |
| --- | --- |
| `sites` | Websites or page URLs, one per line (up to 1,000). `https://` is added when missing. |
| `testServerBlocking` | Default `true`. Also requests the page as three AI crawlers to detect server-level blocking. |

### Output example

One row per site (41 fields; real output for bbc.com on Oct 27, 2026, shortened):

```json
{
    "site": "bbc.com",
    "status": "ok",
    "http_status": 200,
    "readiness_score": 68,
    "grade": "D",
    "search_bots_allowed": [
        "Claude-SearchBot",
        "Claude-User",
        "Googlebot",
        "Bingbot",
        "DuckAssistBot"
    ],
    "search_bots_blocked": [
        "OAI-SearchBot",
        "ChatGPT-User",
        "PerplexityBot",
        "Perplexity-User"
    ],
    "training_bots_blocked": [
        "GPTBot",
        "ClaudeBot",
        "Google-Extended",
        "Applebot-Extended",
        "CCBot",
        "Bytespider",
        "Meta-ExternalAgent",
        "cohere-ai"
    ],
    "firewall": {
        "OAI-SearchBot": 200,
        "PerplexityBot": 200,
        "ClaudeBot": 200
    },
    "llms_txt": false,
    "sitemap": true,
    "schema_types": [
        "WebPage",
        "NewsMediaOrganization"
    ],
    "words_in_html": 2700,
    "issues": [
        {
            "id": "search_bots_blocked",
            "severity": "high",
            "issue": "robots.txt blocks AI search bots: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User",
            "fix": "Allow these user agents if you want the site to be found and cited in AI answers (a blanket Disallow or a copied \"block AI\" template is the usual cause)."
        },
        {
            "id": "no_entity_schema",
            "severity": "medium",
            "issue": "No Organization, LocalBusiness or WebSite structured data",
            "fix": "Add JSON-LD naming the business, its URL, logo and \"sameAs\" profile links so engines can tie the brand to one entity."
        }
    ]
}
```

Export results as JSON, CSV, Excel or HTML, or read them through the API.

### Pricing

Pay per event, no subscription. **$0.005 per site checked** on the free Apify plan, lower on paid plans:

| Apify plan | Per site | Per 1,000 sites |
| --- | --- | --- |
| Free | $0.005 | $5.00 |
| Starter | $0.0045 | $4.50 |
| Scale | $0.004 | $4.00 |
| Business | $0.0035 | $3.50 |

Sites that cannot be fetched (DNS errors, blocked requests, timeouts) are never charged. Set a maximum cost per run in the run options and the Actor stops at that limit.

### AI search readiness API: REST, Python, JavaScript, n8n and MCP

Run it from code through the Apify API and read the rows back as JSON.

```bash
curl -X POST "https://api.apify.com/v2/acts/wrenfield-data~ai-search-readiness-checker/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sites": ["apify.com", "example.com"]}'
```

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("wrenfield-data/ai-search-readiness-checker").call(run_input={"sites": ["apify.com", "example.com"]})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row["site"], row["readiness_score"], row["search_bots_blocked"])
```

It also works in n8n, Make and Zapier through the Apify integrations, and in any MCP client through Apify's hosted MCP server.

### FAQ

**Does a high score mean ChatGPT or Perplexity will cite my site?**
No. The score measures technical access and clarity: whether the bots can fetch your pages and read what is on them. Being cited also depends on content quality, authority and what each engine decides for each question. A low score reliably means something blocks you; a high score means nothing technical does.

**Does allowing a bot in robots.txt guarantee it can crawl my site?**
No. A CDN or firewall can still block it. That is why the Actor also requests the page with three AI crawler user agents. Real crawlers may be verified by IP address, so a pass here is not proof and a block here is a strong warning.

**Should I block training bots?**
That is your decision. Blocking training bots (GPTBot, ClaudeBot, Google-Extended and others) does not affect whether search and answer bots can cite you, so the Actor reports them but does not score them. Note that Googlebot also feeds Google AI Overviews and AI Mode, so blocking it removes you from Google Search too.

**Is llms.txt required?**
No. It is a proposed convention for giving AI systems a Markdown map of your key pages. We report whether it exists and weigh it lightly, because no major AI engine has confirmed that it affects citations.

**Why was my site reported as an error and not charged?**
The site refused our request (HTTP 401, 403, 407 or 429), did not answer in time or returned a server error. Those checks are free.

**Does it check my private pages or log in anywhere?**
No. It requests public URLs only, never logs in, and refuses private network addresses.

**Is this affiliated with OpenAI, Anthropic, Perplexity, Google or Microsoft?**
No. All product names and bot names are trademarks of their owners.

### Feedback and support

Missing a bot, a check or a field? Open an issue on the **Issues** tab and include the run ID. We read every issue and reply there.

### Acceptable use

- This Actor is an independent tool. It is not affiliated with, endorsed by or sponsored by OpenAI, Anthropic, Perplexity, Google or Microsoft.
- It makes a small number of ordinary GET requests per site (page, robots.txt, llms.txt, sitemap and three test requests). Only check sites you are allowed to test.

# Actor input Schema

## `sites` (type: `array`):

One website or page per line (up to 1,000). <code>https://</code> is added when missing. A domain checks the homepage; a full URL checks that page. Site-wide files (robots.txt, llms.txt, sitemap) are always read from the site root.

## `testServerBlocking` (type: `boolean`):

Also requests the page with the published user-agent strings of OAI-SearchBot, PerplexityBot and ClaudeBot and reports whether the server answers 403 or 429 while a browser gets the page. Many CDN bot rules do this even when robots.txt allows the bots. Real crawlers may be verified by IP, so a pass is not proof.

## Actor input object example

```json
{
  "sites": [
    "apify.com",
    "hubspot.com",
    "example.com"
  ],
  "testServerBlocking": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `allFields` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sites": [
        "apify.com",
        "hubspot.com",
        "example.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("wrenfield-data/ai-search-readiness-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sites": [
        "apify.com",
        "hubspot.com",
        "example.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("wrenfield-data/ai-search-readiness-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sites": [
    "apify.com",
    "hubspot.com",
    "example.com"
  ]
}' |
apify call wrenfield-data/ai-search-readiness-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,wrenfield-data/ai-search-readiness-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5fPDncrsePyHod0gU/builds/AOczFu3vLziidqCWj/openapi.json