# AI Search Readiness Audit (`frank_conifer_af4/ai-search-readiness-audit`) Actor

Audit how ready a website is for AI search (GEO): per-bot robots.txt access for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and more, llms.txt, structured data, sitemap freshness, a 0-100 score and prioritized fixes.

- **URL**: https://apify.com/frank_conifer_af4/ai-search-readiness-audit.md
- **Developed by:** [Juddroid](https://apify.com/frank_conifer_af4) (community)
- **Categories:** SEO tools, Automation
- **Stats:** 1 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 website auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Search Readiness Audit

Find out in seconds whether ChatGPT, Claude, Perplexity, Google AI Overviews and Bing Copilot can reach, read and cite a website, with a 0-100 score and a prioritized fix list for every domain.

### What does AI Search Readiness Audit do?

AI Search Readiness Audit checks the technical side of GEO (generative engine optimization): whether a website can show up as a cited source in AI answers. For each domain or URL it parses robots.txt the way crawlers do and gives a verdict for 18 AI and search bots. It also validates llms.txt and checks how much content is visible without JavaScript. Then it inspects indexing and snippet directives, structured data and sitemap freshness. Everything is rolled into one row per website: an AI readiness score, a letter grade, a transparent score breakdown and plain-English fixes that you can hand to a developer or a client.

### Why use it

- **Answers the question clients actually ask.** "Can ChatGPT and Perplexity see us?" gets a direct answer per crawler, with the exact robots.txt line responsible.
- **Separates training from search.** Blocking AI training (GPTBot, Google-Extended, CCBot...) is a legitimate business choice and is reported neutrally. Only blocks that stop you from being *found and cited* cost points.
- **Real robots.txt logic, not keyword matching.** It handles user-agent groups, merged groups, `*` fallback, longest-match precedence, `Allow` vs `Disallow` ties, `*` and `$` wildcards, Apple's documented Googlebot fallback, and Cloudflare-style `Content-Signal` lines.
- **Catches the silent killers.** It finds client-rendered single-page apps that AI crawlers see as empty pages, `noindex`/`nosnippet` in meta tags or `X-Robots-Tag` headers, robots.txt files that return 5xx (crawlers then treat the whole site as blocked), and firewalls that refuse automated requests.
- **One row per domain, ready for reporting.** It is built for agency audits, prospect lists and monitoring portfolios of sites. It replaces separate robots.txt, sitemap and structured-data checkers.
- **Prioritized fixes with copy-paste snippets**, for example the exact `User-agent` group to add so ChatGPT search can cite you while GPTBot stays blocked.

### GEO and AI search readiness, explained for marketers

AI assistants answer questions by searching the web, reading pages and citing sources. To be cited, a page has to pass three gates:

1. **Access:** the assistant's crawler must be allowed by robots.txt and not blocked by a firewall.
2. **Readability:** the crawler must find real text in the HTML. Published crawler analyses show that most AI crawlers, including those from OpenAI, Anthropic and Perplexity, fetch HTML without running JavaScript, so content injected client-side is invisible to them. Googlebot does render JavaScript.
3. **Eligibility and understanding:** the page must be indexable and allowed to be quoted (no `noindex`, `nosnippet` or tiny `max-snippet`). Structured data, a sitemap with honest `lastmod` dates and, optionally, an llms.txt file help machines understand and prioritise it. Google states that a page must be indexed and eligible for a snippet to appear as a supporting link in AI Overviews and AI Mode.

#### Training vs. search vs. user-triggered crawlers

AI companies run different bots for different jobs, and robots.txt lets you treat them independently:

| Purpose | Bots checked | What blocking does | Affects score? |
|---|---|---|---|
| **Training** | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot | Opts your content out of future model training. Google-Extended and Applebot-Extended are control tokens and do not remove you from Google Search or Siri. | No, reported as policy |
| **Search / answer** | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot, DuckAssistBot | Removes you from ChatGPT search, Claude search, Perplexity, Google Search and AI Overviews, Bing/Copilot, Siri/Spotlight and DuckAssist results and citations. | Yes, weighted by reach |
| **User-triggered** | ChatGPT-User, Claude-User, Perplexity-User | Stops the assistant from opening your page when a user asks about it or pastes the link. OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says Perplexity-User "generally ignores" it, so treat these rules as a statement of policy. | Yes, small weight |

A common, sensible setup is to block training crawlers and allow search and user-triggered crawlers. This Actor grades that setup as fully ready.

### What data you get

| Field | Description |
|---|---|
| `aiReadinessScore`, `grade` | 0-100 score and A-F grade (A ≥ 90, B ≥ 80, C ≥ 70, D ≥ 60) |
| `summary` | One-sentence verdict: policy, llms.txt, number of high-priority fixes |
| `scoreBreakdown` | Points per category with every individual check listed (see scoring below) |
| `scoreCaps`, `scoreBeforeCaps` | Critical blockers that cap the score, e.g. a client-only SPA or every search crawler blocked |
| `fixes[]` | Prioritized `{priority, category, issue, recommendation, pointsLost}` items in plain English |
| `blockedSearchBots`, `blockedTrainingBots`, `blockedUserFetchers` | Bots blocked site-wide, by purpose |
| `aiCrawlers[]` | Per bot: operator, purpose, matched robots.txt group, `allowed` / `partially-blocked` / `blocked` for `/` and for the input URL path, blocking rule and line number, crawl-delay |
| `aiPolicy` | Neutral description of the AI policy, including Content-Signal and noai notes |
| `robotsTxt` | Status, groups, user agents, AI bots with their own rules, sitemap declarations, crawl-delays, Content-Signal |
| `llmsTxt`, `llmsFullTxt` | Presence, H1 title, summary, sections, link count, broken-looking links, size |
| `pages.homepage`, `pages.inputPage` | HTTP status, redirect chain, response time, title/description length, `lang`, canonical, meta robots + `X-Robots-Tag` (noindex, nofollow, nosnippet, max-snippet, noai, noimageai), Open Graph, JSON-LD types and errors, words visible without JavaScript, SPA signals |
| `sitemap` | Where it was found, index or urlset, child sitemaps sampled, URL count (sampled and estimated), lastmod coverage, share older than 180 days, newest lastmod, gzip support |
| `partialAudit`, `status`, `error` | Whether some categories could not be evaluated; failure reason for uncharged items |

#### How the score works

| Category | Points | What is checked |
|---|---|---|
| Crawler access | 30 | robots.txt verdicts for search/answer bots (Googlebot 7, Bingbot 5, OAI-SearchBot 4, Claude-SearchBot 3, PerplexityBot 3, Applebot 2, DuckAssistBot 1) and user-triggered fetchers (5 total). Training bots: 0. |
| Content accessibility | 25 | HTTP 200, words visible without JavaScript, response time, redirects, title, meta description, `lang` |
| Structured data | 15 | JSON-LD present and valid, `@type` on every entity, Organization and WebSite, page-type schema, Open Graph |
| Discoverability | 15 | Sitemap found and declared in robots.txt, lastmod coverage and freshness, llms.txt, canonical |
| Snippet controls | 15 | No noindex, nosnippet, restrictive max-snippet or page-level nofollow; Content-Signal does not opt out of AI answers |

Critical blockers cap the total: an unreachable robots.txt or every search crawler blocked (max 40), no page returning 200 (max 50), Googlebot or Bingbot blocked or a noindexed page (max 60), a client-rendered SPA (max 65). If a site refuses our page requests (for example HTTP 403 from bot protection), the page-level categories are marked "not evaluated" and the score is rescaled to the categories that could be checked.

### How to use

1. Open the Actor and paste your domains or URLs into **Websites or URLs to audit**. Bare domains (`example.com`) and deep URLs (`https://example.com/pricing`) both work.
2. Keep **Check llms.txt** on, and adjust **Max child sitemaps to sample** if you want more precise sitemap statistics for large sites.
3. Click **Start**. Each website typically takes 2-8 seconds.
4. Open the **Overview** tab for scores and top fixes, **Fixes** for the full prioritized list, or **AI crawlers** for the per-bot robots.txt verdicts. Export as CSV, Excel or JSON.

```json
{
  "urls": [
    "example.com",
    "https://www.example.org/blog/how-we-work",
    "https://shop.example.net"
  ],
  "checkLlmsTxt": true,
  "maxSitemapsToSample": 3,
  "maxConcurrency": 5
}
```

### Output example

Trimmed result from a test run in October 2026:

```json
{
  "input": "https://www.theguardian.com",
  "domain": "theguardian.com",
  "finalUrl": "https://www.theguardian.com/international",
  "status": "ok",
  "aiReadinessScore": 74,
  "grade": "C",
  "summary": "74/100 (C). Opts out of 6 of 8 AI training crawlers/tokens (ClaudeBot, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot). Blocks search/answer crawlers: Claude-SearchBot, PerplexityBot, DuckAssistBot. Disallows user-triggered fetchers: Claude-User. No llms.txt. 1 high-priority fix.",
  "blockedSearchBots": ["Claude-SearchBot", "PerplexityBot", "DuckAssistBot"],
  "blockedTrainingBots": ["ClaudeBot", "Applebot-Extended", "CCBot", "Bytespider", "Meta-ExternalAgent", "Amazonbot"],
  "blockedUserFetchers": ["Claude-User"],
  "hasLlmsTxt": false,
  "serverRenderedWords": 3301,
  "looksLikeClientOnlySpa": false,
  "sitemapUrlCount": 504,
  "scoreBreakdown": {
    "crawlerAccess": { "score": 22, "max": 30 },
    "contentAccessibility": { "score": 25, "max": 25 },
    "structuredData": { "score": 0, "max": 15 },
    "discoverability": { "score": 12, "max": 15 },
    "snippetControls": { "score": 15, "max": 15 }
  },
  "fixes": [
    {
      "priority": "high",
      "category": "crawlerAccess",
      "issue": "Claude-SearchBot, Claude-User, PerplexityBot and DuckAssistBot are blocked by `Disallow: /` under the group that names them (robots.txt line 102). Claude web search, Perplexity and DuckAssist cannot crawl or cite your pages; Claude cannot open your pages when a user asks about them.",
      "recommendation": "Remove these user agents from the group that names them (or give them their own group with `Allow: /`) if you want to appear in Claude web search, Claude, Perplexity and DuckAssist. Blocking the matching training crawler is a separate, independent setting.",
      "pointsLost": 8.5
    },
    {
      "priority": "medium",
      "category": "structuredData",
      "issue": "No structured data (JSON-LD or microdata) found.",
      "recommendation": "Add JSON-LD to the homepage: an Organization (name, url, logo, sameAs links) and a WebSite entity. On content pages add the matching type (Article, Product, FAQPage, LocalBusiness...). It gives AI systems unambiguous facts about who you are and what each page is.",
      "pointsLost": 5
    }
  ],
  "aiCrawlers": [
    {
      "bot": "OAI-SearchBot",
      "operator": "OpenAI",
      "purpose": "search",
      "matchedGroup": "*",
      "rootAccess": "partially-blocked",
      "typicalPageAllowed": true,
      "blockingRule": null,
      "note": "Allowed under `User-agent: *`, except 48 disallowed path pattern(s) such as `/sendarticle/`, `/Users/`, `/users/`."
    },
    {
      "bot": "Claude-SearchBot",
      "operator": "Anthropic",
      "purpose": "search",
      "matchedGroup": "Claude-SearchBot",
      "rootAccess": "blocked",
      "blockingRule": "Disallow: /",
      "blockingLine": 102,
      "note": "Blocked site-wide by `Disallow: /` under `User-agent: Claude-SearchBot`."
    }
  ],
  "sitemap": { "found": true, "type": "urlset", "urlCountSampled": 504, "lastmodCoveragePct": 100, "newestLastmod": "2026-10-16T08:34:01.000Z" },
  "checkedAt": "2026-10-16T08:35:04.588Z"
}
```

`partially-blocked` means the homepage is allowed but some paths are disallowed (normal housekeeping such as `/admin/`) or, the other way round, the homepage is blocked but some paths are allowlisted. `typicalPageAllowed: false` flags the second case: only a few allowlisted paths are open, and that costs points.

### Pricing

This Actor uses pay-per-event pricing: **$0.01 per audited website or URL**.

- 100 websites: $1
- 1,000 websites: $10
- 10,000 websites: $100

Not charged: invalid inputs, domains that do not resolve or cannot be reached, sites that refuse every request (bot protection, status `blocked`), and duplicate inputs. Each unique URL is one audit, so `example.com` and `example.com/pricing` are two audits. You can set a maximum cost per run, and the Actor stops cleanly when it is reached.

### Integrations and scheduling

- **Schedules:** run weekly or monthly on a client portfolio to catch robots.txt changes, for example a CMS update or a CDN "block AI bots" toggle that silently blocks ChatGPT search.
- **Webhooks and integrations:** send results to Slack, Google Sheets, Make, Zapier or n8n when a run finishes, or alert when a grade drops.
- **API:** start runs and fetch the dataset (`?view=overview` for the compact table) from any language via the Apify API or client libraries.
- **AI agents and MCP:** the Actor works as a tool through the Apify MCP server, so an AI agent can audit a prospect's site and draft recommendations from the `fixes` list.

### FAQ and limitations

**Does blocking GPTBot lower my score?** No. Training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot) are reported under `blockedTrainingBots` as a policy choice. Only search/answer crawlers and user-triggered fetchers count.

**Is llms.txt required?** No. It is an emerging, proposed convention (llmstxt.org) and AI providers' support for it is limited and largely unconfirmed. It is worth 3 of 100 points, and a missing file is always a low-priority fix.

**Why does my site get a 403 / "partial audit"?** Our requests look like a normal browser. If a firewall or bot-protection layer refuses them, AI crawlers may be refused too unless they are allowlisted. Page-level checks are then skipped and the score is rescaled. If every request is refused, nothing is charged.

**Why does a JavaScript site score low?** The Actor reads the HTML as delivered, without running JavaScript, which is how most AI crawlers read pages. If the HTML is an empty app shell, those crawlers have nothing to read or quote.

**Limitations:**

- The Actor reads robots.txt and HTML from the audit server. It cannot see IP-based firewall rules, CDN bot settings, or a robots.txt served differently to verified crawlers.
- Some sites serve different content to bots and browsers (dynamic rendering). The SPA check reflects what a browser-like request receives.
- Sitemap statistics come from up to 20 sampled child sitemaps (3 by default), each read up to 10 MB. `estimatedTotalUrls` is an extrapolation.
- Only the homepage and the URL you enter are inspected; it is not a full-site crawl.
- Response times are measured from the audit server (plus proxy, if used) and are indicative.

**Responsible use:** The Actor only requests public files that crawlers are expected to read (robots.txt, sitemaps, llms.txt) and at most two public pages per input. That is typically 6-12 lightweight requests per website, with at most 3 in parallel per site. It collects no personal data and does not bypass logins, paywalls or bot protection. Audit websites you own, manage, or have a legitimate reason to evaluate, and follow the websites' terms.

# Actor input Schema

## `urls` (type: `array`):

Domains (example.com) or full URLs (https://example.com/pricing). A domain audits the homepage; a deeper URL audits the homepage and that page, and checks whether crawlers may fetch that exact path. Each unique entry is one audit and one charged result. Duplicates are skipped.

## `checkLlmsTxt` (type: `boolean`):

Fetch /llms.txt and /llms-full.txt and validate their structure (title, summary, link sections, broken-looking links). Turning it off skips two requests per site and removes llms.txt from the discoverability score.

## `maxSitemapsToSample` (type: `integer`):

When the sitemap is a sitemap index, how many child sitemaps to download for URL counts and lastmod freshness. They are picked evenly across the index. Higher values give more accurate stats but take longer. 0 reads only the index file.

## `maxConcurrency` (type: `integer`):

How many websites are audited in parallel. Each site uses at most 3 requests at a time. Higher values finish large lists faster but use more memory: give the run 1 GB of memory if you go above 5.

## `proxyConfiguration` (type: `object`):

Optional. Route requests through Apify Proxy or your own proxies, for example if a site rate-limits datacenter IPs. Response times are then measured through the proxy, so the slowness thresholds are relaxed by 1 second.

## Actor input object example

```json
{
  "urls": [
    "https://apify.com"
  ],
  "checkLlmsTxt": true,
  "maxSitemapsToSample": 3,
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://apify.com"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("frank_conifer_af4/ai-search-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://apify.com"],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("frank_conifer_af4/ai-search-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://apify.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call frank_conifer_af4/ai-search-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,frank_conifer_af4/ai-search-readiness-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/yK3JTW1ejydrVVVv9/builds/NtVxXTA1oVqQ6OfUK/openapi.json