# AI Search Readiness Audit: llms.txt, AI Crawlers & GEO Score (`larkmoor/ai-search-readiness-audit`) Actor

Can ChatGPT, Claude, Perplexity, Gemini and Copilot read and cite your site? Bulk GEO/AEO audit: robots.txt rules for 30 AI crawlers, llms.txt check, content without JavaScript, schema, meta robots. 0-100 score + fixes. $6 per 1,000 URLs.

- **URL**: https://apify.com/larkmoor/ai-search-readiness-audit.md
- **Developed by:** [Larkmoor Data](https://apify.com/larkmoor) (community)
- **Categories:** SEO tools, AI
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.10 / 1,000 url auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Search Readiness Audit: llms.txt, AI Crawlers & GEO Score

Find out in seconds whether **ChatGPT, Claude, Perplexity, Google AI Overviews, Gemini and Microsoft Copilot can actually reach, read and cite** a website. Paste a list of domains or page URLs and get one row per URL with a **0-100 AI search readiness score**, a letter grade, a pass/fail list of 27 checks and the **top fixes in plain language**.

This is a technical **GEO (generative engine optimization) / AEO (answer engine optimization) audit** that runs in bulk. It reads only the site's own public files (robots.txt, llms.txt, sitemap, the page HTML) with a plain HTTP client, so it is fast and cheap: **$6 per 1,000 URLs**, and URLs that do not answer at all are free.

### What does it check?

**Crawler access (robots.txt and firewall)**

- The robots.txt verdict for **30 AI crawlers**, split by purpose: AI *search* bots that fetch pages to answer and cite (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, Applebot, DuckAssistBot, Amazonbot, YouBot...) and *training* bots that collect data for models (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, cohere-ai...).
- Whether each bot is named in robots.txt or only covered by a wildcard rule.
- Cloudflare **Content-Signal** lines (`search=yes, ai-train=no`) when present.
- Whether the page answers a plain HTTP client at all, or returns a block, a challenge page or an error (AI fetchers usually see the same thing).
- `noindex`, `nosnippet`, `max-snippet:0`, `noai` and `noimageai` in meta robots and the `X-Robots-Tag` header.

**AI files**

- **llms.txt**: present, well-formed (H1 title + Markdown links), number of sections and links.
- **llms-full.txt**: present and its size.
- XML sitemap (from robots.txt or the usual locations).

**Content and structure**

- **Content visible without JavaScript**: the number of words in the server-rendered HTML, and whether the page is an empty JavaScript app shell. Most AI crawlers do not run JavaScript, so a client-side-only page is invisible to them.
- Title and meta description length, canonical URL, `lang`.
- One H1, subheadings, lists and tables (the parts AI answers quote most easily).
- schema.org **JSON-LD types** (Organization, WebSite, Article, Product, FAQPage, LocalBusiness...), FAQ content.
- Author / publisher, published and modified dates, Open Graph tags, image alt-text coverage.
- Response time and HTML size.

### Who is it for?

- **SEO and GEO agencies** auditing a client list or a prospect list before a pitch.
- **Marketing teams** checking that a robots.txt change or a CDN bot setting did not lock out ChatGPT search or Perplexity.
- **Publishers** deciding which AI bots to block for training while staying visible in AI search.
- **Developers** verifying that a JavaScript site renders its content on the server.
- **Analysts** measuring how many sites in a sector block AI crawlers.

### Input

| Field | What it does |
|---|---|
| `urls` | domains or page URLs, one per line (`stripe.com`, `https://example.com/pricing`) |
| `includeCheckDetails` | add the full `checks` list and per-crawler verdicts `aiCrawlers` (default on) |
| `maxConcurrency` | URLs audited in parallel, default 8 |
| `maxResults` | cap on the number of URLs |

```json
{
  "urls": ["stripe.com", "https://www.nytimes.com", "https://vercel.com/pricing"],
  "includeCheckDetails": true
}
```

robots.txt, llms.txt and the sitemap are read once per site, so auditing 50 pages of one site costs no extra requests for those files.

### Output

One row per URL (shortened):

```json
{
  "url": "https://www.nytimes.com",
  "domain": "nytimes.com",
  "score": 47,
  "grade": "F",
  "aiSearchBotsAllowed": 2,
  "aiSearchBotsChecked": 6,
  "blockedSearchBots": ["OAI-SearchBot", "ChatGPT-User", "Claude-SearchBot", "PerplexityBot"],
  "blockedTrainingBots": ["GPTBot", "ClaudeBot", "Google-Extended", "Applebot-Extended"],
  "llmsTxt": false,
  "sitemap": "https://www.nytimes.com/sitemaps/new/news.xml.gz",
  "httpStatus": 403,
  "pageContentChecked": false,
  "topFixes": [
    "The page answers plain HTTP clients with a block, a challenge page or an error...",
    "robots.txt blocks OAI-SearchBot, so ChatGPT search cannot read or cite your pages..."
  ]
}
```

A healthy site looks like this:

```json
{
  "domain": "vercel.com", "score": 98, "grade": "A",
  "aiSearchBotsAllowed": 6, "llmsTxt": true, "llmsTxtValid": true,
  "contentVisibleWithoutJs": true, "wordsWithoutJs": 546,
  "schemaTypes": ["Organization", "PostalAddress", "ContactPoint"],
  "topFixes": ["Your meta description is 45 characters; make it 50-170 characters..."]
}
```

Other fields: `robotsTxtFound`, `contentSignals`, `llmsTxtUrl`, `llmsTxtSections`, `llmsTxtLinks`, `llmsFullTxt`, `finalUrl`, `responseMs`, `htmlKb`, `title`, `metaDescription`, `canonical`, `lang`, `robotsDirectives`, `noindex`, `nosnippet`, `noai`, `wordsWithoutJs`, `javascriptAppShell`, `h1`, `h1Count`, `h2Count`, `listsCount`, `tablesCount`, `hasOrganizationSchema`, `hasFaqSchema`, `author`, `datePublished`, `dateModified`, `ogImage`, `imagesWithAltPct`, `checksPassed`, `checksTotal`, `fixes` (each with priority and points) and, with details on, `checks` and `aiCrawlers`.

When the page itself is blocked, only the crawler-access and AI-file checks are scored (`pageContentChecked: false`), so a firewall does not count as dozens of content failures.

### How is the score calculated?

Each check has a weight: crawler access weighs most (Googlebot 10, a readable page 10, content without JavaScript 10, OAI-SearchBot 8, Bingbot 8, no noindex 8, PerplexityBot 6), then structured data, AI files, metadata and structure. The score is the share of weighted points passed; grades are A 90+, B 80+, C 70+, D 55+, F below. Blocking *training* bots does not lower the score: that is a legitimate choice, and it is reported separately in `blockedTrainingBots`.

### How much does it cost?

You pay **$0.006 per URL audited** ($6 per 1,000) plus a one-time $0.001 actor start. URLs that give no response are pushed for free. Platform usage is small: a 256 MB run.

- **25 client homepages**: about $0.15.
- **500 prospect domains** for an agency pitch: about $3.00.
- **5,000 domains** for a sector study of AI-crawler blocking: about $30.

Higher Apify plans get automatic volume discounts.

### FAQ

**Is this legal?** Yes. The actor reads publicly served files (robots.txt, llms.txt, sitemaps, page HTML) the same way a browser does, respects that the files are public by design, and never logs in. It identifies itself with its own user agent and never pretends to be an AI crawler.

**Does it test whether my firewall blocks GPTBot?** It reports robots.txt rules and whether the page answers a plain client. It does not send fake crawler user agents. If your CDN blocks verified AI bots by IP, check your bot-management settings as well.

**Does it run JavaScript?** No, on purpose. The point is to see the page the way most AI crawlers see it.

**Can I audit many pages of one site?** Yes: put the page URLs in the list. Site files are read once and reused.

**What is llms.txt?** A proposed standard (llmstxt.org): a Markdown file at the site root with a title, a short summary and links to the pages that matter, so AI tools can find the right content quickly.

### Integrations

Export results as JSON, CSV or Excel, schedule weekly re-audits, call the actor from the Apify API, or connect it to Make, Zapier, n8n, Google Sheets and webhooks to get alerted when a site's score drops.

### Changelog

- **0.1** (Nov 2026): first release with 30 AI crawlers, llms.txt / llms-full.txt, Content-Signal, JS-free content check, schema and 27 weighted checks.

# Actor input Schema

## `urls` (type: `array`):

Domains or page URLs, one per line (e.g. stripe.com or https://example.com/pricing). Each line gets its own row; robots.txt and llms.txt are read once per site.

## `includeCheckDetails` (type: `boolean`):

Add the full pass/fail list (`checks`) and the per-crawler robots.txt verdicts (`aiCrawlers`) to each row. Turn off for a slimmer dataset.

## `maxConcurrency` (type: `integer`):

How many URLs to audit in parallel.

## `maxResults` (type: `integer`):

Stop after this many URLs.

## Actor input object example

```json
{
  "urls": [
    "https://stripe.com",
    "https://www.nytimes.com",
    "https://vercel.com"
  ],
  "includeCheckDetails": true,
  "maxConcurrency": 8,
  "maxResults": 1000
}
```

# Actor output Schema

## `results` (type: `string`):

One row per input as JSON items in the run's default dataset (also downloadable as CSV/Excel from the dataset).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://stripe.com",
        "https://www.nytimes.com",
        "https://vercel.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("larkmoor/ai-search-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [
        "https://stripe.com",
        "https://www.nytimes.com",
        "https://vercel.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("larkmoor/ai-search-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://stripe.com",
    "https://www.nytimes.com",
    "https://vercel.com"
  ]
}' |
apify call larkmoor/ai-search-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,larkmoor/ai-search-readiness-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/wmFFObwnRJ8LWY8N2/builds/7Rxboog1fGpDqWYjc/openapi.json