# AI Search Readiness Audit (AI Crawlers, llms.txt, SEO) (`scrapme/ai-search-readiness-audit`) Actor

Bulk-check websites for AI search readiness: which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more) robots.txt allows or blocks, llms.txt, sitemap and basic on-page SEO and structured data.

- **URL**: https://apify.com/scrapme/ai-search-readiness-audit.md
- **Developed by:** [MDBBusiness](https://apify.com/scrapme) (community)
- **Categories:** SEO tools, AI
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$10.00 / 1,000 domain checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Search Readiness Audit: AI crawlers, llms.txt and basic SEO

Check any list of websites for **AI search readiness** in one run. For each domain the Actor reads the site's own `robots.txt`, `llms.txt`, first sitemap file and (optionally) a few public pages, and returns one clean row per domain: which AI crawlers are allowed or blocked, whether the site publishes `llms.txt`, whether a sitemap exists, how the home page looks to a crawler that does not run JavaScript, and a 0-100 readiness score with a list of fix codes.

No login, no browser, no proxy and no API key. Requests are polite: at most one request per second per domain, a clear user agent (`AISearchReadinessAudit`), and the Actor follows the robots.txt of the site it audits.

### What you get per domain

- **AI crawler matrix** (18 crawlers): for each of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Applebot-Extended, CCBot, Bytespider, Amazonbot, meta-externalagent, cohere-ai, DuckAssistBot and MistralAI-User: `allowed`, `blocked` or `partial`, with the robots.txt rule that decided it and whether it came from a named group or the `*` group. Rules follow RFC 9309 (longest match wins, `Allow` wins ties, wildcards `*` and `$`).
- **AI policy summary**: `open`, `blocks_training_only`, `blocks_some_search_bots`, `blocks_ai_search`, or `unknown`.
- **llms.txt and llms-full.txt**: found or not, size, and whether `llms.txt` follows the format on llmstxt.org (H1 title, summary, sections, link count). HTML pages that answer on that path (soft 404) are not counted as found.
- **Sitemap**: found or not, type, number of entries in the first file.
- **Home page and sampled pages** (optional): HTTP status, response time, HTTPS, title and meta description length, canonical, `lang`, hreflang count, Open Graph tags, H1/H2 count, visible words in the raw HTML (a low number often means the content is built by JavaScript), JSON-LD types, image alt coverage, `noindex` and AI opt-out meta tags, security headers (HSTS, CSP, X-Content-Type-Options).
- **Score and issues**: `geoScore` (0-100) and `issues` such as `SEARCH_BOT_BLOCKED:PerplexityBot`, `LLMS_TXT_MISSING`, `JSON_LD_MISSING`.

### Input

| Field | Meaning | Default |
|---|---|---|
| `domains` | Websites to check, e.g. `example.com` (URLs also work, only the domain is used) | required |
| `checkPages` | Also read the sitemap and home page and compute the full score. Off = fast check of robots.txt and llms.txt only (3 requests per domain) | on |
| `pagesPerDomain` | Home page plus pages spread evenly over the first sitemap file (1-10) | 1 |
| `maxConcurrency` | Domains checked in parallel (each domain still gets at most 1 request per second) | 5 |
| `delaySeconds` | Pause between requests to one domain (minimum 1) | 1 |
| `includePageTexts` | Also return title and meta description text (default: lengths only) | off |

Example:

```json
{
  "domains": ["example.com", "wikipedia.org", "dr.dk"],
  "checkPages": true,
  "pagesPerDomain": 3
}
```

### Output example (shortened)

```json
{
  "domain": "dr.dk",
  "status": "ok",
  "aiPolicy": "blocks_ai_search",
  "geoScore": 62,
  "scoreScope": "robots.txt, llms.txt, sitemap and home page",
  "issues": ["SEARCH_BOT_BLOCKED:OAI-SearchBot", "SEARCH_BOT_BLOCKED:Claude-SearchBot", "SEARCH_BOT_BLOCKED:PerplexityBot", "LLMS_TXT_MISSING"],
  "trainingBotsBlocked": ["GPTBot", "ClaudeBot", "Google-Extended"],
  "crawlers": [{"bot": "GPTBot", "vendor": "OpenAI", "purpose": "training", "status": "blocked", "rule": "Disallow: /", "ruleSource": "named group"}],
  "llmsTxt": {"found": false}
}
```

### How the score works

`geoScore` is a rule of thumb, not an official ranking factor of any search engine or AI product. Points: search bots not blocked (OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot) 30, valid `llms.txt` 10, sitemap 10, and for the home page: title length 10-70 characters 8, meta description 50-170 characters 8, canonical 4, `lang` 3, exactly one H1 6, JSON-LD 10, at least 3 Open Graph tags 4, at least 150 words in raw HTML 7. The score is the share of the points that were checked, so `scoreScope` tells you what it is based on. Do not compare scores with different scopes. Blocking training crawlers is a legitimate choice and is not penalised.

### Limits and honesty

- The Actor reads **robots.txt rules only**. A firewall, CDN or bot-protection service can block AI crawlers even when robots.txt allows them. That cannot be seen from robots.txt.
- Some AI products fetch pages on a user's request and may ignore robots.txt (the vendors document this for user-triggered fetchers). The `purpose` field tells you which crawlers are of that kind.
- The crawler list follows the vendors' public documentation (OpenAI, Anthropic, Perplexity, Google, Apple); other vendors' tokens can change. Unknown bots are not reported.
- If `/robots.txt` cannot be read (server error), the Actor reports `aiPolicy: unknown` and fetches no further pages from that domain. A 404 means no rules (everything allowed), as in RFC 9309.
- Pages are read as raw HTML without JavaScript, up to 1 MB each. The Actor skips pages that the site's robots.txt disallows for its user agent.
- Check only websites you are allowed to check. The Actor does not log in, does not solve CAPTCHAs and does not hide itself.
- Output contains site-level facts only (rules, counts, lengths, schema types, headers). No contact details, names of people or user content.

### Pricing

Pay per checked domain (one row per domain, about 10 USD per 1,000 domains). Domains that cannot be reached are **not charged**; they are listed in the key-value store record `FAILED_DOMAINS`. If you set a maximum cost per run, the Actor stops when the limit is reached and says so in the log and the run status.

### Sources and standards

robots.txt: RFC 9309. llms.txt: llmstxt.org. Crawler tokens: documentation from OpenAI, Anthropic, Perplexity, Google and Apple.

# Actor input Schema

## `domains` (type: `array`):

Websites to check, one per line, e.g. example.com. Full URLs also work (only the domain is used). Only audit websites you are allowed to check; the Actor reads robots.txt, llms.txt, a sitemap and a few public pages, at most one request per second per domain.

## `checkPages` (type: `boolean`):

Also read the first sitemap file and the home page (title, description, headings, structured data, text length, security headers) and compute the readiness score. Turn off for a fast check of robots.txt and llms.txt only (2-3 requests per domain).

## `pagesPerDomain` (type: `integer`):

Number of pages to analyse per domain: the home page plus pages spread evenly over the first sitemap file. 1 = home page only. Only used when 'Check sitemap and pages' is on.

## `maxConcurrency` (type: `integer`):

How many different domains are checked at the same time. Each domain still gets at most one request per second.

## `delaySeconds` (type: `number`):

Pause between requests to the same domain. Minimum 1.

## `includePageTexts` (type: `boolean`):

By default the output only has the lengths of the title and meta description. Turn on to also return the texts. Only use for sites where this is appropriate (the texts of personal websites can contain personal data).

## Actor input object example

```json
{
  "domains": [
    "example.com",
    "wikipedia.org",
    "dr.dk"
  ],
  "checkPages": true,
  "pagesPerDomain": 1,
  "maxConcurrency": 5,
  "delaySeconds": 1,
  "includePageTexts": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "example.com",
        "wikipedia.org",
        "dr.dk"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapme/ai-search-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "example.com",
        "wikipedia.org",
        "dr.dk",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("scrapme/ai-search-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "example.com",
    "wikipedia.org",
    "dr.dk"
  ]
}' |
apify call scrapme/ai-search-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapme/ai-search-readiness-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ZovOl1ROXHNdxXYqd/builds/UDOgFthfiWE1ChwMP/openapi.json