# AI Search Readiness Auditor: SEO, llms.txt, Schema (`atalaya-ai/ai-search-readiness-auditor`) Actor

Audit any site for AI search and SEO: AI-crawler access, llms.txt, schema, on-page checks. Scored fixes in English or Spanish. Pay per page, no API keys.

- **URL**: https://apify.com/atalaya-ai/ai-search-readiness-auditor.md
- **Developed by:** [Atalaya AI](https://apify.com/atalaya-ai) (community)
- **Categories:** SEO tools, Agents, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Search Readiness Auditor: SEO, llms.txt, Schema

**Audit any website for AI search and classic SEO.** The Actor checks which AI crawlers your `robots.txt` lets in (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot and 15 more), your `llms.txt`, sitemap, structured data (JSON-LD) and on-page basics, then returns a **0-100 score and prioritized fixes as JSON, in English or Spanish**. It uses plain HTTP (no browser, no API keys) and costs about **$0.01 per page plus $0.02 per site** (see pricing; free-plan runs are capped at 5 pages). It does not run JavaScript and does not predict rankings; the score is a transparent checklist.

*Resumen en español al final de la página.*

### Who is it for?

- **SEO freelancers and agencies** who need audits as data, plus client-ready fix text in Spanish or English.
- **Developers** who want an AI-readiness or SEO check in CI, a monitoring job or a spreadsheet pipeline.
- **AI agents** that need a factual, scored snapshot of a site before recommending changes (see *Use with AI agents (MCP)*).

### Why do AI crawlers matter for my site?

AI assistants answer questions by fetching and quoting web pages. Whether you get cited depends on things that are easy to break and hard to notice:

- `robots.txt` rules that accidentally block AI **search** crawlers (a blanket `Disallow: /`, a rule meant for a training bot), or a `robots.txt` that returns a server error, which crawlers must treat as "everything is disallowed".
- Pages that are an empty JavaScript shell: most AI crawlers do not run JavaScript, so they see almost no text.
- `noindex`, `nosnippet` or tiny `max-snippet` limits that stop Google and others from quoting you.
- Missing sitemap, canonical or structured data, and inconsistent `lang` / `hreflang` on multilingual sites.

Training crawlers (which collect data to train models) and search/retrieval crawlers (which decide whether you appear in answers) are different things with different trade-offs. This Actor reports them separately, so you can block training without disappearing from AI search.

### What does it check?

For every site you submit:

- **AI-crawler access in robots.txt** for 20 crawlers, parsed per RFC 9309 (group matching, longest match, Allow wins ties, `*` and `$`). Each is labelled `training`, `search` or `user-fetch` (an assistant fetching a page on a user's request).
- **llms.txt / llms-full.txt** presence and a format check against [llmstxt.org](https://llmstxt.org) (H1, summary blockquote, link sections).
- **Sitemap** presence, URL count (first file only) and whether robots.txt declares it.
- **Page checks**: HTTP status, HTTPS, response time, `X-Robots-Tag`, meta robots (`noindex`, `nosnippet`, `max-snippet`), title, meta description, canonical, `lang`, `hreflang`, Open Graph, JSON-LD types (Organization, WebSite, Product, Article, FAQPage, BreadcrumbList...), H1 count, and how much text is in the raw HTML (flags JavaScript shells).
- **Bilingual signals**: `<html lang>` vs hreflang self-reference vs `x-default` vs `og:locale` vs `Content-Language`, plus a conservative English/Spanish text check that catches pages declared in the wrong language.
- **Prioritized recommendations** (`critical` > `high` > `medium` > `low` > `info`), each with a one-line fix and the score points at stake, in `en` or `es`.

#### How is the score calculated?

100 points, all weights visible in `scoreBreakdown` on every result:

| Component | Points |
|---|---|
| AI **search** crawlers allowed in robots.txt (weighted by reach) | 20 |
| AI assistant **user-fetch** crawlers allowed | 8 |
| robots.txt reachable and well-formed | 4 |
| llms.txt present and in llmstxt.org format | 8 |
| Sitemap found, non-empty, declared in robots.txt | 6 |
| Indexability (no `noindex` / `nosnippet` / tiny `max-snippet`) | 12 |
| Text present in server-rendered HTML (not a JS shell) | 14 |
| HTTP 2xx, HTTPS, response time | 6 |
| Structured data (JSON-LD / microdata) | 12 |
| Metadata: title, description, canonical, lang consistency, Open Graph, H1 | 10 |

**Training crawlers are not scored**: allowing or blocking model training is a policy decision, not a quality signal. The crawler list, the official documentation URL of each entry and the weights live in one file, `src/core/ai-crawlers.ts` (last verified 2026-10-19): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Applebot, Applebot-Extended, CCBot, Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot, Amzn-SearchBot, Amzn-User, Bytespider (no first-party documentation found; flagged as unverified), DuckAssistBot.

### How do I run it?

1. Paste website URLs or bare domains into **Websites to audit** (one per line).
2. Choose the **language** of the recommendations (`en` or `es`).
3. Set **Max pages per site** (default 5) to control cost, then run. Results appear in the dataset; the *Overview*, *Prioritized fixes* and *AI crawler access* views give table summaries.

#### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `urls` | string\[] | required | Website URLs or domains. URLs on the same site are grouped; the first URL for a site is audited as its main page. Public http(s) sites on ports 80/443/8080/8443 only. |
| `language` | `en` | `es` | `en` | Language of the `recommendations` text. |
| `maxPagesPerSite` | integer 1-10 | `5` | Cost control: pages audited per site. Extra pages come from the URLs you listed for the site, then from the first sitemap file. A site with fewer pages is billed only for the pages it has. |
| `maxConcurrency` | integer 1-10 | `4` | Different sites audited in parallel. Requests to one site are always sequential. |

```json
{ "urls": ["https://example.com", "mi-sitio.mx"], "language": "es", "maxPagesPerSite": 3 }
```

### What does the output look like?

One dataset item per site. Real output for `https://example.com` (abbreviated); the full sample for three sites is in [`sample-output.json`](sample-output.json).

```json
{
  "site": "https://example.com",
  "auditStatus": "ok",
  "language": "en",
  "score": 52,
  "grade": "D",
  "summary": {
    "searchCrawlersAllowed": "7/7 allowed",
    "userFetchersAllowed": "5/5 allowed",
    "trainingCrawlersBlocked": "0/8 blocked",
    "llmsTxt": "missing",
    "sitemap": "missing",
    "pagesAudited": 1
  },
  "billableEvents": { "page-audited": 1, "site-summary": 1 },
  "aiCrawlers": [
    { "token": "GPTBot", "category": "training", "status": "allowed", "matchedGroup": "(robots.txt missing)" },
    { "token": "OAI-SearchBot", "category": "search", "status": "allowed", "matchedGroup": "(robots.txt missing)" }
  ],
  "recommendations": [
    {
      "id": "page-structured-data-missing",
      "severity": "medium",
      "area": "structured-data",
      "title": "No structured data (JSON-LD or microdata) found",
      "fix": "Add JSON-LD: Organization + WebSite on the homepage, Article/Product/FAQPage/BreadcrumbList where they apply.",
      "pointsAtStake": 12
    }
  ]
}
```

In the same smoke run `apify.com` scored 100 and `www.gob.mx` scored 76 (no llms.txt, no structured data, sitemap not declared in robots.txt).

| Field | Meaning |
|---|---|
| `score`, `grade`, `scoreBreakdown[]` | 0-100 score, A-F grade and every component with weight and points earned. |
| `summary` | Headline findings (crawlers allowed, llms.txt, sitemap, pages audited). |
| `aiCrawlers[]` | robots.txt verdict per crawler: category, status (`allowed` / `blocked` / `partial` / `unknown`), matched group, vendor caveats. |
| `robotsTxt`, `llmsTxt`, `llmsFullTxt`, `sitemap` | Raw findings for the site-level files. |
| `pages[]` | Per-page HTTP, robots-meta, metadata, JSON-LD, content and language findings. |
| `recommendations[]` | Prioritized fixes: `severity`, `area`, `title`, one-line `fix`, `pointsAtStake`, `affectedUrls`. |
| `billableEvents`, `notes[]`, `errors[]` | What this item is billed for, context, and fetch errors. |

A site that cannot be reached still produces an item with `auditStatus: "failed"` and the reason in `errors`; the run itself does not fail. Each run also stores a `RUN-SUMMARY` record (counts, skipped sites, events charged).

### How much does it cost?

> **Pricing note.** These are the planned pay-per-event prices. They are set in Apify Console when the Actor is published, so the *Pricing* tab on the Actor page is authoritative.

| Event | Charged when | Price |
|---|---|---|
| `page-audited` | once per page that was fetched (HTTP 2xx, HTML) and analysed | $0.01 |
| `site-summary` | once per site when the site-level checks completed (robots.txt AI rules, llms.txt, sitemap, score and recommendations) | $0.02 |

**Worked examples**

| Run | Calculation | Cost |
|---|---|---|
| 1 site, 1 page (for example `example.com`, which has a single page) | 1 x $0.01 + $0.02 | $0.03 |
| 1 site, 5 pages (the default maximum) | 5 x $0.01 + $0.02 | $0.07 |
| 1 site, 10 pages | 10 x $0.01 + $0.02 | $0.12 |
| 5 sites, 10 pages each (50 pages) | 50 x $0.01 + 5 x $0.02 | $0.60 |
| 50 sites, 1 page each (50 pages) | 50 x $0.01 + 50 x $0.02 | $1.50 |

**Free plan:** runs started from a free Apify plan audit at most **5 pages in total per run** (sites are then processed one at a time). Paying users have no page cap. **Spending limit:** if you set a maximum cost per run, the Actor reserves budget before starting each site, finishes the sites in progress and stops gracefully when the limit cannot pay for another one; `RUN-SUMMARY` lists the sites that were skipped.

### Use with AI agents (MCP)

The Actor returns structured JSON, needs no credentials of its own and asks for Apify's *limited permissions* (it only uses its own storages), so AI agents can call it through [Apify's MCP server](https://mcp.apify.com) from any MCP client. Search for "AI search readiness audit" in the Actor search tool, or call it directly by ID: `atalaya-ai/ai-search-readiness-auditor`.

Copy-paste prompt for an agent:

> Use the AI Search Readiness Auditor on `example.com` with `language: "en"` and `maxPagesPerSite: 3`. Report the score, any `critical` or `high` recommendations with their one-line fixes, and which AI search crawlers are blocked.

Notes for agents: pass the input shown above; read one dataset item per site; `score` (0-100) and `recommendations[]` (sorted by severity, each with a single-line `fix`) are the fields to act on; `billableEvents` tells you what the item cost; set `maxPagesPerSite` to bound spend.

### What are the limits?

- HTTP only: no JavaScript rendering, no screenshots, no Core Web Vitals.
- robots.txt, llms.txt and the sitemap are read from the host you supply (after redirects). If the page redirects to another host, a note says so.
- Only the first sitemap file is read; for a sitemap index the count is the number of child sitemaps.
- hreflang reciprocity (the other language versions linking back) is not verified; only the page's own annotations are checked. The English/Spanish text check is a conservative heuristic and can miss or, rarely, misjudge short or mixed-language pages.
- The crawler list reflects vendor documentation on 2026-10-19; vendors change tokens and behaviour. Bytespider has no first-party documentation.
- The score is a transparent checklist, not a ranking prediction or a guarantee of being cited by any AI product.
- Sites that block or rate-limit our User-Agent or datacenter IPs may return errors or different content. The Actor does not try to evade blocks, captchas or paywalls.
- Targets must be public websites. Private/internal addresses, odd ports and non-http(s) schemes are refused. DNS is resolved before connecting; a DNS-rebinding race between check and connection is theoretically possible.
- Not affiliated with OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, ByteDance, DuckDuckGo or Common Crawl; product names are used only to identify their crawlers.

### Frequently asked questions

#### Does it respect robots.txt?

Yes. It identifies as `AtalayaBot/0.1 (+https://apify.com/atalaya-ai)`, reads robots.txt first, and will not fetch a page that robots.txt disallows for `AtalayaBot` or `*`; such pages are reported, not billed. Requests to one host are sequential with a short pause, with a 15 s timeout, a 5 MB response cap and at most 5 redirects. It only reads public pages.

#### What is never billed?

- Sites that cannot be reached (DNS, TLS or connection errors, timeouts): no `page-audited` and no `site-summary`.
- Pages that return an HTTP error (4xx/5xx), are not HTML, or are skipped because robots.txt disallows them.
- The `site-summary` of a site whose robots.txt could not be fetched at the network level (the site-level checks did not complete).
- Sites that were not started because your spending limit (or the free-plan page cap) was reached.
- Invalid inputs (not an http(s) URL or domain), and the same URL listed twice (it is audited once).

Results are saved to the dataset before they are charged, and the Actor never starts a site the run's spending limit cannot pay for.

#### Does llms.txt help me show up in AI search?

Unproven. llms.txt is a community proposal; the vendors listed above have not confirmed that they use it for ranking or citation. It is worth 8 of 100 points and its recommendation is low severity.

#### Should I block AI training crawlers?

That is your call. Blocking GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot or Meta-ExternalAgent does not remove you from ChatGPT search, Claude search, Google Search or Apple search, which use separate crawlers (OAI-SearchBot, Claude-SearchBot, Googlebot, Applebot). The report shows both groups so you can decide.

#### Why does a user-fetch crawler show as blocked if it ignores robots.txt?

Several vendors state that user-initiated fetchers (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User) may not follow robots.txt. The report shows what your robots.txt says and attaches the vendor's caveat.

#### Does it run JavaScript?

No, on purpose: it shows what crawlers that do not run JavaScript receive. A page that needs JavaScript to show its text is reported as a JavaScript shell.

#### Can I run it from the API or a schedule?

Yes, like any Apify Actor: API, schedules, webhooks, integrations and MCP. Input and output are plain JSON.

#### Something is wrong or a crawler changed. Where do I report it?

Open an issue in the Actor's *Issues* tab with the URL and the field you disagree with.

### Resumen en español

**Auditoría SEO y de preparación para búsqueda con IA.** Revisa qué rastreadores de IA permite tu `robots.txt` (ChatGPT search, Claude, Perplexity, Google AI Overviews), tu `llms.txt`, el sitemap, los datos estructurados y lo básico de cada página, y devuelve una **puntuación de 0 a 100** con **correcciones priorizadas** en español o inglés (`language: "es"`). Solo HTTP, sin navegador ni claves de API.

- Analiza `robots.txt` según el RFC 9309 para 20 rastreadores y los separa en **entrenamiento**, **búsqueda** y **consulta a petición del usuario**, para que puedas bloquear el entrenamiento sin desaparecer de la búsqueda con IA.
- Revisa `llms.txt`, el sitemap, `noindex`/`nosnippet`, el HTML que ve un rastreador sin JavaScript (detecta carcasas de JavaScript), metadatos, JSON-LD y la coherencia de `lang`, `hreflang`, `og:locale` y `Content-Language`.
- Respeta el `robots.txt` del sitio auditado, se identifica como `AtalayaBot` y limita la velocidad por dominio.
- **Precio por evento (se fija en Apify Console; la pestaña de precios es la referencia):** `page-audited` $0.01 por página auditada y `site-summary` $0.02 por sitio con revisión completa. Ejemplo: un sitio con 10 páginas cuesta $0.12. Solo se cobra lo que se completó, después de guardar el resultado, y la ejecución se detiene con elegancia al llegar a tu límite de gasto. Los planes gratuitos auditan hasta 5 páginas por ejecución.
- **Límites:** no ejecuta JavaScript; la puntuación es una lista de verificación transparente, no una garantía de aparecer en ninguna IA; el detector de idioma es heurístico; no verifica la reciprocidad de hreflang.

# Actor input Schema

## `urls` (type: `array`):

Website URLs or bare domains, one per line (for example `example.com` or `https://www.example.com/pricing`). URLs on the same site are grouped: the first URL listed for a site is audited as its main page, the others are audited too if 'Max pages per site' allows. Only public http(s) websites on ports 80/443/8080/8443 are accepted.

## `language` (type: `string`):

Language used for the titles and fixes in the `recommendations` field. The data fields (scores, crawler names, URLs) are language-neutral.

## `maxPagesPerSite` (type: `integer`):

Cost control: the most pages audited per site (1-10, default 5). The first is the URL you gave; if you gave fewer URLs than this, extra pages are picked from the first sitemap file. A site with fewer pages is only billed for the pages it has. Each audited page is billed as one `page-audited` event; each site is billed one `site-summary` event. Free-plan runs are capped at 5 pages in total.

## `maxConcurrency` (type: `integer`):

How many different sites are audited at the same time (1-10). Requests to the same website are always sent one at a time with a short pause, regardless of this setting.

## Actor input object example

```json
{
  "urls": [
    "https://example.com",
    "https://www.example.com/pricing"
  ],
  "language": "en",
  "maxPagesPerSite": 3,
  "maxConcurrency": 4
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset items: one record per audited site with page details, score, grade and prioritized fixes.

## `runSummary` (type: `string`):

Key-value record with skipped inputs, billed events and the stop reason.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com"
    ],
    "maxPagesPerSite": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("atalaya-ai/ai-search-readiness-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "urls": ["https://example.com"],
    "maxPagesPerSite": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("atalaya-ai/ai-search-readiness-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com"
  ],
  "maxPagesPerSite": 3
}' |
apify call atalaya-ai/ai-search-readiness-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,atalaya-ai/ai-search-readiness-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/6cCKiuejFbQvMgIbN/builds/G7UX9Ksomvvr0nja9/openapi.json