AI Search Readiness Auditor: SEO, llms.txt, Schema

Audit any website for AI search and classic SEO. The Actor checks which AI crawlers your robots.txt lets in (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot and 15 more), your llms.txt, sitemap, structured data (JSON-LD) and on-page basics, then returns a 0-100 score and prioritized fixes as JSON, in English or Spanish. It uses plain HTTP (no browser, no API keys) and costs about $0.01 per page plus $0.02 per site (see pricing; free-plan runs are capped at 5 pages). It does not run JavaScript and does not predict rankings; the score is a transparent checklist.

Resumen en español al final de la página.

Who is it for?

• SEO freelancers and agencies who need audits as data, plus client-ready fix text in Spanish or English.
• Developers who want an AI-readiness or SEO check in CI, a monitoring job or a spreadsheet pipeline.
• AI agents that need a factual, scored snapshot of a site before recommending changes (see Use with AI agents (MCP)).

Why do AI crawlers matter for my site?

AI assistants answer questions by fetching and quoting web pages. Whether you get cited depends on things that are easy to break and hard to notice:

• robots.txt rules that accidentally block AI search crawlers (a blanket Disallow: /, a rule meant for a training bot), or a robots.txt that returns a server error, which crawlers must treat as "everything is disallowed".
• Pages that are an empty JavaScript shell: most AI crawlers do not run JavaScript, so they see almost no text.
• noindex, nosnippet or tiny max-snippet limits that stop Google and others from quoting you.
• Missing sitemap, canonical or structured data, and inconsistent lang / hreflang on multilingual sites.

Training crawlers (which collect data to train models) and search/retrieval crawlers (which decide whether you appear in answers) are different things with different trade-offs. This Actor reports them separately, so you can block training without disappearing from AI search.

What does it check?

For every site you submit:

• AI-crawler access in robots.txt for 20 crawlers, parsed per RFC 9309 (group matching, longest match, Allow wins ties, * and $). Each is labelled training, search or user-fetch (an assistant fetching a page on a user's request).
• llms.txt / llms-full.txt presence and a format check against llmstxt.org (H1, summary blockquote, link sections).
• Sitemap presence, URL count (first file only) and whether robots.txt declares it.
• Page checks: HTTP status, HTTPS, response time, X-Robots-Tag, meta robots (noindex, nosnippet, max-snippet), title, meta description, canonical, lang, hreflang, Open Graph, JSON-LD types (Organization, WebSite, Product, Article, FAQPage, BreadcrumbList...), H1 count, and how much text is in the raw HTML (flags JavaScript shells).
• Bilingual signals: <html lang> vs hreflang self-reference vs x-default vs og:locale vs Content-Language, plus a conservative English/Spanish text check that catches pages declared in the wrong language.
• Prioritized recommendations (critical > high > medium > low > info), each with a one-line fix and the score points at stake, in en or es.

How is the score calculated?

100 points, all weights visible in scoreBreakdown on every result:

Component	Points
AI search crawlers allowed in robots.txt (weighted by reach)	20
AI assistant user-fetch crawlers allowed	8
robots.txt reachable and well-formed	4
llms.txt present and in llmstxt.org format	8
Sitemap found, non-empty, declared in robots.txt	6
Indexability (no noindex / nosnippet / tiny max-snippet)	12
Text present in server-rendered HTML (not a JS shell)	14
HTTP 2xx, HTTPS, response time	6
Structured data (JSON-LD / microdata)	12
Metadata: title, description, canonical, lang consistency, Open Graph, H1	10

Training crawlers are not scored: allowing or blocking model training is a policy decision, not a quality signal. The crawler list, the official documentation URL of each entry and the weights live in one file, src/core/ai-crawlers.ts (last verified 2026-10-19): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Applebot, Applebot-Extended, CCBot, Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot, Amzn-SearchBot, Amzn-User, Bytespider (no first-party documentation found; flagged as unverified), DuckAssistBot.

How do I run it?

1. Paste website URLs or bare domains into Websites to audit (one per line).
2. Choose the language of the recommendations (en or es).
3. Set Max pages per site (default 5) to control cost, then run. Results appear in the dataset; the Overview, Prioritized fixes and AI crawler access views give table summaries.

Input

Field	Type	Default	Description
urls	string[]	required	Website URLs or domains. URLs on the same site are grouped; the first URL for a site is audited as its main page. Public http(s) sites on ports 80/443/8080/8443 only.
language	en | es	en	Language of the recommendations text.
maxPagesPerSite	integer 1-10	5	Cost control: pages audited per site. Extra pages come from the URLs you listed for the site, then from the first sitemap file. A site with fewer pages is billed only for the pages it has.
maxConcurrency	integer 1-10	4	Different sites audited in parallel. Requests to one site are always sequential.

{ "urls": ["https://example.com", "mi-sitio.mx"], "language": "es", "maxPagesPerSite": 3 }

What does the output look like?

One dataset item per site. Real output for https://example.com (abbreviated); the full sample for three sites is in sample-output.json.

{
  "site": "https://example.com",
  "auditStatus": "ok",
  "language": "en",
  "score": 52,
  "grade": "D",
  "summary": {
    "searchCrawlersAllowed": "7/7 allowed",
    "userFetchersAllowed": "5/5 allowed",
    "trainingCrawlersBlocked": "0/8 blocked",
    "llmsTxt": "missing",
    "sitemap": "missing",
    "pagesAudited": 1
  },
  "billableEvents": { "page-audited": 1, "site-summary": 1 },
  "aiCrawlers": [
    { "token": "GPTBot", "category": "training", "status": "allowed", "matchedGroup": "(robots.txt missing)" },
    { "token": "OAI-SearchBot", "category": "search", "status": "allowed", "matchedGroup": "(robots.txt missing)" }
  ],
  "recommendations": [
    {
      "id": "page-structured-data-missing",
      "severity": "medium",
      "area": "structured-data",
      "title": "No structured data (JSON-LD or microdata) found",
      "fix": "Add JSON-LD: Organization + WebSite on the homepage, Article/Product/FAQPage/BreadcrumbList where they apply.",
      "pointsAtStake": 12
    }
  ]
}

In the same smoke run apify.com scored 100 and www.gob.mx scored 76 (no llms.txt, no structured data, sitemap not declared in robots.txt).

Field	Meaning
score, grade, scoreBreakdown[]	0-100 score, A-F grade and every component with weight and points earned.
summary	Headline findings (crawlers allowed, llms.txt, sitemap, pages audited).
aiCrawlers[]	robots.txt verdict per crawler: category, status (allowed / blocked / partial / unknown), matched group, vendor caveats.
robotsTxt, llmsTxt, llmsFullTxt, sitemap	Raw findings for the site-level files.
pages[]	Per-page HTTP, robots-meta, metadata, JSON-LD, content and language findings.
recommendations[]	Prioritized fixes: severity, area, title, one-line fix, pointsAtStake, affectedUrls.
billableEvents, notes[], errors[]	What this item is billed for, context, and fetch errors.

A site that cannot be reached still produces an item with auditStatus: "failed" and the reason in errors; the run itself does not fail. Each run also stores a RUN-SUMMARY record (counts, skipped sites, events charged).

How much does it cost?

Pricing note. These are the planned pay-per-event prices. They are set in Apify Console when the Actor is published, so the Pricing tab on the Actor page is authoritative.

Event	Charged when	Price
page-audited	once per page that was fetched (HTTP 2xx, HTML) and analysed	$0.01
site-summary	once per site when the site-level checks completed (robots.txt AI rules, llms.txt, sitemap, score and recommendations)	$0.02

Worked examples

Run	Calculation	Cost
1 site, 1 page (for example example.com, which has a single page)	1 x $0.01 + $0.02	$0.03
1 site, 5 pages (the default maximum)	5 x $0.01 + $0.02	$0.07
1 site, 10 pages	10 x $0.01 + $0.02	$0.12
5 sites, 10 pages each (50 pages)	50 x $0.01 + 5 x $0.02	$0.60
50 sites, 1 page each (50 pages)	50 x $0.01 + 50 x $0.02	$1.50

Free plan: runs started from a free Apify plan audit at most 5 pages in total per run (sites are then processed one at a time). Paying users have no page cap. Spending limit: if you set a maximum cost per run, the Actor reserves budget before starting each site, finishes the sites in progress and stops gracefully when the limit cannot pay for another one; RUN-SUMMARY lists the sites that were skipped.

Use with AI agents (MCP)

The Actor returns structured JSON, needs no credentials of its own and asks for Apify's limited permissions (it only uses its own storages), so AI agents can call it through Apify's MCP server from any MCP client. Search for "AI search readiness audit" in the Actor search tool, or call it directly by ID: atalaya-ai/ai-search-readiness-auditor.

Copy-paste prompt for an agent:

Use the AI Search Readiness Auditor on example.com with language: "en" and maxPagesPerSite: 3. Report the score, any critical or high recommendations with their one-line fixes, and which AI search crawlers are blocked.

Notes for agents: pass the input shown above; read one dataset item per site; score (0-100) and recommendations[] (sorted by severity, each with a single-line fix) are the fields to act on; billableEvents tells you what the item cost; set maxPagesPerSite to bound spend.

What are the limits?

• HTTP only: no JavaScript rendering, no screenshots, no Core Web Vitals.
• robots.txt, llms.txt and the sitemap are read from the host you supply (after redirects). If the page redirects to another host, a note says so.
• Only the first sitemap file is read; for a sitemap index the count is the number of child sitemaps.
• hreflang reciprocity (the other language versions linking back) is not verified; only the page's own annotations are checked. The English/Spanish text check is a conservative heuristic and can miss or, rarely, misjudge short or mixed-language pages.
• The crawler list reflects vendor documentation on 2026-10-19; vendors change tokens and behaviour. Bytespider has no first-party documentation.
• The score is a transparent checklist, not a ranking prediction or a guarantee of being cited by any AI product.
• Sites that block or rate-limit our User-Agent or datacenter IPs may return errors or different content. The Actor does not try to evade blocks, captchas or paywalls.
• Targets must be public websites. Private/internal addresses, odd ports and non-http(s) schemes are refused. DNS is resolved before connecting; a DNS-rebinding race between check and connection is theoretically possible.
• Not affiliated with OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, ByteDance, DuckDuckGo or Common Crawl; product names are used only to identify their crawlers.

Frequently asked questions

Does it respect robots.txt?

Yes. It identifies as AtalayaBot/0.1 (+https://apify.com/atalaya-ai), reads robots.txt first, and will not fetch a page that robots.txt disallows for AtalayaBot or *; such pages are reported, not billed. Requests to one host are sequential with a short pause, with a 15 s timeout, a 5 MB response cap and at most 5 redirects. It only reads public pages.

What is never billed?

• Sites that cannot be reached (DNS, TLS or connection errors, timeouts): no page-audited and no site-summary.
• Pages that return an HTTP error (4xx/5xx), are not HTML, or are skipped because robots.txt disallows them.
• The site-summary of a site whose robots.txt could not be fetched at the network level (the site-level checks did not complete).
• Sites that were not started because your spending limit (or the free-plan page cap) was reached.
• Invalid inputs (not an http(s) URL or domain), and the same URL listed twice (it is audited once).

Results are saved to the dataset before they are charged, and the Actor never starts a site the run's spending limit cannot pay for.

Does llms.txt help me show up in AI search?

Unproven. llms.txt is a community proposal; the vendors listed above have not confirmed that they use it for ranking or citation. It is worth 8 of 100 points and its recommendation is low severity.

Should I block AI training crawlers?

That is your call. Blocking GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot or Meta-ExternalAgent does not remove you from ChatGPT search, Claude search, Google Search or Apple search, which use separate crawlers (OAI-SearchBot, Claude-SearchBot, Googlebot, Applebot). The report shows both groups so you can decide.

Why does a user-fetch crawler show as blocked if it ignores robots.txt?

Several vendors state that user-initiated fetchers (ChatGPT-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User) may not follow robots.txt. The report shows what your robots.txt says and attaches the vendor's caveat.

Does it run JavaScript?

No, on purpose: it shows what crawlers that do not run JavaScript receive. A page that needs JavaScript to show its text is reported as a JavaScript shell.

Can I run it from the API or a schedule?

Yes, like any Apify Actor: API, schedules, webhooks, integrations and MCP. Input and output are plain JSON.

Something is wrong or a crawler changed. Where do I report it?

Open an issue in the Actor's Issues tab with the URL and the field you disagree with.

Resumen en español

Auditoría SEO y de preparación para búsqueda con IA. Revisa qué rastreadores de IA permite tu robots.txt (ChatGPT search, Claude, Perplexity, Google AI Overviews), tu llms.txt, el sitemap, los datos estructurados y lo básico de cada página, y devuelve una puntuación de 0 a 100 con correcciones priorizadas en español o inglés (language: "es"). Solo HTTP, sin navegador ni claves de API.

• Analiza robots.txt según el RFC 9309 para 20 rastreadores y los separa en entrenamiento, búsqueda y consulta a petición del usuario, para que puedas bloquear el entrenamiento sin desaparecer de la búsqueda con IA.
• Revisa llms.txt, el sitemap, noindex/nosnippet, el HTML que ve un rastreador sin JavaScript (detecta carcasas de JavaScript), metadatos, JSON-LD y la coherencia de lang, hreflang, og:locale y Content-Language.
• Respeta el robots.txt del sitio auditado, se identifica como AtalayaBot y limita la velocidad por dominio.
• Precio por evento (se fija en Apify Console; la pestaña de precios es la referencia): page-audited $0.01 por página auditada y site-summary $0.02 por sitio con revisión completa. Ejemplo: un sitio con 10 páginas cuesta $0.12. Solo se cobra lo que se completó, después de guardar el resultado, y la ejecución se detiene con elegancia al llegar a tu límite de gasto. Los planes gratuitos auditan hasta 5 páginas por ejecución.
• Límites: no ejecuta JavaScript; la puntuación es una lista de verificación transparente, no una garantía de aparecer en ninguna IA; el detector de idioma es heurístico; no verifica la reciprocidad de hreflang.