# AI Search Readiness Audit (GEO/AEO) & llms.txt Generator (`dev-hoss/ai-search-readiness-audit`) Actor

Audit a website's readiness for AI search (ChatGPT, Perplexity, Claude, Gemini): AI crawler access in robots.txt, llms.txt, sitemap, schema.org, meta tags and content extractability. Get a 0-100 score, prioritized fixes and a ready-to-use llms.txt draft.

- **URL**: https://apify.com/dev-hoss/ai-search-readiness-audit.md
- **Developed by:** [Hossam Mohamed](https://apify.com/dev-hoss) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 33.3% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Search Readiness Audit (GEO/AEO) & llms.txt Generator

Audit how ready any website is for **AI search engines** — ChatGPT, Perplexity, Claude, Gemini, Google AI Overviews and Copilot. Get a **0–100 AI-readiness score**, a prioritized fix list, a per-domain AI-crawler access matrix, and a **ready-to-upload `llms.txt` draft**.

More search traffic is shifting to AI answers every quarter. But AI engines can only cite your content if their crawlers can access it, understand its structure, and extract it cleanly. This Actor audits exactly that — deterministically, with no LLM calls and no guesswork.

### What it checks

| Category | Checks |
|---|---|
| **Crawlability** | robots.txt presence; access rules for 20 AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Applebot, Bytespider, CCBot, Amazonbot, meta-externalagent, and more); sitemap discovery (robots.txt directives, /sitemap.xml, sitemap indexes); HTTP status of every audited page |
| **Discoverability** | llms.txt presence and validity (llmstxt.org format), meta descriptions, canonical URLs, Open Graph tags |
| **Content structure** | H1 presence and heading hierarchy, subheading coverage, content depth (word count), schema.org JSON-LD coverage |
| **Machine readability** | text-to-HTML ratio (extractability), JSON-LD parse errors, entity schema types (Organization, Article, Product, FAQPage…), llms.txt |

### Quick start

```json
{
    "startUrls": ["https://apify.com"],
    "maxPages": 10
}
```

That's it. Each distinct domain in `startUrls` gets its own full report.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array (required) | — | 1–10 URLs. Each distinct domain is audited separately. Any page on the domain works; the homepage is recommended. |
| `maxPages` | integer | `10` | Pages crawled per domain (1–200). 10 for a quick check, 25–50 for a real audit. |
| `maxConcurrency` | integer | `5` | Parallel requests. Lower to 1–2 for rate-limiting sites. |
| `respectRobotsTxt` | boolean | `true` | Whether the audit crawler itself obeys robots.txt. Does **not** affect AI-crawler checks, which are computed analytically. |
| `proxyConfiguration` | object | Apify proxy off | Optional proxy for sites that rate-limit datacenter IPs. |

### Output

Results land in **two datasets**, so each stays clean:

**1. Default dataset — `summary`, one per domain** (the headline result):

```json
{
    "type": "summary",
    "domain": "example.com",
    "overall": 62,
    "scoreInterpretation": "Fair — several important fixes needed",
    "categories": {
        "crawlability": 18,
        "discoverability": 8,
        "contentStructure": 22,
        "machineReadability": 14
    },
    "aiCrawlerMatrix": {
        "GPTBot": { "label": "OpenAI GPTBot (model training)", "verdict": "allowed" },
        "PerplexityBot": { "label": "PerplexityBot (Perplexity index)", "verdict": "disallowed" }
    },
    "robotsTxt": { "exists": true, "sitemapDirectives": ["https://example.com/sitemap.xml"] },
    "llmsTxt": { "present": false, "valid": false, "url": "https://example.com/llms.txt" },
    "sitemap": { "found": true, "url": "https://example.com/sitemap.xml", "urlCount": 412 },
    "topIssues": [
        {
            "code": "AI_BOTS_BLOCKED",
            "severity": "critical",
            "category": "crawlability",
            "count": 2,
            "detail": "Fully blocked in robots.txt: PerplexityBot, Bytespider.",
            "fix": "Remove the Disallow rules for the AI crawlers you want to be visible to.",
            "exampleUrls": ["https://example.com/robots.txt"]
        }
    ],
    "pagesAudited": 10,
    "llmsTxtDraft": "# Example Company\n\n> We help teams...\n\n## Pages\n\n- [Pricing](https://example.com/pricing): Plans for teams of every size...",
    "stats": { "avgScore": 58, "minScore": 31, "maxScore": 88, "avgWordCount": 743 }
}
```

**2. Named dataset `pages` — one `page` record per audited URL** (per-page detail): title, meta description, canonical, meta robots, headings, JSON-LD types and parse errors, Open Graph, word count, text-to-HTML ratio, per-crawler access verdict, per-page score (0–100) and issues. Pages that can't be fetched after retries produce a `page-error` record here. In Apify Console, open the run's **Storage → Datasets → pages** to browse them; via API:

```
GET https://api.apify.com/v2/datasets/{pagesDatasetId}/items?clean=true
```

If no page on a domain is reachable, the default dataset gets an `error` record explaining why.

### The generated llms.txt draft

When `llms.txt` is missing, the summary includes a complete draft built from your best pages (title + description + URL each), in llmstxt.org format. Save it as `llms.txt`, review, upload it to your domain root — instant fix for the most common GEO gap.

### Scoring

Category scores are weighted: crawlability 25, content structure 30, machine readability 25, discoverability 20 → overall 0–100. Issues subtract points by severity (critical −25, high −12, medium −6, low −2). Scoring is fully deterministic — the same site always gets the same score, so re-runs measure your real progress.

### How much does it cost?

Pay per event: you pay only for pages actually audited (plus a small summary event per domain). A 10-page audit of one domain costs a few cents — compare that to GEO SaaS platforms at $99–$500+/month or agency audits at $1,500+.

### Who is this for?

- **SEO/GEO agencies & freelancers** — add AI-readiness audits to your service offering
- **Marketing teams** — check whether ChatGPT/Perplexity can even see your site
- **Developers** — CI/monitoring integration via Apify API; deterministic scores for regression tracking
- **AI agents** — structured JSON output, predictable records, machine-readable verdicts

### Use with AI agents

This Actor is designed for agent consumption: documented JSON input schema, stable output records, no auth beyond the caller's own Apify token. The simplest integration is the run-sync endpoint, which starts the run and returns the results in one call:

```
POST https://api.apify.com/acts/dev-hoss~ai-search-readiness-audit/run-sync-get-dataset-items?token=YOUR_API_TOKEN
Content-Type: application/json

{"startUrls": ["https://example.com"], "maxPages": 10}
```

It also works with Apify's hosted MCP server (`mcp.apify.com`) — add the Actor to your MCP client and the input schema drives the tool call. Record shapes never change silently: `summary` records carry the score and fixes, `page` records live in the named `pages` dataset.

# Actor input Schema

## `startUrls` (type: `array`):

One or more website homepages (or any page on the site) to audit. Each distinct domain gets its own AI-readiness report and llms.txt draft. Maximum 10 domains per run.

## `maxPages` (type: `integer`):

Maximum number of pages crawled and audited per domain. More pages give a more complete picture (and cost more). Recommended: 10 for a quick check, 25-50 for a real audit.

## `maxConcurrency` (type: `integer`):

How many pages are fetched in parallel. Lower this (1-2) for small or rate-limiting sites.

## `respectRobotsTxt` (type: `boolean`):

If enabled, the audit crawler itself obeys robots.txt rules for its own user agent (auditing does not require this, but it is the polite default). Note: this does not affect the AI-crawler access checks, which are computed analytically for every major AI bot.

## `proxyConfiguration` (type: `object`):

Optional proxy settings. Usually not needed — most sites are happy to serve robots.txt and normal pages to a well-behaved crawler. Use residential proxies only for sites that aggressively rate-limit datacenter IPs.

## Actor input object example

```json
{
  "startUrls": [
    "https://apify.com"
  ],
  "maxPages": 10,
  "maxConcurrency": 5,
  "respectRobotsTxt": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `domainSummaries` (type: `string`):

One record per audited domain: 0-100 score, category scores, AI crawler matrix, prioritized fixes and the generated llms.txt draft.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://apify.com"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("dev-hoss/ai-search-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://apify.com"],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("dev-hoss/ai-search-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://apify.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call dev-hoss/ai-search-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,dev-hoss/ai-search-readiness-audit"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/wbVt7Q5h2ysedJpmp/builds/V4aexpR5Kwvn9baKk/openapi.json