# SEO & AI Search Readiness Audit (robots, llms.txt, schema) (`errandkit/seo-ai-readiness-audit`) Actor

Audit web pages for SEO and AI-search readiness: titles, meta, headings, canonical, images, links, JSON-LD, plus which AI crawlers robots.txt blocks, sitemap and llms.txt checks. 0-100 scores with fix hints. No browser, no API keys, built for AI agents.

- **URL**: https://apify.com/errandkit/seo-ai-readiness-audit.md
- **Developed by:** [Killis Haraway](https://apify.com/errandkit) (community)
- **Categories:** SEO tools, Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $4.00 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## SEO & AI Search Readiness Audit (robots.txt, llms.txt, schema)

Audit any web page for **classic SEO** and **AI-search readiness** in one pass. You get a **0-100 SEO score**, a **0-100 AI-readiness score**, and a list of issues, each with a stable code, a severity and a one-line fix.

It also checks the things AI assistants care about and most SEO tools skip:

- **Which AI crawlers your robots.txt blocks**: AI *search* crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot …) and AI *training* crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended …), kept apart so you can tell a policy choice from a mistake.
- **llms.txt**: whether `/llms.txt` exists and follows the llmstxt.org format.
- **JavaScript-only pages**: most AI crawlers don't run JavaScript. If a page's HTML is an empty app shell, you'll see `JS_ONLY_CONTENT`.
- **Structured data**: JSON-LD types found, broken JSON-LD, missing `@context`, missing article dates.

Plain HTTP, no browser, no API keys. It's fast and cheap, and it was built to be called by AI agents.

### What you get

**One item per page** (`type: "page"`):

| Field | What it is |
|---|---|
| `seo_score`, `ai_score` | 0-100. Each category starts at 100 and loses 15 per high, 7 per medium and 3 per low issue |
| `issues` | `[{code, category, severity, message, fix, details}]`, e.g. `TITLE_TOO_LONG`, `NOINDEX`, `JS_ONLY_CONTENT` |
| `title`, `meta_description`, `h1`, `headings`, `word_count`, `lang` | On-page basics |
| `canonical`, `meta_robots`, `x_robots_tag`, `indexable`, `hreflang` | Indexing signals |
| `open_graph`, `twitter_card`, `structured_data_types`, `jsonld_blocks` | Social previews and schema.org |
| `images`, `images_missing_alt`, `links_internal`, `links_external`, `broken_links` | Content and links |
| `http_status`, `response_ms`, `html_bytes`, `redirects` | Transport |

**One item per site** (`type: "site"`, when *Site checks* is on): `robots_txt`, `ai_crawlers` (each crawler's purpose and whether it may fetch the homepage), `ai_search_bots_blocked`, `ai_training_bots_blocked`, `sitemap` (found, valid, entry count), `llms_txt` (found, valid, title, sections, links, problems) and site-level issues.

A readable **Markdown report** is saved as `REPORT` in the key-value store, and a run summary as `OUTPUT` (averages, most common issues, error codes).

### Quick start

**In the Console:** paste one or more URLs and click **Start**.

**API (Python):**

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("errandkit/seo-ai-readiness-audit").call(run_input={
    "urls": ["https://example.com/", "https://example.com/pricing"],
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["type"] == "site":
        print("AI search bots blocked:", item["ai_search_bots_blocked"], "llms.txt:", item["llms_txt"]["found"])
    else:
        print(item["url"], item["seo_score"], item["ai_score"], [i["code"] for i in item["issues"]])
```

**Crawl part of a site and check links** (follows internal links up to 50 pages, checks up to 100 links per page):

```json
{
  "urls": ["https://example.com/"],
  "maxPagesPerSite": 50,
  "checkLinks": true
}
```

**Example page item (trimmed):**

```json
{
  "type": "page",
  "url": "https://example.com/",
  "status": "ok",
  "seo_score": 70,
  "ai_score": 93,
  "title": "Example Domain",
  "word_count": 26,
  "issues": [
    {"code": "META_DESCRIPTION_MISSING", "category": "seo", "severity": "medium",
     "message": "No meta description.", "fix": "Add a meta description of about 70-160 characters."},
    {"code": "STRUCTURED_DATA_MISSING", "category": "ai", "severity": "medium",
     "message": "No JSON-LD or microdata found.", "fix": "Add JSON-LD (schema.org) describing the page: ..."}
  ]
}
```

### Use it from an AI agent (MCP)

This Actor works as a tool in Claude, Cursor, ChatGPT connectors and other MCP clients through Apify's MCP server:

```
https://mcp.apify.com/?actors=errandkit/seo-ai-readiness-audit
```

Ask things like "Audit example.com and tell me the three most important fixes" or "Can ChatGPT search and Perplexity crawl my site?". Results are JSON with stable codes, so the agent doesn't have to guess.

### Input options

| Option | Default | Notes |
|---|---|---|
| `urls` | – | Pages to audit. `example.com` is read as `https://example.com/` |
| `maxPagesPerSite` | 1 | 1 = only the URLs you give. Higher = also follow internal links on the same site, up to this many pages per site |
| `checkSite` | `true` | Once per site: robots.txt, AI crawler access, XML sitemap, llms.txt |
| `checkLinks` | `false` | Request each link on audited pages (HEAD, then GET if needed) and report broken ones |
| `maxLinksPerPage` | 100 | Cap for link checking (max 200) |
| `saveReport` | `true` | Save the Markdown `REPORT` |

### Pricing

Pay per event. Failed and skipped pages are free.

| Event | Price | Per 1,000 |
|---|---|---|
| Page audited (HTML page fetched and analyzed) | $0.004 | $4.00 |
| Site checked (robots.txt, AI crawlers, sitemap, llms.txt; once per site) | $0.005 | $5.00 |
| Link checked (only with `checkLinks`; each URL once per run) | $0.0005 | $0.50 |
| Actor start (Apify standard) | $0.00005 | $0.05 |

Examples: one page + its site check costs **$0.009**. A 50-page crawl of one site costs **$0.205**; with link checking of ~2,000 unique links, add $1.00. Pages that time out, return 4xx/5xx, aren't HTML or are blocked by robots.txt are **not charged**. Set *Maximum cost per run* and the Actor stops before going over it, keeping everything already finished.

### Error codes (per page)

| Code | Meaning |
|---|---|
| `HTTP_ERROR` | The page returned 4xx or 5xx (status in `http_status`) |
| `NOT_HTML` | The URL is a PDF, image, JSON … not an HTML page |
| `ROBOTS_DISALLOWED` | robots.txt doesn't allow our crawler there, so it wasn't fetched (`status: "skipped"`) |
| `FETCH_FAILED`, `TIMEOUT`, `TOO_MANY_REDIRECTS` | Network problems |
| `URL_NOT_ALLOWED` | Not http(s), or a private network address |
| `MAX_COST_REACHED` | Your maximum cost per run was reached |

### How it crawls (good-citizen rules)

- It always obeys **robots.txt** for its own user agent `ErrandkitAuditBot` (and `*`), including `Crawl-delay` up to 5 s. If robots.txt returns a server error, it crawls nothing on that site, as Google does.
- At most one request every 0.5 s per site, at most 5 sites at a time.
- It only fetches the pages you list and, if you ask, internal links on the same site. It never logs in, never submits forms and doesn't collect personal data (no emails or phone numbers).

### Good to know

- **No JavaScript is run.** That's deliberate: it shows what search and AI crawlers that don't render JavaScript see. Core Web Vitals and Lighthouse scores aren't measured.
- Scores are a simple, transparent sum of issue weights, not a guess at rankings. Use the `issues` list to decide what to fix.
- Link checking requests each external link once per run (HEAD, falling back to GET for servers that refuse HEAD). `429 Too Many Requests` is not counted as broken.

### Privacy

Pages are fetched and analyzed inside your own Apify run. Nothing is sent to any third-party service or AI model, and nothing is kept by the developer.

### Feedback

Found a false positive? Open an issue on the Actor's *Issues* tab with the URL.

Part of the Errandkit catalog of data tools, APIs and MCP servers for AI agents: https://errandkit.com

# Actor input Schema

## `urls` (type: `array`):

Pages to audit. A bare domain like example.com is read as https://example.com/.

## `maxPagesPerSite` (type: `integer`):

1 = audit only the URLs above. Higher values also follow internal links on the same site until this many pages per site (the URLs above always count).

## `checkSite` (type: `boolean`):

Once per site: which AI search and training crawlers robots.txt blocks, whether a valid XML sitemap and llms.txt exist.

## `checkLinks` (type: `boolean`):

Request every link on each audited page (HEAD, then GET if needed) and report broken ones. Each URL is checked once per run.

## `maxLinksPerPage` (type: `integer`):

Only used when link checking is on.

## `saveReport` (type: `boolean`):

Also save a readable REPORT (Markdown) in the key-value store.

## Actor input object example

```json
{
  "urls": [
    "https://docs.apify.com/"
  ],
  "maxPagesPerSite": 1,
  "checkSite": true,
  "checkLinks": false,
  "maxLinksPerPage": 100,
  "saveReport": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `report` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://docs.apify.com/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("errandkit/seo-ai-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://docs.apify.com/"] }

# Run the Actor and wait for it to finish
run = client.actor("errandkit/seo-ai-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://docs.apify.com/"
  ]
}' |
apify call errandkit/seo-ai-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,errandkit/seo-ai-readiness-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/kDjkSODhGSmgMy5gV/builds/oSphxWvQWrS21qYKX/openapi.json