# AEO Audit: AI Crawler, Agent & llms.txt Readiness (`ruesandora/ai-agent-readiness-audit`) Actor

One audit for AI search and AI agent readiness: AI crawler access, readable content, structured data, llms.txt and sitemap, agent discovery files. One score 0-100, scores per section and a ranked fix list per site. Bulk URLs.

- **URL**: https://apify.com/ruesandora/ai-agent-readiness-audit.md
- **Developed by:** [Rues Forum](https://apify.com/ruesandora) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

$50.00 / 1,000 site auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Search + Agent Readiness Audit

Can AI assistants find, read and cite your site, and can AI agents discover what it offers? This Actor runs **one audit per site** and returns **one score (0-100)**, a score per section and a **ranked fix list**. Paste one site or a bulk list. No API keys, no login.

### What does it check?

Five sections, each with a handful of yes/no/partial checks on public files and the page itself:

- **AI crawler access**: whether common AI search, user-triggered and training crawlers are allowed by `robots.txt`.
- **Readable content and page basics**: main text present without JavaScript, title, meta description, canonical, H1, language, indexability.
- **Structured data**: JSON-LD present and valid, organization or site markup, page-type markup, required fields.
- **Discovery files**: `llms.txt` (optional), XML sitemap, sitemap declared in `robots.txt`.
- **Agent readiness**: public discovery files for agents, each shown as `pass` (found and well formed), `partial` (found but malformed or incomplete), `fail` (not found) or `unknown` (could not be read: blocked by `robots.txt`, access denied or server error). Covered: API catalog (RFC 9727), MCP server card (draft), A2A agent card, OpenAPI description, OAuth authorization server metadata (RFC 8414), OAuth protected resource metadata (RFC 9728) and discovery links in the home page `Link` header (RFC 8288). All of these only matter if you offer an API, an agent, OAuth or an MCP server, so they are marked `optional`: missing ones never lower your score, found ones count.

### Input

| Field | Description | Default |
|---|---|---|
| `urls` | 1-100 sites (home page or domain) | required |
| `urlsText` | Optional bulk text: many sites separated by new lines, spaces or commas | empty |
| `maxExtraPages` | More pages read from the sitemap besides the home page (0-5) | 2 |
| `language` | `en` or `tr` (titles and fixes) | `en` |

```json
{ "urls": ["https://example.com"], "maxExtraPages": 2, "language": "en" }
```

### Output

One row per site: `score`, `grade`, `sections[]` (each with `score` and `checks[]`: `id`, `status` = pass / partial / fail / unknown, `optional`, `evidence`, `title`, `fix`), `fixes[]` (ranked, up to 10), `aiCrawlers`, `aiCrawlersBlocked`, `pagesChecked`. A final `summary` row (also saved as `SUMMARY`) with the average score and the most common fixes.

`unknown` means the check could not be measured (for example `robots.txt` was unreachable). It is left out of the score. The score is a readiness indicator for public signals. It is not a ranking, citation or traffic prediction.

### Pricing (pay per event)

See the Pricing tab: one event per site that could be audited. Unreachable sites and sites whose `robots.txt` blocks the crawler are not charged. You can set a maximum cost per run and the Actor stops when it is reached.

### Safety and politeness

At most 22 requests per site, one every 0.5 seconds, own User-Agent, `robots.txt` respected (a site that disallows the home page is skipped and not charged). Every request and every redirect is checked: private, loopback, link-local and metadata addresses are refused. No cookies, no login, no personal data. Only public files are read.

Full report: https://ruesandora.com

# Actor input Schema

## `urls` (type: `array`):

Sites to audit (1-100). Home page URL or domain, for example https://example.com. Public http(s) sites only; private and local addresses are rejected.

## `urlsText` (type: `string`):

Optional. Paste many sites separated by new lines, spaces or commas. Merged with the list above, one audit per site.

## `maxExtraPages` (type: `integer`):

Besides the home page, how many more pages from the sitemap are read (same host, robots.txt respected).

## `language` (type: `string`):

Language of titles and fixes.

## Actor input object example

```json
{
  "urls": [
    "https://example.com"
  ],
  "maxExtraPages": 2,
  "language": "en"
}
```

# Actor output Schema

## `results` (type: `string`):

Audit rows pushed to the default dataset. Views: sites.

## `summary` (type: `string`):

Run summary JSON stored as the SUMMARY record in the default key-value store.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("ruesandora/ai-agent-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://example.com"] }

# Run the Actor and wait for it to finish
run = client.actor("ruesandora/ai-agent-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com"
  ]
}' |
apify call ruesandora/ai-agent-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ruesandora/ai-agent-readiness-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/rf7KYBq0DlPlxTgMf/builds/gb9bsKhK6scwq8Jwo/openapi.json