# Website AI-Readiness Audit (`madpai/ai-readiness-audit`) Actor

Enter a domain to get a 0–100 score, a grade and a prioritized list of fixes. It checks AI crawler access in robots.txt, llms.txt, your sitemap, HTTPS, and samples pages for text that loads without JavaScript, JSON-LD, titles and headings. Includes a Markdown report.

- **URL**: https://apify.com/madpai/ai-readiness-audit.md
- **Developed by:** [William Schultz](https://apify.com/madpai) (community)
- **Categories:** SEO tools, AI
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$8.00 / 1,000 site audits

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website AI-Readiness Audit

**Give it a domain. Get a 0–100 AI-readiness score, a letter grade and the specific fixes, from robots.txt, llms.txt, the sitemap and sampled pages.**

No API key, no login. $8 per 1,000 audits ($0.008 each), rising to $20 per 1,000 on 17 Nov 2026; sites that are unreachable, blocked or invalid get a free item that says why.

### What you get

- What robots.txt tells 16 known AI crawlers, plus checks for `/llms.txt`, a sitemap and HTTPS. Training bots are reported but don't affect the score.
- Sampled pages checked for text in the raw HTML (no JavaScript), JSON-LD, title and meta description, headings, noindex/noai directives, dates and canonical links.
- Structured JSON plus a ready-to-share Markdown report (`reportMarkdown`). The score is our own heuristic checklist, not a prediction of AI rankings or citations.

### Quick start

1. Open this Actor in Apify Console.
2. On the Input tab, switch to the JSON view and paste the input below.
3. Click **Start**. When the run finishes, open the **Output** tab to view or export the results.

Input: one audit of crawlee.dev with 3 sampled pages, so $0.008.

```json
{
  "urls": [
    "crawlee.dev"
  ],
  "maxPagesPerSite": 3
}
```

### Sample output

From cloud run etWPp2rSYEZEyg4Vh (2 Nov 2026), the single item this exact input returned. `findings` is cut to its 1 failing check (of 14), and `input`, `domain`, `redirectedTo`, `aiCrawlers`, `llmsTxt`, `llmsFullTxt`, `pages`, `reportMarkdown`, `error` and `auditedAt` are left out. That cloud run took about 8.5 seconds.

```json
{
  "url": "https://crawlee.dev/",
  "score": 90,
  "grade": "A",
  "summary": "crawlee.dev scores 90/100 (A). 1 failing and 2 warning checks.",
  "findings": [
    {
      "id": "freshness",
      "category": "Content",
      "weight": 3,
      "status": "fail",
      "message": "33% of 3 sampled page(s) expose a published or modified date.",
      "fix": "Expose dateModified in JSON-LD or article:modified_time."
    }
  ],
  "robotsTxtFound": true,
  "sitemapUrl": "https://crawlee.dev/sitemap.xml",
  "pagesAudited": 3,
  "errorCode": null
}
```

### API

Same input as above. Every example reads your token from the `APIFY_TOKEN` environment variable; never paste a real token into code you share.

#### curl

```bash
curl -X POST "https://api.apify.com/v2/acts/madpai~ai-readiness-audit/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls": ["crawlee.dev"], "maxPagesPerSite": 3}'
```

#### JavaScript (`apify-client`)

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const input = {
  "urls": [
    "crawlee.dev"
  ],
  "maxPagesPerSite": 3
};

const run = await client.actor('madpai/ai-readiness-audit').call(input);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const item of items) {
    console.log(item.url, item.score, item.grade);
}
```

#### Python (`apify-client`)

```python
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run_input = {
  "urls": [
    "crawlee.dev"
  ],
  "maxPagesPerSite": 3
}

run = client.actor('madpai/ai-readiness-audit').call(run_input=run_input)
for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item.get('url'), item.get('score'), item.get('grade'))
```

#### Run summary

The run also writes an `OUTPUT_SUMMARY` record to its default key-value store: `await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('OUTPUT_SUMMARY')` (Python: `client.key_value_store(run['defaultKeyValueStoreId']).get_record('OUTPUT_SUMMARY')`).

With curl, `run-sync-get-dataset-items` returns only the items, not the run ID, so it can't read `OUTPUT_SUMMARY`. To get the summary from curl, start the run and wait for it to finish, then fetch the record from the run's key-value store (needs `jq`):

```bash
RUN=$(curl -s -X POST "https://api.apify.com/v2/acts/madpai~ai-readiness-audit/runs?waitForFinish=300" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"urls": ["crawlee.dev"], "maxPagesPerSite": 3}')
KVS=$(echo "$RUN" | jq -r '.data.defaultKeyValueStoreId')
DS=$(echo "$RUN" | jq -r '.data.defaultDatasetId')
curl -s "https://api.apify.com/v2/key-value-stores/$KVS/records/OUTPUT_SUMMARY" -H "Authorization: Bearer $APIFY_TOKEN"
curl -s "https://api.apify.com/v2/datasets/$DS/items?format=json" -H "Authorization: Bearer $APIFY_TOKEN"
```

### How it works

It checks how easy it is for AI search and answer engines to reach and read a website, then gives a 0–100 score, a letter grade, and a list of specific fixes. Per site it checks:

- **Crawler access:** what `robots.txt` says to 16 known AI crawlers. Search and user-fetch bots count toward the score; training bots are reported but don't affect it, since blocking them is a legitimate choice.
- **AI guidance:** whether `/llms.txt` (and the optional `/llms-full.txt`) exists.
- **Discoverability and basics:** sitemap, HTTPS.
- **Content (sampled pages):** whether main text is in the raw HTML without JavaScript, JSON-LD structured data, title and meta description, heading structure, noindex/noai directives, dates and canonical links.

### What the score is and isn't

The score is our heuristic: our own weighted checklist of technical signals. It shows what you can fix; it does not measure or guarantee whether any AI product will crawl, cite or rank you.

### Who is it for?

SEO and content teams preparing for AI search, agencies running quick audits for clients, and AI agents that need a structured readiness check.

### Input

| Field | Default | What it does |
|---|---|---|
| `urls` (required) | prefill `crawlee.dev` | Domains or homepages to audit. One audit, one charge, per site. Different spellings of the same site (`crawlee.dev`, `https://crawlee.dev/`, `www.crawlee.dev`) are audited and charged once. |
| `maxPagesPerSite` | 5 (max 25) | Homepage plus pages spread across the sitemap. |
| `includePageDetails` | true | Add per-page checks to each result. |
| `respectRobotsTxt` | true | Don't sample pages robots.txt disallows. Can only be turned off together with `ownSiteConfirmed`. |
| `requestDelayMs` | 1000 | Delay between requests to one site (about 1 request per second, longer if robots.txt sets a Crawl-delay). Shorter delays need `ownSiteConfirmed`. |
| `maxConcurrency` | 3 (max 10) | Sites audited in parallel. Each site still gets about 1 request per second. |
| `ownSiteConfirmed` | false | Tick only for sites you own or are authorised to crawl. Unlocks turning off robots.txt and delays under 1 second. |

Minimal input:

```json
{ "urls": ["crawlee.dev"] }
```

### Output

One item per site: `score`, `grade`, `summary`, `findings` (each with category, weight, status, message and fix), `aiCrawlers`, `robotsTxtFound`, `llmsTxt`, `llmsFullTxt`, `sitemapUrl`, `pagesAudited`, `pages`, `reportMarkdown`, `redirectedTo`, `error`, `errorCode`. Real excerpt from a local test run (1 Nov 2026):

```json
{
  "domain": "example.com",
  "score": 38,
  "grade": "F",
  "summary": "example.com scores 38/100 (F). 8 failing and 2 warning checks.",
  "findings": [
    { "id": "sitemap", "category": "Discoverability", "weight": 10, "status": "fail",
      "message": "No sitemap found via robots.txt or common paths.",
      "fix": "Publish /sitemap.xml and reference it in robots.txt." }
  ]
}
```

`reportMarkdown` is a ready-to-share report you can paste into a doc or hand to an LLM.

If the homepage redirects to another host, that host is audited instead: `redirectedTo` shows where it went and a `redirected` finding says so.

Sites that aren't audited still get an item with an `error` message, and aren't charged. Every one also has an `errorCode`. From the same local run:

```json
{ "input": "nytimes.com", "url": "https://nytimes.com/", "error": "Homepage unreachable: HTTP 403", "errorCode": "http-error" }
```

| `errorCode` | Meaning |
|---|---|
| `robots-disallowed` | robots.txt disallows the homepage (or a redirect hop to it) for our crawler. |
| `robots-unavailable` | robots.txt couldn't be fetched (server error, 429 or network error). Under RFC 9309 that means "disallow everything", so the site isn't audited. |
| `redirect-signin` | The homepage redirects to a sign-in page. We never log in. |
| `blocked-private` | The address is private, loopback, link-local or otherwise reserved. It's never requested. |
| `http-error` / `unreachable` | The homepage returned an error status (for example 403) or couldn't be reached. A domain that doesn't resolve is `unreachable` with "DNS lookup failed (ENOTFOUND)", and nothing is requested, not even robots.txt. |
| `invalid-input` | The input isn't a valid URL or domain, so nothing was requested. |
| `internal-error` | The audit stopped on an unexpected error. Please report it. |

### How it treats websites

- robots.txt is read before any other request to a host, following RFC 9309. llms.txt, llms-full.txt, sitemap probes and sampled pages are all checked against it.
- About 1 request per second per host, or the robots.txt Crawl-delay if that's longer (up to 30 s), counted from the robots.txt request itself. Retry-After is honoured, and after 3 HTTP 429 responses in a row the host gets no further requests.
- Redirects are followed one hop at a time, and each hop is checked against its own host's robots.txt.
- `llms-full.txt` is never downloaded in full: only the first 256 KB is read, and its size comes from the response headers.
- It reads public pages and files only. No logins, no CAPTCHA solving, no proxies.

### How long does it take?

Local test numbers (17 fixture runs, 1 Nov 2026): a run took 6.7 s at the median and 30.7 s at the slowest, with a peak of 165 MB against the 256 MB default memory. Because requests to one host are spaced about 1 second apart, more sampled pages or a robots.txt Crawl-delay make an audit slower.

### How much does it cost?

Pay per event: one `site-audit` event per completed audit, at $8 per 1,000 audits ($0.008 per audit). Example: auditing 50 sites costs $0.40. Sites that are unreachable, blocked (by robots.txt, a 403 or a sign-in redirect) or on a private address are saved with an error and not charged. Duplicate spellings of a site are audited and charged once. The price rises to $20 per 1,000 audits ($0.02 per audit) on 17 November 2026; until then it's $8 per 1,000.

### Use with AI agents

Designed to run with limited permissions. Output is structured JSON plus a Markdown report, so an agent can read the score and the fixes directly. Example prompt: "Audit example.com for AI readiness and list the top three fixes."

### Limits

- It samples a few pages, not the whole site, so the content score is an estimate for the site.
- Sites that block automated requests (for example with a 403) return an error, which is never bypassed.
- It reads raw HTML only and doesn't run JavaScript.
- The list of AI crawlers reflects what's publicly documented as of the release date; new bots appear often.

### FAQ

**Does a high score mean AI products will crawl, cite or rank my site?** No. The score is our own weighted checklist of technical signals. It shows what you can fix; it doesn't measure or guarantee what any AI product does.
**Does blocking AI training bots lower my score?** No. Training bots are reported, but only search and user-fetch bots count toward the score, since blocking training bots is a legitimate choice.
**Am I charged for sites that can't be audited?** No. Unreachable sites, sites blocked by robots.txt, a 403 or a sign-in redirect, private addresses and invalid inputs are saved with an `error` and an `errorCode`, and aren't charged. Different spellings of the same site are audited and charged once.
**Does it run JavaScript?** No. It reads raw HTML only, which is why it checks whether a page's main text is there without JavaScript.
**What if my homepage redirects to another domain?** That host is audited instead. `redirectedTo` shows where it went, and a `redirected` finding says so.

### Run it on a schedule

To track the score after you make fixes, create a schedule for this Actor in Apify Console (Schedules), for example weekly, with the same input.

### Related Actors

- [Sitemap Extractor & URL Discovery](https://apify.com/madpai/sitemap-url-discovery): list the URLs a site publishes in its sitemaps.
- [Broken Link & Redirect Checker](https://apify.com/madpai/broken-link-checker): find broken links and redirect chains on the site.

### Legal

Reads public pages and files only, respects robots.txt by default, and collects no personal data. You're responsible for using it in line with each site's terms. Not affiliated with any AI company or crawler operator named in the results.

### Changelog

See CHANGELOG.md.

### Support

Report problems in the Issues tab. We aim to reply within 24 hours.

# Changelog

This Actor's version history is a separate document: https://apify.com/madpai/ai-readiness-audit/changelog.md

# Actor input Schema

## `urls` (type: `array`):

Domains or homepage URLs to audit. One audit (and one charge) per site.

## `maxPagesPerSite` (type: `integer`):

Homepage plus pages spread across the sitemap. More pages = a more representative score, slightly slower.

## `includePageDetails` (type: `boolean`):

Add the per-page checks (title, JSON-LD types, word count, robots directives) to each result.

## `respectRobotsTxt` (type: `boolean`):

Don't sample pages that robots.txt disallows. Can only be turned off together with "I own these sites".

## `requestDelayMs` (type: `integer`):

Per-site delay between requests (about 1 request per second by default). robots.txt Crawl-delay is honoured if longer. Values under 1000 ms only apply with "I own these sites" checked.

## `maxConcurrency` (type: `integer`):

How many sites to audit at once.

## `ownSiteConfirmed` (type: `boolean`):

Only check this for your own sites. It allows turning off robots.txt and using delays under 1 second.

## Actor input object example

```json
{
  "urls": [
    "crawlee.dev"
  ],
  "maxPagesPerSite": 5,
  "includePageDetails": true,
  "respectRobotsTxt": true,
  "requestDelayMs": 1000,
  "maxConcurrency": 3,
  "ownSiteConfirmed": false
}
```

# Actor output Schema

## `audits` (type: `string`):

Dataset with one item per input site: input, url, domain, score (0-100), grade, summary, findings (prioritized fixes), aiCrawlers, robotsTxtFound, llmsTxt, llmsFullTxt, sitemapUrl, pagesAudited, pages, reportMarkdown, error, errorCode, redirectedTo, auditedAt. Sites that could not be audited carry error and errorCode and are not charged.

## `summary` (type: `string`):

JSON summary: items pushed, sites audited, failed sites, sites skipped for budget, finish time.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "crawlee.dev"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("madpai/ai-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["crawlee.dev"] }

# Run the Actor and wait for it to finish
run = client.actor("madpai/ai-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "crawlee.dev"
  ]
}' |
apify call madpai/ai-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,madpai/ai-readiness-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/cIIWbGNZVRwKb1wlW/builds/mqD3qv1Lh0LWtnx3F/openapi.json