# Website Audit & AI Readiness Checker (`professional_automation_tools1/website-audit-ai-readiness`) Actor

AI readiness score per site and SEO score per page: access of 20 AI crawlers (ChatGPT, Claude, Perplexity, Google, Bing), content visible without JavaScript, structured data, nosnippet, sitemap, llms.txt. Fix hints and HTML report.

- **URL**: https://apify.com/professional_automation_tools1/website-audit-ai-readiness.md
- **Developed by:** [Dominik Schlosser](https://apify.com/professional_automation_tools1) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 analyzed pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website Audit & AI Readiness Checker

Find out whether search engines **and AI assistants** (ChatGPT, Claude, Perplexity, Google AI Overviews, Copilot) can find, read and quote a website. The Actor crawls a site and returns:

- an **AI readiness score (0-100, grade A-F) per site** with a breakdown and the most important fixes,
- an **SEO score (0-100) per page** with a prioritized list of issues and a concrete fix for each one,
- a **client-ready HTML report** you can send as it is.

### What it checks

**AI crawler access (robots.txt)** for 20 agents, grouped by purpose:

| Purpose | Agents |
|---|---|
| AI search and assistants | OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Applebot, Amazonbot, DuckAssistBot, MistralAI-User, Meta-ExternalFetcher |
| Search engines that feed AI answers | Googlebot, Bingbot |
| Model training only | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider |

Each agent is classified for the home page as `allowed`, `partial` (home page open, some sections closed) or `blocked`. Rules are matched like real crawlers do (RFC 9309): groups for the same agent are merged, `*` and `$` wildcards are supported, the most specific rule wins. Blocking only training crawlers does **not** lower the score: that is a legitimate way to opt out of model training while staying visible in AI answers.

**Other site checks (once per site)**

- robots.txt reachable. A server error on robots.txt can make crawlers skip the whole site.
- XML sitemap: actually loaded and validated, number of entries
- `llms.txt` and `llms-full.txt`: present and well-formed (title, summary, links)
- HTTP redirects to HTTPS

**Page checks**

- **Content visible without JavaScript.** Many AI crawlers do not run JavaScript; client-side rendered pages look empty to them.
- **Structured data**: JSON-LD and microdata, invalid JSON, all schema.org types found, and **missing recommended properties** for Organization, LocalBusiness, WebSite, Article, NewsArticle, BlogPosting, Product, FAQPage, BreadcrumbList, Event, Recipe, HowTo and VideoObject
- `noindex` and `nosnippet` / `max-snippet:0` (meta robots, googlebot meta, X-Robots-Tag). `nosnippet` also keeps a page out of AI-generated answers.
- Title and meta description (missing, too short, too long), **duplicates across the site**
- H1 headings, canonical, `lang`, viewport, Open Graph, images without alt text, thin content
- HTTPS, redirects, response time, HTTP errors (4xx/5xx), last modified date

### AI readiness score

| Part | Points |
|---|---|
| AI search and assistant crawlers can access the site (weighted by reach: OpenAI 3, Anthropic 2, Perplexity 2, others 1) | 30 |
| Content readable without JavaScript | 20 |
| Google and Bing can crawl the site | 15 |
| Valid and complete structured data | 15 |
| Pages may be indexed and quoted (no noindex / nosnippet) | 10 |
| llms.txt present and well-formed | 5 |
| Valid XML sitemap | 5 |

Grades: A 90+, B 80+, C 65+, D 50+, F below 50. If site checks are switched off, the score is based on the page checks only.

### Why use it

- **Agencies**: audit many client sites in one run, send the HTML report, export the data to CSV/Excel/JSON.
- **Developers**: run it in CI or on a schedule after each release.
- **AI agents**: callable via the Apify MCP server to check a site on request.

### Input

| Field | Description | Default |
|---|---|---|
| `startUrls` | Sites to audit, usually the home page. `example.com` and `www.example.com` count as one site. | crawlee.dev |
| `maxPagesPerSite` | Max pages analyzed per site | 10 |
| `includeSiteChecks` | robots.txt, AI crawlers, sitemap, llms.txt, HTTPS | true |
| `maxConcurrency` | Parallel requests | 5 |

The crawler stays on the site, follows the redirect of a start URL (e.g. to `www.` or another domain), skips links to files (PDF, images, ...) and respects robots.txt.

### Output

**Dataset**, one item per:

- `type: "summary"`: per site, AI readiness score and breakdown, average SEO score, broken pages, duplicates, top issues (use the view "Summary")
- `type: "site"`: site checks including the access of every AI crawler
- `type: "page"`: page checks with score, issues and fixes
- `type: "skipped"`: a start URL that robots.txt does not allow to crawl

Example page item:

```json
{
  "type": "page",
  "url": "https://example.com/blog",
  "statusCode": 200,
  "score": 78,
  "issues": [
    { "code": "structured-data-incomplete", "severity": "medium", "message": "Structured data is missing recommended properties: Article (author, image).", "fix": "Add the missing properties so rich results and AI systems can use the data." }
  ],
  "jsDependent": false,
  "nosnippet": false,
  "structuredData": { "jsonLdBlocks": 1, "jsonLdInvalid": 0, "types": ["Article", "Person"], "microdata": false, "incomplete": [{ "type": "Article", "missing": ["author", "image"] }] }
}
```

**Key-value store**

- `OUTPUT`: the summaries of all sites as JSON
- `REPORT`: the HTML report (open it from the run's Storage tab)

### Pricing

Pay per event, no subscription: one event per analyzed page and one per site check. Error pages count as analyzed pages. Pages blocked by robots.txt, links that redirect to other sites, the summary and the HTML report are free.

### Data and privacy

The Actor only reads technical website data (HTML structure, meta tags, structured data types, robots.txt). It does **not** extract e-mail addresses, phone numbers, names or any other personal data.

### Limits

- HTML is analyzed without executing JavaScript, the same way most AI crawlers see a page. Content that only appears in the browser is reported as `js-rendered-content`.
- Core Web Vitals / Lighthouse are not part of this Actor.
- robots.txt tells well-behaved crawlers what to do. Some user-triggered AI fetchers may not follow it.

# Actor input Schema

## `startUrls` (type: `array`):

Start URLs, usually the home page of each site. example.com and www.example.com count as one site; the redirect of a start URL is followed.

## `maxPagesPerSite` (type: `integer`):

Maximum number of pages analyzed per site. Each analyzed page is one billable event (error pages included). Pages blocked by robots.txt and links that redirect to other sites are not charged.

## `includeSiteChecks` (type: `boolean`):

Run the site-level checks once per site. One billable event per site. Needed for the full AI readiness score.

## `maxConcurrency` (type: `integer`):

Parallel requests. Keep it low to be polite to the audited site.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ],
  "maxPagesPerSite": 10,
  "includeSiteChecks": true,
  "maxConcurrency": 5
}
```

# Actor output Schema

## `summary` (type: `string`):

AI readiness score, SEO average, broken pages and top issues per site.

## `results` (type: `string`):

Site checks and page checks with issues and fixes.

## `report` (type: `string`):

Client-ready report with scores, fixes, crawler access and all pages.

## `output` (type: `string`):

The per-site summaries as one JSON record.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("professional_automation_tools1/website-audit-ai-readiness").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://crawlee.dev" }] }

# Run the Actor and wait for it to finish
run = client.actor("professional_automation_tools1/website-audit-ai-readiness").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    }
  ]
}' |
apify call professional_automation_tools1/website-audit-ai-readiness --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,professional_automation_tools1/website-audit-ai-readiness"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/EjneI32qE36QzahMT/builds/mUjc3beUNtIC8H7iH/openapi.json