# SEO Audit & AI Search Readiness Checker (GPTBot, llms.txt) (`jtpalms/seo-ai-readiness-audit`) Actor

Crawl a site and audit every page for SEO basics (title, meta, H1, canonical, hreflang, noindex, alt text, JSON-LD, status, speed) plus AI search readiness: which AI crawlers robots.txt blocks, llms.txt, AI meta tags. One row per page, one per site. USD 2 per 1,000 pages.

- **URL**: https://apify.com/jtpalms/seo-ai-readiness-audit.md
- **Developed by:** [JT Palms](https://apify.com/jtpalms) (community)
- **Categories:** SEO tools, Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.60 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## SEO Audit & AI Search Readiness Checker

A technical SEO audit of every page, plus the new question: **can ChatGPT, Claude, Perplexity and other AI search engines see this site?** Give it one site or a thousand. For each site you get one row with its AI crawler access (robots.txt rules for 18 AI bots, llms.txt, Content-Signal), and for each page one row with the on-page SEO basics and a list of issue codes.

**USD 2 per 1,000 pages, USD 3 per 1,000 sites.** Pages that fail to load are free.

### What people use it for

- **Client audits.** Crawl a client's site and hand over a sortable list of pages with missing titles, long meta descriptions, missing H1s, wrong canonicals, `noindex` and missing alt text.
- **Release checks.** Run it after every deploy and catch the page that suddenly got `noindex` or lost its title.
- **AI visibility checks.** See at a glance which AI bots a site's robots.txt blocks, whether that is on purpose, and whether the site publishes an `llms.txt`.
- **Bulk AI-access check over a domain list.** Turn on **AI checks only** and paste a list of domains: one site row per domain, no page crawl, USD 3 per 1,000 domains.

### Sample output

Real rows from a run on 2026-10-05.

Site rows:

| host | AI bots blocked | allowed | partial | llms.txt | llms-full.txt | robots.txt |
|---|---|---|---|---|---|---|
| crawlee.dev | 0 | 18 | 0 | 200 (30,542 bytes) | 200 (2,886,889 bytes) | 200 |
| apify.com | 0 | 18 | 0 | 200 (16,837 bytes) | 404 | 200 (Content-Signal: search=yes, ai-input=yes, ai-train=yes) |
| www.nytimes.com (AI checks only) | 16 | 0 | 2 (Amazonbot, MistralAI-User) | 404 | 404 | 200 (8,547 bytes) |

Page rows:

| url | status | title | words | ms | issues |
|---|---|---|---|---|---|
| https://crawlee.dev/ | 200 | Crawlee · Build reliable crawlers. Fast. | 282 | 67 | no-jsonld |
| https://crawlee.dev/python | 200 | Crawlee for Python · Fast, reliable Python web crawlers. | 677 | 15 | canonical-other, no-jsonld |
| https://crawlee.dev/blog | 200 | Crawlee Blog - learn how to build better scrapers | Crawlee for JavaScript · Build reliable crawlers. Fast. | 1,597 | 15 | title-too-long, no-h1 |
| https://apify.com/actors | 200 | Automate anything with Apify Actors · Apify | 737 | 15 | description-too-long |

One site row in full (the `aiBots` object is cut to three of its 18 bots here):

```json
{
  "type": "site",
  "host": "crawlee.dev",
  "robotsTxtStatus": 200,
  "robotsTxtBytes": 103,
  "sitemaps": ["https://crawlee.dev/sitemap.xml", "https://crawlee.dev/python/sitemap.xml"],
  "contentSignals": null,
  "crawlDelaySecs": null,
  "aiBots": {
    "GPTBot": { "verdict": "allowed", "matchedGroup": "*" },
    "ClaudeBot": { "verdict": "allowed", "matchedGroup": "*" },
    "PerplexityBot": { "verdict": "allowed", "matchedGroup": "*" }
  },
  "aiBlockedCount": 0,
  "aiAllowedCount": 18,
  "aiPartialCount": 0,
  "llmsTxtStatus": 200,
  "llmsTxtBytes": 30542,
  "llmsTxtTitle": "Crawlee for Python · Fast, reliable Python web crawlers.",
  "llmsTxtLinkCount": 361,
  "llmsFullTxtStatus": 200,
  "llmsFullTxtBytes": 2886889,
  "checkedAt": "2026-10-05T16:39:19.598Z",
  "error": null
}
```

One page row in full:

```json
{
  "type": "page",
  "url": "https://crawlee.dev/",
  "finalUrl": "https://crawlee.dev/",
  "status": 200,
  "responseMs": 67,
  "redirects": 0,
  "bytes": 53556,
  "title": "Crawlee · Build reliable crawlers. Fast.",
  "titleLength": 40,
  "metaDescription": "Crawlee helps you build and maintain your crawlers. It's open source, but built by developers who scrape millions of pages every day for a living.",
  "metaDescriptionLength": 146,
  "h1Count": 1,
  "h1": "Build reliable web scrapers. Fast.",
  "canonical": "https://crawlee.dev/",
  "canonicalStatus": "self",
  "metaRobots": null,
  "xRobotsTag": null,
  "noindex": false,
  "aiDirectives": [],
  "lang": "en",
  "hreflangCount": 2,
  "ogTitle": "Crawlee for JavaScript · Build reliable crawlers. Fast.",
  "ogImage": true,
  "jsonLdTypes": [],
  "jsonLdErrors": 0,
  "imageCount": 14,
  "imagesMissingAlt": 0,
  "wordCount": 282,
  "internalLinks": 17,
  "externalLinks": 10,
  "issues": ["no-jsonld"],
  "checkedAt": "2026-10-05T16:39:19.100Z",
  "error": null
}
```

### Input

| Field | What it does |
|---|---|
| `startUrls` | Sites or pages to audit. Each host is audited once for AI access. In the default crawl mode, if the first start URL of a host redirects to another host (for example to `www.`), that host is the one audited. |
| `maxPagesPerSite` | Pages fetched per site (default 50, up to 10,000). Pages blocked by robots.txt and non-HTML files do not count. |
| `useSitemap` | Take page URLs from the sitemaps listed in robots.txt (or `/sitemap.xml` when none is listed) instead of following links. |
| `sameHostOnly` | On by default: follow links only on the site's own host. Off: also follow links to other subdomains of the same domain, sharing the same page limit. Links to other domains are never followed. |
| `aiChecksOnly` | Only the site row, no page crawl. The cheapest mode for long domain lists. |
| `maxConcurrency` | Requests in flight for the whole run (default 5). Per host the limits below always apply. |
| `outputIssuesOnly` | Only page rows that have at least one issue (or an error). Clean pages are left out and not charged. |

### Site row fields

| Field | Meaning |
|---|---|
| `host` | The host audited. |
| `robotsTxtStatus`, `robotsTxtBytes` | HTTP status of `/robots.txt` (after up to 5 redirects) and its size. |
| `sitemaps` | `Sitemap:` lines from robots.txt. |
| `contentSignals` | `Content-Signal:` lines from robots.txt, if any (for example `{"search": "yes", "ai-train": "no"}`). |
| `crawlDelaySecs` | The `Crawl-delay` that applies to this crawler, if any. |
| `aiBots` | For each AI bot: `verdict` is `allowed`, `blocked` (the whole site), `partial` (some paths blocked, or only some allowed) or `unknown` (robots.txt answered 5xx); `matchedGroup` is the bot's own name when robots.txt has a group for it, `*` when the catch-all group decides, or `null` when there is no robots.txt or no group applies. |
| `aiBlockedCount`, `aiAllowedCount`, `aiPartialCount` | Counts over the 18 bots. |
| `llmsTxtStatus`, `llmsTxtBytes`, `llmsTxtTitle`, `llmsTxtLinkCount` | `/llms.txt`: HTTP status, size, first `#` heading and number of Markdown links. Status is `soft-404` when the site answers with an HTML page instead of a text file, and `skipped-by-robots` when robots.txt keeps this crawler out. |
| `llmsFullTxtStatus`, `llmsFullTxtBytes` | The same for `/llms-full.txt` (size counted up to 50 MB). |
| `error` | Why the site could not be checked (DNS failure, private address, robots.txt 5xx). |

When robots.txt answers 404 (or another 4xx), every bot is `allowed`, as Google treats it. When it answers 5xx or 429, the verdicts are `unknown` and no pages are crawled.

### Page row fields and issue codes

Each page row has `url`, `finalUrl`, `status`, `responseMs` (time to response headers, summed over redirects), `redirects`, `bytes`, `title`, `metaDescription`, `h1Count`, `h1`, `canonical`, `canonicalStatus` (`self`, `other` or `missing`), `metaRobots`, `xRobotsTag`, `noindex`, `aiDirectives` (`noai`, `noimageai` found in meta tags or X-Robots-Tag), `lang`, `hreflangCount`, `ogTitle`, `ogImage`, `jsonLdTypes`, `jsonLdErrors`, `imageCount`, `imagesMissingAlt`, `wordCount` (visible text), `internalLinks`, `externalLinks`, `issues` and `error`.

| Issue code | When |
|---|---|
| `missing-title` | No `<title>` or an empty one. |
| `title-too-long` | Title longer than 60 characters. |
| `missing-description` | No meta description. |
| `description-too-long` | Meta description longer than 160 characters. |
| `no-h1` / `multiple-h1` | No `<h1>`, or more than one. |
| `noindex` | `noindex` or `none` in meta robots or the X-Robots-Tag header. |
| `canonical-other` | The canonical URL points to a different URL. |
| `missing-canonical` | No canonical link. |
| `no-jsonld` | No JSON-LD structured data block that parses. |
| `missing-alt` | At least one `<img>` without an `alt` attribute (`alt=""` counts as present). |
| `thin-content` | Fewer than 200 words of visible text. |
| `possibly-js-rendered` | Fewer than 50 words and 5 or more scripts: the content is probably built by JavaScript, which this actor does not run. |
| `slow` | More than 1,500 ms to the response headers. |
| `http-error` | Status 400 or higher (the only issue reported for such pages). |

### AI bots checked (list updated 2026-10-05)

| Bot | Owner | Purpose |
|---|---|---|
| GPTBot | OpenAI | training |
| OAI-SearchBot | OpenAI | search |
| ChatGPT-User | OpenAI | user-triggered |
| ClaudeBot | Anthropic | training |
| Claude-SearchBot | Anthropic | search |
| Claude-User | Anthropic | user-triggered |
| anthropic-ai | Anthropic (older token) | training |
| PerplexityBot | Perplexity | search |
| Perplexity-User | Perplexity | user-triggered |
| Google-Extended | Google (Gemini training and grounding control) | training |
| Applebot-Extended | Apple (Apple Intelligence training control) | training |
| CCBot | Common Crawl | training |
| Bytespider | ByteDance | training |
| meta-externalagent | Meta | training |
| Amazonbot | Amazon | search |
| cohere-ai | Cohere | training |
| DuckAssistBot | DuckDuckGo | search |
| MistralAI-User | Mistral | user-triggered |

**Training** bots collect pages to train models. **Search** bots build an AI search or answer index. **User-triggered** fetchers load a page when a person asks an AI assistant about it. Blocking training bots while allowing search bots is a common, deliberate choice; blocking search bots means the site is unlikely to be cited in those AI answers.

### Pricing

| What | Price |
|---|---|
| Page audited (any status under 500, including 404s: that is the answer) | USD 0.002 (USD 2 per 1,000) |
| Site AI-access check (robots.txt answered with a status under 500, other than 429) | USD 0.003 (USD 3 per 1,000) |
| Pages that fail to load (DNS, timeout, TLS), 5xx pages, invalid URLs, unreachable sites, pages blocked by robots.txt, clean pages left out by `outputIssuesOnly` | Free |

Examples on the free plan:

- The default input (2 sites, 10 pages each): 2 x 0.003 + 20 x 0.002 = **USD 0.046**.
- One site, 500 pages: 0.003 + 500 x 0.002 = **USD 1.003**.
- 1,000 domains with **AI checks only**: 1,000 x 0.003 = **USD 3**.

Higher Apify plans get a discount. Set a maximum cost per run in the run options; the actor stops cleanly when it is reached and keeps everything written so far.

### Schedule it weekly

Save the input as a task and schedule it weekly. Compare runs to see which issues appeared or were fixed, and whether a site changed its AI bot rules. Export to Google Sheets, Excel or a webhook.

### How it crawls (politeness)

- Every request identifies the actor: `Mozilla/5.0 (compatible; seo-ai-readiness-audit/1.0; +https://apify.com/JTPalms)`. It never pretends to be a browser or an AI bot. The AI bot verdicts are read from robots.txt, never tested by sending a bot's name.
- It obeys robots.txt for its own names (`seo-ai-readiness-audit` and `jtpalms-seo-audit`, or the `*` group when neither is named): disallowed pages are skipped, free, and counted as `skippedByRobots` in the run summary. If robots.txt keeps it out of `/llms.txt`, that file is reported as `skipped-by-robots` instead of fetched.
- At most 2 requests at a time to one host, at least 250 ms between them, and a robots.txt `Crawl-delay` is honoured up to 10 seconds.
- 20 seconds timeout per request; network errors and 5xx answers are retried twice, 4xx answers are not.
- HTTP only: no browser, no proxies. Some sites block data-center traffic or bots; you see their real answer (often 403), which is charged as an `http-error` page because it is the page's status.
- Private and internal addresses (localhost, 10.x, 169.254.x and similar) are refused.
- Crawling a site may be restricted by that site's terms; only audit sites you own or are allowed to audit.

### What this does not do

- **No JavaScript rendering.** Pages are read as the server sends them. Sites built entirely in the browser show thin content and get `possibly-js-rendered`.
- **No Core Web Vitals or Lighthouse scores.** `responseMs` is server response time only.
- **No backlink or authority data.** For that, see [Domain Authority Checker](https://apify.com/JTPalms/domain-authority-checker).
- **No broken-link report for every link.** It audits the pages it crawls; for every broken link on a site use [Broken Link Checker](https://apify.com/JTPalms/broken-link-checker), and to list every URL in a sitemap use [Sitemap URL Extractor](https://apify.com/JTPalms/sitemap-url-extractor).
- It does not check whether AI bots actually visit the site (that needs the site's server logs), only what robots.txt and the pages tell them.

### Related actors

- [Broken Link Checker: Find Dead Links on Any Website](https://apify.com/JTPalms/broken-link-checker): Crawl a website and find every broken link, image, script and stylesheet: 404s, 5xx, DNS and TLS errors, redirect loops and redirected...
- [URL Status Checker: Bulk HTTP Status Code & Redirect Checker](https://apify.com/JTPalms/url-status-checker): Bulk URL status and HTTP status code checker.
- [Sitemap URL Extractor: All Pages of Any Website](https://apify.com/JTPalms/sitemap-url-extractor): Get every URL of a website from its XML sitemaps.
- [Domain Authority Checker: Backlink Rank from Open Data](https://apify.com/JTPalms/domain-authority-checker): Bulk domain authority checker built on open data: website authority and backlink strength from the Majestic Million (referring subnets...

### FAQ

**A page works in my browser but shows 403 here. Why?** The site blocks automated requests, and this actor says plainly that it is one. That 403 is what search and AI crawlers from data centers may see too.

**Why is `llmsTxtStatus` "soft-404"?** The site answered `/llms.txt` with an ordinary HTML page (often its home page or a "not found" page with status 200) instead of a text file. AI tools looking for llms.txt get nothing useful.

**Does `allowed` mean AI search engines will show the site?** No. It means robots.txt does not keep that bot out. Whether an AI answer cites the site depends on the engine.

**Something missing or wrong?** Open an issue on the actor page with the URL.

# Actor input Schema

## `startUrls` (type: `array`):

Sites or pages to audit. Each host is audited once for AI access (robots.txt and llms.txt), then its pages are crawled from these URLs. A URL without http:// or https:// is treated as https://.

## `maxPagesPerSite` (type: `integer`):

Pages fetched per site (start host). Pages blocked by robots.txt and non-HTML files do not count.

## `useSitemap` (type: `boolean`):

Take page URLs from the sitemaps listed in robots.txt (or /sitemap.xml when none is listed) instead of following links from the start URL.

## `sameHostOnly` (type: `boolean`):

Only follow links on the exact host of the site (for example www.example.com). Turn off to also follow links to other subdomains of the same domain (blog.example.com); they share the same page limit and get no site row of their own. Links to other domains are never followed.

## `aiChecksOnly` (type: `boolean`):

Only the site row: robots.txt rules for 18 AI bots, llms.txt and llms-full.txt, sitemaps. No pages are crawled. The cheapest mode for long domain lists.

## `maxConcurrency` (type: `integer`):

Requests in flight at the same time for the whole run. Whatever you set, the actor sends at most 2 at a time to one host, waits at least 250 ms between requests to the same host, and honours a robots.txt Crawl-delay up to 10 seconds.

## `outputIssuesOnly` (type: `boolean`):

Output only page rows that have at least one issue code or an error. Clean pages are left out and not charged. Site rows are always output.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    },
    {
      "url": "https://apify.com"
    }
  ],
  "maxPagesPerSite": 10,
  "useSitemap": false,
  "sameHostOnly": true,
  "aiChecksOnly": false,
  "maxConcurrency": 5,
  "outputIssuesOnly": false
}
```

# Actor output Schema

## `results` (type: `string`):

Site rows (type: site) and page rows (type: page) in the default dataset (JSON, CSV, Excel via the format parameter).

## `summary` (type: `string`):

Counts: sites, pages audited, pages skipped by robots.txt, clean pages left out, billed events.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev"
        },
        {
            "url": "https://apify.com"
        }
    ],
    "maxPagesPerSite": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("jtpalms/seo-ai-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://crawlee.dev" },
        { "url": "https://apify.com" },
    ],
    "maxPagesPerSite": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("jtpalms/seo-ai-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev"
    },
    {
      "url": "https://apify.com"
    }
  ],
  "maxPagesPerSite": 10
}' |
apify call jtpalms/seo-ai-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,jtpalms/seo-ai-readiness-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/JBDMtTZz5SRX6CCsc/builds/Nu8NoJZBy4C3nSlMU/openapi.json