# AI Search Readiness & Technical SEO Audit (GEO) (`mercury_ops/ai-search-readiness-audit`) Actor

Check whether ChatGPT, Claude, Perplexity, Gemini and Google can crawl, read and cite your site: AI bot access in robots.txt, llms.txt, sitemap, structured data, noindex, JS-only pages and more. One scored report per site.

- **URL**: https://apify.com/mercury_ops/ai-search-readiness-audit.md
- **Developed by:** [Mercury](https://apify.com/mercury_ops) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 site reports

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Search Readiness & Technical SEO Audit (GEO)

**Can ChatGPT, Claude, Perplexity, Gemini and Google actually crawl, read and cite your website?** This Actor checks in seconds and returns one scored report per site with a prioritised list of fixes.

### What it checks

| Area | What you get |
|---|---|
| **AI crawler access** | A matrix of 17 crawlers (GPTBot, OAI-AdsBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Amazonbot, meta-externalagent, CCBot, Bytespider, Googlebot, Bingbot) showing whether your robots.txt allows each one. The parser follows RFC 9309: group matching, longest match, `*` and `$` wildcards. |
| **Search vs training split** | Blocking *training* bots is a legitimate choice and is flagged as info only. Blocking *search/answer* bots (which drive citations and referral traffic) is flagged as critical. |
| **CDN / firewall test (opt-in)** | robots.txt can say "allowed" while Cloudflare or another WAF quietly returns 403 to GPTBot or ClaudeBot. Turn this on for **your own sites** and the Actor requests the page with the real AI-crawler user agents, then flags any that get blocked. |
| **llms.txt** | Whether `/llms.txt` exists and is well formed. Soft-404 HTML pages are detected. |
| **Sitemaps** | Finds sitemaps via robots.txt or `/sitemap.xml`, follows sitemap indexes, and counts URLs. |
| **Per page** | HTTP status, response time, title, meta description, canonical, `noindex` (meta and X-Robots-Tag), `lang`, H1 count, word count, **JavaScript-only rendering** (many AI crawlers don't run JS), Open Graph, **JSON-LD structured data types and parse errors**, image alt coverage, and internal/external links. |
| **Score** | 0–100, weighted by severity (critical, high, medium, low, info). |

### Who it's for

- **SEO and marketing agencies** auditing client sites for AI-search visibility ("GEO" / "AEO").
- **Site owners** checking that a CDN or bot-protection change hasn't silently blocked AI search crawlers.
- **Monitoring:** schedule it weekly across a portfolio of sites and alert on score drops via Apify integrations (Slack, email, webhooks).

### Input

```json
{
  "startUrls": [{ "url": "https://www.example.com/" }],
  "maxPagesPerSite": 5,
  "includePageDetails": true
}
```

### Output (one item per site, abridged)

```json
{
  "site": "https://www.bbc.co.uk",
  "score": 72,
  "robotsTxtFound": true,
  "aiBotAccess": [{ "bot": "OAI-SearchBot", "purpose": "ChatGPT search index", "allowed": false, "rule": "specific" }],
  "llmsTxt": { "present": false, "valid_heading": false, "links": 0, "bytes": 0 },
  "sitemapUrlCount": 18681,
  "pagesAudited": 3,
  "issueCounts": { "critical": 1, "high": 0, "medium": 0, "low": 3, "info": 1 },
  "issues": [{ "severity": "critical", "code": "search_bots_blocked", "message": "robots.txt blocks AI/search crawlers…" }],
  "pages": [{ "url": "https://www.bbc.co.uk/", "status": 200, "jsonLdTypes": ["WebSite"], "likelyJsRendered": false }]
}
```

*(Real output from a test run on 21 Oct 2026.)*

### Pricing (pay per event)

- **$0.02 per site report**
- **$0.004 per page audited**

A 5-page audit costs **$0.04**, and 100 sites at 5 pages each cost about **$4**. You pay nothing else; platform usage is included. Set a maximum charge per run and the Actor stops cleanly when it's reached.

### Responsible crawling

- Identifies itself honestly as `AIReadinessAudit/1.0`, and **obeys robots.txt for its own requests** (pages disallowed for it are skipped and listed in `pagesSkippedByRobots`).
- Polite: one request per second per site, at most 50 pages per site, no login, no form submission, no bypassing bot protection.
- Collects no personal data. It reads only technical page metadata.
- Intended for sites you own or manage, or for benchmarking public pages.

### Limits

- It doesn't run JavaScript. That's deliberate, because it mirrors what most AI crawlers see. Pages flagged `likelyJsRendered` need server-side rendering or pre-rendering to be visible to them.
- The firewall test is a spot check of one URL with four user agents. Some CDNs verify real crawler IPs, so a pass from our IP doesn't prove the real bot gets through. Check your server logs as well.
- llms.txt is an emerging convention, not a standard, so a missing file is scored as low severity only.

### Use from AI agents (MCP)

This Actor is pay-per-event, charges only for events, and needs no special permissions, so AI agents can call it as a tool through the Apify MCP server (`mcp.apify.com`) and pay per call. Example agent prompt: *"Audit example.com for AI search readiness and list the critical fixes."*

### Changelog

- 1.0 (Oct 2026): first release.

# Actor input Schema

## `startUrls` (type: `array`):

One URL per site (homepage or any page). Audit sites you own or manage, or public pages of competitors for comparison.

## `maxPagesPerSite` (type: `integer`):

Pages fetched and analysed per site (start URL first, then sitemap URLs, then homepage links). Each analysed page is one billable 'page-audited' event.

## `includePageDetails` (type: `boolean`):

Include the per-page field breakdown in the output, not just the site summary and issues.

## `checkFirewallWithBotUserAgents` (type: `boolean`):

Requests the start URL once each with the real GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot user-agent strings to detect CDN/WAF blocks that robots.txt doesn't show. Only enable for sites you own or manage. 4 extra requests per site, no extra charge.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.gov.uk/"
    }
  ],
  "maxPagesPerSite": 5,
  "includePageDetails": true,
  "checkFirewallWithBotUserAgents": false
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `reports` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.gov.uk/"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("mercury_ops/ai-search-readiness-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://www.gov.uk/" }] }

# Run the Actor and wait for it to finish
run = client.actor("mercury_ops/ai-search-readiness-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.gov.uk/"
    }
  ]
}' |
apify call mercury_ops/ai-search-readiness-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,mercury_ops/ai-search-readiness-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/A8FwPphEOm2Gg9KTN/builds/3Fw7UZgv4dx9ETv7c/openapi.json