# AI Crawler Access & AEO Audit (`aiseewhy/ai-crawler-access-aeo-audit`) Actor

Fetches each URL as 14 AI and search crawlers next to a real browser, names the firewall that blocks them, checks robots.txt, JSON-LD, answer-first content, freshness, hreflang and llms.txt, and returns ready-to-paste fixes. Reports in 30 languages. By AISeeWhy.

- **URL**: https://apify.com/aiseewhy/ai-crawler-access-aeo-audit.md
- **Developed by:** [AISeeWhy](https://apify.com/aiseewhy) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Features:** Limited permissions
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$15.00 / 1,000 url auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler Access & AEO Audit

Can AI search and AI assistants read your pages, and if they can't, what exactly do you change?

This Actor fetches each URL **as 14 real AI and search crawlers plus curl**, and once as a normal browser from the same IP. It also checks robots.txt rules for 16 crawler tokens, including Google-Extended and Applebot-Extended. It compares the responses and tells you which crawler is blocked, which firewall or CDN did it, and what to paste to fix it. It also checks what helps a page get quoted in AI answers: robots.txt rules per crawler, structured data, answer-first content, freshness dates, sitemaps, hreflang and llms.txt.

You get findings with evidence and **ready-to-paste fixes**: a robots.txt diff, a Cloudflare rule, JSON-LD blocks, a draft llms.txt. There is no vague 0–100 score.

### What it checks

| Area | What you learn |
|---|---|
| **Crawler access** | For each of GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Googlebot, Bingbot, Applebot, CCBot, Bytespider, Meta-ExternalAgent and curl, it shows whether the crawler gets the same page as a browser, or a 403, a challenge page, a stub page or a redirect. The firewall is identified: Cloudflare, Akamai, Vercel, Sucuri, Imperva or AWS WAF. |
| **robots.txt** | Rules per crawler under RFC 9309, including Google-Extended and Applebot-Extended. It flags conflicts between robots.txt and what the server actually does, and gives a diff that fixes them. It also notes whether Content-Signal is present. |
| **Content** | Whether there is a 40–60 word answer under the H1, whether headings are phrased as questions, whether the page links to outside sources, whether the content only appears after JavaScript runs, and whether the email address is hidden by obfuscation. |
| **Structured data** | Article author and dates, SoftwareApplication `offers`, an identifiable publisher, `sameAs`, FAQPage. It also catches broken JSON-LD. |
| **Freshness & discovery** | Whether `dateModified` matches the sitemap `lastmod`, whether the page is in the sitemap, whether hreflang links are reciprocal and canonical-consistent, and whether llms.txt exists (a draft is generated if not). |

#### Training vs. search crawlers

Blocking a **training** crawler (GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, Google-Extended, Applebot-Extended) is a policy choice. It is reported as P3 with a note, not as an error. Blocking a **search** or **user-fetch** crawler (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User and others) means AI answers can't cite you, so it is P1.

#### What the access test can and cannot prove

The requests come from Apify's servers, not from each vendor's verified IP ranges. A result shows how your firewall treats that **user agent from an unverified IP**. Firewalls that verify crawlers by IP may treat the real crawler differently. Every block finding says this. If even the browser request is blocked, you get one "inconclusive" finding instead of 15 false alarms.

### Input

| Field | Description |
|---|---|
| `urls` | Page URLs to audit. A missing `https://` is added. |
| `sitemapUrl` | Optional. Audits the pages in this sitemap or sitemap index (`.xml.gz` works). |
| `maxUrls` | The most pages per run. Default 100, maximum 5000. |
| `language` | The report language, one of 30 (en, de, es, fr, ja, zh-Hans, ru, …). |
| `bots` | Optional subset of the crawlers. |
| `includeMarkdown` | Adds a readable Markdown report to each item. |

If you give no URL and no sitemap, the Actor runs a free demo audit of `https://example.com/`, so you can see the output format.

```json
{ "urls": ["https://example.com/pricing"], "sitemapUrl": "https://example.com/sitemap.xml", "maxUrls": 50, "language": "en" }
```

### Output

Each audited URL produces one dataset item:

```json
{
  "url": "https://example.com/guide/",
  "final_url": "https://example.com/guide/",
  "counts": { "P1": 1, "P2": 1, "P3": 2 },
  "findings": [
    {
      "id": "access.blocked", "priority": "P1", "bot": "OAI-SearchBot", "bot_class": "search",
      "title": "OAI-SearchBot is blocked",
      "detail": "A request with the OAI-SearchBot user agent got status 403 or a challenge page, while a browser got the page. …",
      "evidence": { "status": 403, "control_status": 200, "waf": "cloudflare" },
      "fixes": [ { "kind": "cloudflare", "title": "Let OAI-SearchBot through in Cloudflare", "body": "Cloudflare dashboard → Security → WAF → …" } ]
    }
  ],
  "passed": ["robots.ok", "schema.ok"],
  "billable": true
}
```

The key-value store gets a `SUMMARY` table with one row per URL.

Priorities: **P1** is critical (AI answers can't read or cite the page), **P2** is important, and **P3** is nice to have or a policy choice.

### Pricing

You pay **per audited URL**. You are not charged for URLs that could not be audited: the page did not respond, returned an error status, or blocked even a normal browser. Those items have `"billable": false`. If you set a spending limit on the run, the Actor stops cleanly when it is reached.

### Also available

- An MCP server, so AI agents can call the same audit directly (coming soon).
- A free one-URL check at [aiseewhy.com](https://aiseewhy.com).

Made by **AISeeWhy**. Questions or false positives: open an issue on the Actor page.

# Actor input Schema

## `urls` (type: `array`):

Page URLs to audit, one per line. A missing https:// is added.

## `sitemapUrl` (type: `string`):

Optional. Audit the pages listed in this sitemap or sitemap index (gzip is fine), up to Max URLs.

## `maxUrls` (type: `integer`):

Upper limit of pages audited in this run (list and sitemap together).

## `language` (type: `string`):

Language of finding titles, details and fix titles. Fix code (robots.txt, JSON-LD) is never translated.

## `bots` (type: `array`):

Optional. Leave empty to test all 17 crawlers.

## `includeMarkdown` (type: `boolean`):

Add a human-readable Markdown report to each dataset item.

## Actor input object example

```json
{
  "urls": [
    "https://example.com/"
  ],
  "maxUrls": 100,
  "language": "en",
  "includeMarkdown": false
}
```

# Actor output Schema

## `overview` (type: `string`):

One row per audited URL: P1/P2/P3 counts, findings and whether the URL was charged.

## `results` (type: `string`):

One item per audited URL with findings, evidence and ready-to-paste fixes.

## `summary` (type: `string`):

A Markdown table with one row per URL.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://example.com/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("aiseewhy/ai-crawler-access-aeo-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://example.com/"] }

# Run the Actor and wait for it to finish
run = client.actor("aiseewhy/ai-crawler-access-aeo-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://example.com/"
  ]
}' |
apify call aiseewhy/ai-crawler-access-aeo-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,aiseewhy/ai-crawler-access-aeo-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/KLkkkx34RXn1exbZv/builds/n3d0aie6NTajNC5ED/openapi.json