# AI Crawler Access Checker (`persistent_violet_qhb/ai-crawler-audit-1`) Actor

AI assistants now send their own crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended…). If robots.txt blocks them — or an old noai meta tag is still in your templates — your site is invisible in AI answers and nothing in Google Search Console will ever tell you.

- **URL**: https://apify.com/persistent_violet_qhb/ai-crawler-audit-1.md
- **Developed by:** [Anna Wang](https://apify.com/persistent_violet_qhb) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.05 / site-audit

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler Access & GPTBot Checker

**See your website the way AI does.**

AI assistants and answer engines — ChatGPT search, Claude, Perplexity, Gemini, Meta AI — no longer discover your pages only through Google. They send their **own crawlers**. If your `robots.txt` blocks GPTBot or PerplexityBot, or someone added a `noai` meta tag years ago, your site is invisible in AI answers — and you would never know from your analytics.

This Actor audits any website and tells you exactly:

- which AI crawlers your **robots.txt** allows or blocks — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, Bytespider, and more (list is fully editable)
- whether **/llms.txt** and **/llms-full.txt** exist — the emerging standard that helps AI models index your site
- whether AI opt-out **meta tags** (`noai`, `noimageai`) or **X-Robots-Tag headers** are present
- your **AI Visibility Score (0–100)** with a prioritized, actionable fix list

### Use cases

- **SEO & AI-visibility consultants** — audit client sites before selling an "AI search optimization" engagement, then re-audit after fixes.
- **Marketing teams** — check whether a CMS migration, WAF rule, or new security plugin silently blocked AI crawlers.
- **Developers** — paste any domain to understand its AI-crawler policy in seconds; export JSON via API for monitoring.

### Input

| Field | Type | Description |
|---|---|---|
| `websiteUrls` | string list (required) | One or more origins to audit, e.g. `https://example.com`. Up to 50, audited in parallel. |
| `bots` | string list | robots.txt user-agent tokens to check. Defaults cover the major AI crawlers (November 2026). |
| `checkLlmsTxt` | boolean | Probe `/llms.txt` and `/llms-full.txt`. Default: true. |
| `checkHtmlSignals` | boolean | Inspect meta robots tags and X-Robots-Tag headers. Default: true. |
| `requestTimeoutSecs` | integer | Per-request timeout. Default: 15. |
| `generateMarkdownReport` | boolean | Save a shareable `report.md` to the key-value store. Default: true. |

### Output

**Dataset** — one item per site:

```json
{
  "url": "https://example.com",
  "aiVisibilityScore": 72,
  "verdict": "Partially open",
  "robotsTxt": "found",
  "llmsTxt": "missing",
  "blockedBots": ["GPTBot", "ClaudeBot"],
  "botResults": [ { "bot": "GPTBot", "operator": "OpenAI", "allowed": false, "matched_rule": "Disallow: /", "...": "..." } ],
  "recommendations": [ { "issue": "...", "impact": "...", "how_to_fix": "..." } ]
}
```

**Key-value store** — `report.md`: a ready-to-send markdown report with a per-bot table and recommendations.

### How the score works

Start at 100. Subtract for every blocked AI crawler, weighted by how much it matters for AI visibility: search bots (−20), live browsing (−10), model training (−8), open-web aggregators (−5). Missing `/llms.txt` (−10) and explicit AI opt-out tags (−15) also reduce the score. The score is a transparent heuristic, not a ranking — the per-bot table is the ground truth.

### Getting started

1. Run the Actor with the default input (`https://www.example.com`) to see the output format.
2. Enter your own domain(s).
3. Open **Storage → Key-value store → report.md** for the human-readable report, or consume the dataset via API / webhooks / Make & Zapier integrations.

### FAQ

**Does this work on sites behind bot protection?**
The Actor only fetches `robots.txt`, `/llms.txt` and the homepage like a normal browser. Hard-blocked hosts will show `error` status rather than fail the run.

**Does it render JavaScript?**
No — meta tags and robots.txt are static artifacts. Sites that inject robots directives only via JS are out of scope for this tool.

**Is the bot list up to date?**
The defaults reflect the major AI crawlers as of November 2026. New crawlers appear regularly — you can audit any token by editing the `bots` input, and updates ship via version bumps.

**How is this compliant?**
The Actor requests three publicly accessible URLs per site, honours redirects and standard timeouts, stores no personal data, and does not bypass any access control.

### Limitations

- robots.txt matching follows RFC 9309 semantics (groups, `*` wildcards, `$` anchors, longest-match) with the common "tie goes to Allow" convention; exotic edge cases may differ from individual vendors' parsers.
- Score weights are a published heuristic, not an industry standard.

### Changelog

- **0.1.0** — initial release: 15 default AI crawlers, llms.txt probing, HTML/X-Robots-Tag opt-out detection, visibility score, markdown report.

# Actor input Schema

## `websiteUrls` (type: `array`):

One or more website origins to audit, e.g. https://example.com. Multiple URLs are audited in parallel.

## `bots` (type: `array`):

robots.txt user-agent tokens to evaluate. Defaults cover the major AI crawlers as of November 2026. You can add or remove tokens freely.

## `checkLlmsTxt` (type: `boolean`):

Also probe /llms.txt and /llms-full.txt — the emerging standard that helps AI models index your site better.

## `checkHtmlSignals` (type: `boolean`):

Inspect the homepage for AI opt-out meta tags (noai, noimageai, googlebot) and X-Robots-Tag response headers.

## `requestTimeoutSecs` (type: `integer`):

Per-request timeout in seconds.

## `generateMarkdownReport` (type: `boolean`):

Save a human-readable report.md to the key-value store, ready to share with clients or colleagues.

## Actor input object example

```json
{
  "websiteUrls": [
    "https://www.example.com"
  ],
  "bots": [
    "GPTBot",
    "OAI-SearchBot",
    "ChatGPT-User",
    "ClaudeBot",
    "Claude-User",
    "Claude-SearchBot",
    "PerplexityBot",
    "Perplexity-User",
    "Google-Extended",
    "Applebot-Extended",
    "meta-externalagent",
    "Bytespider",
    "Amazonbot",
    "CCBot",
    "cohere-ai"
  ],
  "checkLlmsTxt": true,
  "checkHtmlSignals": true,
  "requestTimeoutSecs": 15,
  "generateMarkdownReport": true
}
```

# Actor output Schema

## `siteAudits` (type: `string`):

One audit item per website: per-bot access matrix, llms.txt status, AI visibility score and prioritized recommendations.

## `markdownReport` (type: `string`):

Human-readable report stored as report.md in the default key-value store.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "websiteUrls": [
        "https://www.example.com"
    ],
    "bots": [
        "GPTBot",
        "OAI-SearchBot",
        "ChatGPT-User",
        "ClaudeBot",
        "Claude-User",
        "Claude-SearchBot",
        "PerplexityBot",
        "Perplexity-User",
        "Google-Extended",
        "Applebot-Extended",
        "meta-externalagent",
        "Bytespider",
        "Amazonbot",
        "CCBot",
        "cohere-ai"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("persistent_violet_qhb/ai-crawler-audit-1").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "websiteUrls": ["https://www.example.com"],
    "bots": [
        "GPTBot",
        "OAI-SearchBot",
        "ChatGPT-User",
        "ClaudeBot",
        "Claude-User",
        "Claude-SearchBot",
        "PerplexityBot",
        "Perplexity-User",
        "Google-Extended",
        "Applebot-Extended",
        "meta-externalagent",
        "Bytespider",
        "Amazonbot",
        "CCBot",
        "cohere-ai",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("persistent_violet_qhb/ai-crawler-audit-1").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "websiteUrls": [
    "https://www.example.com"
  ],
  "bots": [
    "GPTBot",
    "OAI-SearchBot",
    "ChatGPT-User",
    "ClaudeBot",
    "Claude-User",
    "Claude-SearchBot",
    "PerplexityBot",
    "Perplexity-User",
    "Google-Extended",
    "Applebot-Extended",
    "meta-externalagent",
    "Bytespider",
    "Amazonbot",
    "CCBot",
    "cohere-ai"
  ]
}' |
apify call persistent_violet_qhb/ai-crawler-audit-1 --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,persistent_violet_qhb/ai-crawler-audit-1"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/cddFu79fl8zSW6AFP/builds/5ejHCCl2Gqfathjm0/openapi.json