# AI Crawler Access Audit: GPTBot, ClaudeBot, PerplexityBot (GEO) (`sefran1/ai-crawler-access-audit`) Actor

Can ChatGPT, Claude, Perplexity, Gemini and Copilot read and cite your pages? Checks robots.txt rules for 19 AI crawlers, CDN/WAF blocks of bot user-agents, noindex/nosnippet, JavaScript-only content, sitemap and llms.txt. No API key needed. $0.02 per page.

- **URL**: https://apify.com/sefran1/ai-crawler-access-audit.md
- **Developed by:** [Iberia Channel Desk](https://apify.com/sefran1) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 page auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler Access Audit: can ChatGPT, Claude, Perplexity and Gemini read your site?

If an AI crawler can't fetch your page, the AI can't cite it. This Actor checks, page by page, whether the crawlers behind **ChatGPT search, Claude, Perplexity, Gemini, Google AI Overviews, Copilot, Apple and DuckDuckGo** are allowed in and can actually read your content. No API key, no login, **$0.02 per page**.

It focuses on the things that block AI visibility outright, and labels the rest honestly:

| Check | Why it matters |
|---|---|
| **robots.txt for 19 AI crawlers** (OAI-SearchBot, ChatGPT-User, GPTBot, Claude-SearchBot, Claude-User, ClaudeBot, PerplexityBot, Perplexity-User, Googlebot, Google-Extended, Bingbot, Applebot, Applebot-Extended, DuckAssistBot, MistralAI-User, meta-externalagent, Amazonbot, CCBot, Bytespider) | Blocking an *answer* bot (e.g. OAI-SearchBot, PerplexityBot) means that engine cannot cite you. Blocking a *training* bot (GPTBot, CCBot) is a policy choice and is reported separately. Uses RFC 9309 matching: exact user-agent groups, longest rule wins, Allow wins ties, unreachable robots.txt = full disallow. |
| **CDN / firewall blocks** | Re-requests the page with the published user-agent of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot. A 403 or challenge page while a browser gets 200 usually means a "block AI bots" rule (e.g. Cloudflare) that robots.txt doesn't show. |
| **noindex / nosnippet / X-Robots-Tag** | Search-based AI answers (Google AI Overviews and AI Mode, Bing/Copilot) won't use noindex pages; nosnippet keeps text out of AI Overviews. |
| **Content without JavaScript** | Most AI crawlers don't run JavaScript. If the raw HTML has almost no text, they see an empty page. |
| Sitemap, title, meta description, H1 | Discovery and basic readability. |
| llms.txt, JSON-LD structured data | Reported as **info only**: server-log studies found AI crawlers rarely request llms.txt, and controlled tests found no measurable citation effect from schema markup. We tell you, but we don't pretend they're critical. |

Each page gets a 0–100 score and a list of issues ranked `critical` → `info`. One `site-summary` record per site rolls everything up.

### Sample output

Real output recorded on 2026-10-17 for `https://www.nytimes.com/` (a site that blocks most AI crawlers on purpose). Crawler list trimmed to 5 of 19 for readability.

```json
{
  "type": "page",
  "url": "https://www.nytimes.com/",
  "finalUrl": "https://www.nytimes.com/",
  "status": 403,
  "responseMs": 47,
  "fetchError": null,
  "title": "nytimes.com",
  "metaDescription": null,
  "lang": "en",
  "canonical": null,
  "h1Count": 0,
  "headingCount": 0,
  "noindex": false,
  "nosnippet": false,
  "xRobotsTag": null,
  "metaRobots": null,
  "htmlBytes": 774,
  "wordsWithoutJavaScript": 8,
  "likelyNeedsJavaScript": false,
  "structuredDataTypes": [],
  "aiCrawlerAccess": [
    {
      "bot": "OAI-SearchBot",
      "owner": "OpenAI",
      "purpose": "answer",
      "usedFor": "ChatGPT search results and citations",
      "allowedByRobots": false,
      "matchedGroup": "specific",
      "rule": "Disallow: /"
    },
    {
      "bot": "ChatGPT-User",
      "owner": "OpenAI",
      "purpose": "answer",
      "usedFor": "ChatGPT fetching a page a user asked about",
      "allowedByRobots": false,
      "matchedGroup": "specific",
      "rule": "Disallow: /"
    },
    {
      "bot": "GPTBot",
      "owner": "OpenAI",
      "purpose": "training",
      "usedFor": "OpenAI model training",
      "allowedByRobots": false,
      "matchedGroup": "specific",
      "rule": "Disallow: /"
    },
    {
      "bot": "Claude-SearchBot",
      "owner": "Anthropic",
      "purpose": "answer",
      "usedFor": "Claude search results",
      "allowedByRobots": false,
      "matchedGroup": "specific",
      "rule": "Disallow: /"
    },
    {
      "bot": "Googlebot",
      "owner": "Google",
      "purpose": "search",
      "usedFor": "Google Search, AI Overviews and AI Mode",
      "allowedByRobots": true,
      "matchedGroup": "specific",
      "rule": null
    }
  ],
  "issues": [
    {
      "severity": "high",
      "code": "refused-auditor",
      "message": "The page answered 403 to this audit (a normal browser user-agent from a cloud IP). Bot protection may also be refusing AI crawlers; check your CDN/WAF rules. robots.txt results below are still valid."
    },
    {
      "severity": "critical",
      "code": "answer-bots-blocked",
      "message": "robots.txt blocks OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, DuckAssistBot. These fetch pages for AI answers/search, so this page cannot be cited by OpenAI, Anthropic, Perplexity, DuckDuckGo."
    },
    {
      "severity": "info",
      "code": "training-bots-blocked",
      "message": "robots.txt blocks training crawlers GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, meta-externalagent, CCBot, Bytespider. That is a policy choice: it keeps content out of future model training but does not stop AI search citations."
    },
    {
      "severity": "low",
      "code": "missing-meta-description",
      "message": "No meta description (used as a summary by some crawlers and search snippets)."
    },
    {
      "severity": "low",
      "code": "missing-h1",
      "message": "No <h1> heading."
    },
    {
      "severity": "info",
      "code": "no-llms-txt",
      "message": "No /llms.txt. Optional: server-log studies found AI crawlers rarely request it, so expect little effect."
    },
    {
      "severity": "info",
      "code": "no-structured-data",
      "message": "No JSON-LD structured data. Useful for Google rich results; controlled tests found no measurable effect on AI citations, so this is low priority."
    }
  ],
  "score": 51,
  "checkedAt": "2026-10-17T19:24:38.918Z"
}
```

Site summary:

```json
{
  "type": "site-summary",
  "origin": "https://www.nytimes.com",
  "pagesAudited": 1,
  "averageScore": 51,
  "robotsTxt": {
    "found": true,
    "status": 200,
    "error": null,
    "unreachable": false
  },
  "llmsTxt": {
    "found": false,
    "status": 404,
    "bytes": 0
  },
  "sitemap": {
    "found": true,
    "url": "https://www.nytimes.com/sitemaps/new/news.xml.gz",
    "declaredInRobots": true
  },
  "answerBotsBlockedOnHomepageOrFirstPage": [
    "OAI-SearchBot",
    "ChatGPT-User",
    "Claude-SearchBot",
    "Claude-User",
    "PerplexityBot",
    "Perplexity-User",
    "DuckAssistBot"
  ],
  "trainingBotsBlocked": [
    "GPTBot",
    "ClaudeBot",
    "Google-Extended",
    "Applebot-Extended",
    "meta-externalagent",
    "CCBot",
    "Bytespider"
  ],
  "issueCounts": {
    "refused-auditor": 1,
    "answer-bots-blocked": 1,
    "training-bots-blocked": 1,
    "missing-meta-description": 1,
    "missing-h1": 1,
    "no-llms-txt": 1,
    "no-structured-data": 1
  },
  "checkedAt": "2026-10-17T19:24:40.136Z"
}
```

### How to use

1. Paste one or more **page URLs** (homepage plus your most important product, pricing or blog pages).
2. Optional: set **Extra pages from sitemap** to audit up to 200 more pages per site from its XML sitemap.
3. Run. Use the **Overview** table, or export JSON/CSV. Schedule it monthly (or after CDN/robots changes) to catch regressions.

Typical fixes it points to: remove `Disallow` for OAI-SearchBot/PerplexityBot/Claude-SearchBot, turn off a CDN "block AI bots" rule for answer bots, drop accidental `noindex`, server-render key content, publish a sitemap.

### Pricing

**$0.02 per page audited** (robots.txt, sitemap and llms.txt checks per site are included). Pages that refuse the audit (401/403/429) or can't be reached are reported but **not charged**. Example: homepage + 24 sitemap pages = $0.50.

### Limits

- The user-agent probe comes from a cloud IP. Some CDNs block *unverified* bots that use an AI user-agent but allow the real crawler (verified by IP), so treat a probe block as "check your CDN settings", not proof.
- Being allowed in doesn't guarantee being cited. To measure how often ChatGPT, Gemini and Perplexity actually mention your brand, use [AI Brand Visibility Tracker](https://apify.com/sefran1/ai-brand-visibility-checker).
- Reads public pages only, never logs in, and makes at most 7 requests per page (1 normal + up to 5 bot user-agents + site files once per site).

# Actor input Schema

## `urls` (type: `array`):

Pages to audit (homepage, product pages, blog posts). Each site's robots.txt, sitemap and llms.txt are checked once.

## `sitemapPagesPerSite` (type: `integer`):

Also audit up to this many pages listed in each site's XML sitemap. 0 = only the URLs above. Each page is one charged audit.

## `probeUserAgents` (type: `boolean`):

Re-request each page with the published user-agent of GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot to detect firewalls that block AI crawlers even when robots.txt allows them.

## Actor input object example

```json
{
  "urls": [
    "https://apify.com/"
  ],
  "sitemapPagesPerSite": 0,
  "probeUserAgents": true
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://apify.com/"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("sefran1/ai-crawler-access-audit").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://apify.com/"] }

# Run the Actor and wait for it to finish
run = client.actor("sefran1/ai-crawler-access-audit").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://apify.com/"
  ]
}' |
apify call sefran1/ai-crawler-access-audit --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sefran1/ai-crawler-access-audit"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/YG1JZ7BcU5geEm85U/builds/FbvoYIacuAdBeuAju/openapi.json