# AI Crawler Access Auditor (`steadfeed/ai-crawler-access-auditor`) Actor

See whether your site's rules block ChatGPT, Perplexity, Claude, Google AI or Apple crawlers. Reads robots.txt per RFC 9309 and page noindex/X-Robots-Tag rules, runs a simulated crawler firewall test (a warning signal, not proof of real-crawler access), and gives copy-paste fixes. No API key needed.

- **URL**: https://apify.com/steadfeed/ai-crawler-access-auditor.md
- **Developed by:** [Steadfeed](https://apify.com/steadfeed) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 website auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler Access Auditor

**Can ChatGPT, Perplexity, Claude, Apple and Google actually reach your website?** This Actor checks your robots.txt the way the crawlers read it, finds `noindex` tags and firewall blocks, and tells you in plain English what to fix, with copy-paste snippets for your developer.

No API key needed. One website takes about 20–30 seconds.

> Allowing a crawler only makes your pages *available* to it. It does not guarantee that any AI assistant or search engine will index, cite, recommend or rank your site. This tool checks access, the first step.

### What it checks

| Check | How | Result type |
|---|---|---|
| **robots.txt rules per crawler** | Parsed per [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309): exact crawler-name groups, catch-all `*` fallback, longest-match Allow/Disallow, `$` and `*` wildcards, 500 KiB limit, 4xx = "allow all", 5xx/unreachable = "crawlers may stop". | **Declared** (confirmed) |
| **Apple's Googlebot fallback** | If `Applebot` isn't named, Apple says it follows your `Googlebot` rules. We apply that. | Declared |
| **noindex / nosnippet** | `<meta name="robots">`, bot-specific meta tags and the `X-Robots-Tag` header, with bot scoping (`googlebot: noindex`). | Declared |
| **noai / noimageai** | Detected and flagged as informal: the major AI vendors don't document support. | Info |
| **Firewall / bot protection** | One polite test request per crawler using its name (homepage only), compared with a normal browser request. Detects Cloudflare, Akamai, Imperva, DataDome, HUMAN, AWS WAF, Sucuri and Vercel challenge or block pages, plus 403/429 responses. | **Signal, not proof** |
| **robots.txt health** | Typos (`Dissallow`), rules before any `User-agent`, `noindex` in robots.txt, HTML served as robots.txt, Cloudflare-managed sections, `Content-Signal` lines. | Confirmed |
| **llms.txt** | Present / missing / HTML soft-404. Informational only: no major AI crawler operator documents using it. | Info |

#### Crawlers covered (14)

| Type | Crawlers |
|---|---|
| AI search (can show your pages in answers) | OAI-SearchBot (ChatGPT search), PerplexityBot, Claude-SearchBot, Applebot |
| AI user fetches (open a page when someone asks) | ChatGPT-User, Perplexity-User, Claude-User |
| AI training (separate choice from search) | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot |
| Classic search (baseline) | Googlebot, Bingbot |

Add up to 10 more names (e.g. `Amazonbot`, `meta-externalagent`, `Bytespider`) with **Other robots.txt names**. They are evaluated in robots.txt only.

### Confirmed vs. signal vs. unknown

Each finding is labelled so you know how much to trust it:

- **Confirmed / declared**: what your robots.txt, meta tags and headers say. This is deterministic.
- **Signal**: how your site responded to our test request that used the crawler's name. The requests come from Apify's servers, **not** OpenAI's, Perplexity's or anyone else's IP addresses. Firewalls that verify real crawlers by IP address may treat the genuine crawler differently, in either direction. Treat a signal as "worth checking", not as proof.
- **Unknown**: for example, robots.txt timed out, or the site blocks all cloud traffic (including our normal-browser request), so we can't tell.

We **never** send test requests as a crawler that your robots.txt blocks, and we stop testing a site as soon as it rate-limits us.

### Output

One dataset row per website:

- `verdict`, which is one of `open`, `ai-search-blocked`, `ai-search-partly-blocked`, `noindex`, `possible-firewall-block`, `some-pages-restricted`, `robots-unreadable`, `inconclusive` or `unknown`
- `headline` and `summary` in plain English
- `aiSearchAccess`, `userFetchAccess`, `trainingAccess` and `searchEngineAccess`, each one of `allowed`, `blocked`, `mixed` or `unknown`
- `confirmed[]`, `uncertain[]` and `suggestedActions[]` (with priority)
- `fixes[]`: robots.txt snippets that keep your existing private-path rules, plus firewall guidance with links to the vendors' published IP ranges
- `crawlers[]`: for each crawler, the robots.txt decision with the exact rule and line number, the test-request result, and vendor docs
- `robotsTxt`, `pages`, `llmsTxt` and `cdn`

For each site there's also a readable **Markdown report** in the key-value store (`REPORT-<host>`), plus a run `SUMMARY`.

Example (shortened):

```json
{
  "site": "https://www.example.com",
  "verdict": "ai-search-partly-blocked",
  "headline": "robots.txt blocks some AI search crawlers: OAI-SearchBot and PerplexityBot.",
  "aiSearchAccess": "mixed",
  "trainingAccess": "blocked",
  "fixes": [{ "title": "Allow OAI-SearchBot and PerplexityBot", "snippet": "User-agent: OAI-SearchBot\nUser-agent: PerplexityBot\nAllow: /\nDisallow: /admin" }]
}
```

### Input

| Field | Default | Notes |
|---|---|---|
| `urls` | `["https://apify.com"]` | Up to 25 websites per run, audited one at a time. A page URL audits that page and the homepage. |
| `extraPaths` | `[]` | Up to 4 more paths per site, e.g. `/pricing`. |
| `liveChecks` | `true` | Firewall test requests (homepage only). |
| `crawlers` | all | Limit to some crawlers. |
| `extraUserAgents` | `[]` | Extra robots.txt names. |
| `requestDelayMs` | `1000` | Pause between requests to the same site (500–10000). |
| `aiSummary` + `openaiApiKey` | off | Optional plain-English summary written by OpenAI **with your own key** (billed to you by OpenAI, typically well under $0.001 per site with `gpt-6-luna`). The model only rewrites our findings. The audit never needs AI and never uses anyone else's key. |

### Common fixes it suggests

- **Your catch-all `User-agent: *` with `Disallow: /` blocks AI search**: add a dedicated group for the AI search crawlers that keeps your private-path rules.
- **A crawler has its own blocking group**: replace that group, or remove just its `User-agent` line from a shared group. We warn you when deleting a line would make its rules attach to the group above.
- **Cloudflare "Block AI bots" / Bot Fight Mode / AI Crawl Control**: where to look, and why WAF Skip rules should match the user-agent *and* the vendor's IP ranges.
- **noindex left over from staging**: where it is, and what to remove.
- **Training vs. search**: blocking GPTBot, ClaudeBot or Google-Extended is a separate choice. It doesn't remove you from ChatGPT search, Claude search or Google Search, per those vendors' docs.

### Pricing

Pay per event: you're charged once per website audited. Websites we can't reach at all are free. See the Pricing tab.

### Next step: is AI actually mentioning you?

Access is necessary but not sufficient. To measure whether ChatGPT, Perplexity, Claude and Gemini actually mention and cite your brand for your buyers' questions, use the **[AI Search Visibility Tracker](https://apify.com/steadfeed/ai-search-visibility-tracker)**.

### Limits and honesty notes

- Test requests can't reproduce IP-verified allowlisting. Results are signals.
- When look-alike Googlebot and Bingbot requests are both blocked, we treat it as normal anti-spoofing protection (`look-alike-blocked`), not a problem, because real search bots are verified by IP.
- Some sites serve a different robots.txt or page to verified crawlers (by IP address). We report the robots.txt we received.
- Bots that execute JavaScript challenges, or that get country-specific responses, may see something different.
- Vendor rules change. The crawler list and its notes were checked against vendor docs on 2026-11-05 and are linked in every result.
- Please audit only sites you own or are authorised to check. The Actor is polite (one site at a time, up to about 20 requests per site, 1 second apart by default), and it identifies its test requests with `SteadfeedAuditProbe/1.0`.

By Steadfeed.

# Actor input Schema

## `urls` (type: `array`):

Website addresses to audit, e.g. example.com or https://www.example.com/pricing. Up to 25 per run, checked one at a time. If you give a page address, that page is audited plus the homepage.

## `extraPaths` (type: `array`):

Up to 4 more paths on each site, e.g. /pricing or /blog/. Each is checked against robots.txt and for noindex tags.

## `liveChecks` (type: `boolean`):

Send one polite request per crawler (homepage only) using that crawler's name and compare the answer with a normal browser request. Shows firewall or bot-protection blocks as signals. Skipped for crawlers your robots.txt blocks.

## `crawlers` (type: `array`):

Leave empty to check all of them.

## `extraUserAgents` (type: `array`):

Up to 10 more crawler names to evaluate in robots.txt only (no test requests), e.g. Amazonbot, meta-externalagent, Bytespider.

## `requestDelayMs` (type: `integer`):

Minimum pause between requests to the same site.

## `saveReports` (type: `boolean`):

Save a readable report per website (REPORT-\*) and a SUMMARY in the key-value store.

## `aiSummary` (type: `boolean`):

Adds a short plain-English summary written by OpenAI from the audit findings. Needs your own OpenAI API key below; billed by OpenAI to you. The audit itself never needs AI.

## `openaiApiKey` (type: `string`):

Only used when AI-written summary is on. Stored encrypted by Apify.

## `openaiModel` (type: `string`):

Model for the summary.

## Actor input object example

```json
{
  "urls": [
    "https://apify.com"
  ],
  "extraPaths": [],
  "liveChecks": true,
  "crawlers": [],
  "extraUserAgents": [],
  "requestDelayMs": 1000,
  "saveReports": true,
  "aiSummary": false,
  "openaiModel": "gpt-6-luna"
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        "https://apify.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("steadfeed/ai-crawler-access-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": ["https://apify.com"] }

# Run the Actor and wait for it to finish
run = client.actor("steadfeed/ai-crawler-access-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    "https://apify.com"
  ]
}' |
apify call steadfeed/ai-crawler-access-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,steadfeed/ai-crawler-access-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/S7o8TlR8xMZsXtYAZ/builds/uKjaIbzRQU0COQpOi/openapi.json