# AI Crawler Content Visibility Auditor (`kingii98/ai-crawler-visibility-auditor`) Actor

Check whether AI crawler user agents can read public pages. Audit robots.txt, HTTP status, extracted text, required markers, and cloaking for each URL and bot.

- **URL**: https://apify.com/kingii98/ai-crawler-visibility-auditor.md
- **Developed by:** [kingii98](https://apify.com/kingii98) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 url auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AC-VIS-01 — AI Crawler Content Visibility Auditor

This Actor checks whether AI answer-engine bots (for example GPTBot,
ClaudeBot, PerplexityBot) can read the text on your pages. It is HTTP-only:
it never uses a browser, a proxy, or JavaScript rendering.

### What it does

For each URL in the input:

1. It checks the URL points at a public HTTP(S) host. Private and reserved
   IP targets (for example `127.0.0.1`, `10.0.0.0/8`, link-local addresses)
   are rejected before any request is sent.
2. It fetches the page once with a reference (normal browser) user agent.
3. For each bot user agent in the input, it:
   - Checks `robots.txt` for that bot.
   - Fetches the page with that bot's user agent, if `robots.txt` allows it.
   - Extracts the visible plain text and any `<meta name="robots">`
     directives.
   - Compares the extracted text length against the reference fetch.
   - Checks each required text marker against the raw HTTP response body
     (not the extracted plain text), so a marker can match text inside an
     HTML tag or attribute, not only visible page text.
   - Assigns a verdict: `VISIBLE`, `THIN`, `BLOCKED_ROBOTS`,
     `BLOCKED_SERVER`, or `CLOAKED`.

One record is pushed to the default dataset for each URL and bot user agent
pair.

### Input

See `.actor/input_schema.json` for the full schema. Main fields:

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `urls` | array of string | yes | Public HTTP(S) page URLs to check. Max 500. |
| `botUserAgents` | array of string | no | Bot user agents to check. Default: `GPTBot`, `ClaudeBot`, `PerplexityBot`, `CCBot`, `Google-Extended`, `Bingbot`. Add or replace tokens as new AI bots appear. Max 20. |
| `referenceUserAgent` | string | no | Baseline user agent used once per URL to detect cloaking. |
| `requiredMarkers` | array of string | no | Plain-text strings that must appear in the raw HTTP response body of the page. Max 20. |
| `maxBytesPerFetch` | integer | no | Response body byte cap per fetch. 1,024–5,000,000. Default 2,000,000. |
| `concurrency` | integer | no | Max URLs audited at the same time. 1–20. Default 10. |
| `hostDelaySeconds` | number | no | Minimum delay between two requests to the same host. 0–5. Default 0.2. |
| `previousDatasetId` | string | no | Dataset ID from an earlier run. If set, the run reports which URLs changed verdict since that dataset. |

### Output

One dataset record per URL and bot user agent. See
`.actor/dataset_schema.json` for the full field list. Key fields:

- `url`, `bot` — the audited page and the bot user agent.
- `verdict` — `VISIBLE`, `THIN`, `BLOCKED_ROBOTS`, `BLOCKED_SERVER`, or
  `CLOAKED`.
- `reasonCode` — a machine-readable reason for the verdict.
- `httpStatus`, `finalUrl`, `redirectHops`, `robotsAllowed`.
- `extractedTextChars`, `textDeltaVsReference`, `requiredMarkersFound`.

### Run summary

At the end of a run, the Actor writes one summary record to the default
key-value store, under the key `OUTPUT`. It holds:

- `urlsAudited`, `botUserAgentsChecked`, `recordsProduced` — run size.
- `verdictCounts` — a count of dataset records per verdict, for example
  `{"VISIBLE": 40, "THIN": 3, "BLOCKED_ROBOTS": 2}`.
- `previousDatasetId` — the dataset ID given in the input, or `null` if
  none was given.
- `changedVerdictUrls`, `changedVerdictCount` — the URLs whose verdict
  changed since `previousDatasetId`, and their count. Both are empty when
  `previousDatasetId` is not set.

#### Changed-verdict comparison

If you set `previousDatasetId`, the Actor reads that dataset's items after
the audit finishes and compares each `(url, bot)` pair's verdict against
the current run. A URL is reported as changed when at least one of its bot
verdicts differs from the matching record in the previous dataset. A URL or
bot that has no matching record in the previous dataset is not reported as
changed; there is nothing to compare it against.

The read from the previous dataset is bounded: it stops after 10,000 items
(500 URLs x 20 bot user agents, the largest one run can produce), even if
the dataset holds more. If the previous dataset cannot be read at all — a
bad ID, a permission error, or a network error — the run fails with a
clear status message. It does not produce a partial or silently-empty
comparison.

### Limits and safety

- HTTP and HTTPS only. No browser, no proxy, no JavaScript rendering.
- Private, loopback, link-local, and other reserved IP targets are
  rejected, on the initial URL and on every redirect hop.
- Response bytes, redirect hops, concurrency, and timeouts are all bounded.
- DNS is resolved once per request, ahead of the connection. A DNS answer
  can still change between that check and the connection (DNS rebinding).
  This is a known residual risk, not a guarantee.

### Billing

This Actor uses Apify's pay-per-event pricing model, with one charge event:

- **Event name:** `url-audited`.
- **Billing unit:** one charge per input URL that completes audit
  orchestration. The charge covers every selected bot user agent checked
  for that URL; individual bot records are never charged on their own. For
  example, one URL checked against 6 bot user agents produces 6 dataset
  records but is billed as 1 `url-audited` event.
- **When it is charged:** after the audit finishes and the URL's records
  are ready, before the records are pushed to the dataset.
- **What is not charged:** URLs rejected during input validation (for
  example, an empty `urls` list, or an input that fails the schema), and
  runs that fail before any URL result is produced. A URL rejected by the
  SSRF guard *during* the audit still counts, because it produces a
  `BLOCKED_SERVER` record and so completes audit orchestration.
- **If a charge call fails** (for example, a transient error reaching the
  Apify platform), the Actor logs the failure and still publishes the
  audit results. It does not discard correct, already-computed results
  over a billing error. The operator can reconcile a missed charge from
  the run logs.

The price for the `url-audited` event is **not set in this codebase**. It
must be configured separately, in the Apify Console or through the Apify
API, on the Actor's pay-per-event pricing page.

### Development

```bash
uv sync
uv run pytest
uv run ruff check .
```

# Actor input Schema

## `urls` (type: `array`):

Public HTTP(S) page URLs to check. Private and reserved IP targets are rejected. Maximum 500 URLs.

## `botUserAgents` (type: `array`):

User-Agent header values and robots.txt tokens for the AI crawlers to check. Optional; defaults to GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, and Bingbot. Add or replace tokens as new AI bots appear. Maximum 20.

## `referenceUserAgent` (type: `string`):

User-Agent used for one baseline fetch per URL. Each bot fetch is compared against this baseline to detect cloaking.

## `requiredMarkers` (type: `array`):

Plain-text strings that must appear on the page, for example a price or a contact heading. Optional. Maximum 20.

## `maxBytesPerFetch` (type: `integer`):

Response body byte cap for each HTTP fetch. Bytes past the cap are dropped.

## `concurrency` (type: `integer`):

Maximum number of URLs audited at the same time.

## `hostDelaySeconds` (type: `number`):

Minimum delay between two requests sent to the same host.

## `previousDatasetId` (type: `string`):

Optional. Dataset ID of an earlier run of this Actor. If set, the run reads that dataset and reports the URLs whose verdict changed since then. The dataset must be readable with the current Apify token.

## Actor input object example

```json
{
  "urls": [
    "https://example.com/"
  ],
  "botUserAgents": [
    "GPTBot",
    "ClaudeBot",
    "PerplexityBot",
    "CCBot",
    "Google-Extended",
    "Bingbot"
  ],
  "referenceUserAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
  "requiredMarkers": [],
  "maxBytesPerFetch": 2000000,
  "concurrency": 10,
  "hostDelaySeconds": 0.2
}
```

# Actor output Schema

## `auditRecords` (type: `string`):

No description

## `runSummary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("kingii98/ai-crawler-visibility-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("kingii98/ai-crawler-visibility-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call kingii98/ai-crawler-visibility-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kingii98/ai-crawler-visibility-auditor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/kd7oB62yOoazsUmYK/builds/zLJ4OybtbelmMeJe6/openapi.json