AC-VIS-01 — AI Crawler Content Visibility Auditor

This Actor checks whether AI answer-engine bots (for example GPTBot, ClaudeBot, PerplexityBot) can read the text on your pages. It is HTTP-only: it never uses a browser, a proxy, or JavaScript rendering.

What it does

For each URL in the input:

1. It checks the URL points at a public HTTP(S) host. Private and reserved IP targets (for example 127.0.0.1, 10.0.0.0/8, link-local addresses) are rejected before any request is sent.
2. It fetches the page once with a reference (normal browser) user agent.
3. For each bot user agent in the input, it:
   • Checks robots.txt for that bot.
   • Fetches the page with that bot's user agent, if robots.txt allows it.
   • Extracts the visible plain text and any <meta name="robots"> directives.
   • Compares the extracted text length against the reference fetch.
   • Checks each required text marker against the raw HTTP response body (not the extracted plain text), so a marker can match text inside an HTML tag or attribute, not only visible page text.
   • Assigns a verdict: VISIBLE, THIN, BLOCKED_ROBOTS, BLOCKED_SERVER, or CLOAKED.

One record is pushed to the default dataset for each URL and bot user agent pair.

Input

See .actor/input_schema.json for the full schema. Main fields:

Field	Type	Required	Description
urls	array of string	yes	Public HTTP(S) page URLs to check. Max 500.
botUserAgents	array of string	no	Bot user agents to check. Default: GPTBot, ClaudeBot, PerplexityBot, CCBot, Google-Extended, Bingbot. Add or replace tokens as new AI bots appear. Max 20.
referenceUserAgent	string	no	Baseline user agent used once per URL to detect cloaking.
requiredMarkers	array of string	no	Plain-text strings that must appear in the raw HTTP response body of the page. Max 20.
maxBytesPerFetch	integer	no	Response body byte cap per fetch. 1,024–5,000,000. Default 2,000,000.
concurrency	integer	no	Max URLs audited at the same time. 1–20. Default 10.
hostDelaySeconds	number	no	Minimum delay between two requests to the same host. 0–5. Default 0.2.
previousDatasetId	string	no	Dataset ID from an earlier run. If set, the run reports which URLs changed verdict since that dataset.

Output

One dataset record per URL and bot user agent. See .actor/dataset_schema.json for the full field list. Key fields:

• url, bot — the audited page and the bot user agent.
• verdict — VISIBLE, THIN, BLOCKED_ROBOTS, BLOCKED_SERVER, or CLOAKED.
• reasonCode — a machine-readable reason for the verdict.
• httpStatus, finalUrl, redirectHops, robotsAllowed.
• extractedTextChars, textDeltaVsReference, requiredMarkersFound.

Run summary

At the end of a run, the Actor writes one summary record to the default key-value store, under the key OUTPUT. It holds:

• urlsAudited, botUserAgentsChecked, recordsProduced — run size.
• verdictCounts — a count of dataset records per verdict, for example {"VISIBLE": 40, "THIN": 3, "BLOCKED_ROBOTS": 2}.
• previousDatasetId — the dataset ID given in the input, or null if none was given.
• changedVerdictUrls, changedVerdictCount — the URLs whose verdict changed since previousDatasetId, and their count. Both are empty when previousDatasetId is not set.

Changed-verdict comparison

If you set previousDatasetId, the Actor reads that dataset's items after the audit finishes and compares each (url, bot) pair's verdict against the current run. A URL is reported as changed when at least one of its bot verdicts differs from the matching record in the previous dataset. A URL or bot that has no matching record in the previous dataset is not reported as changed; there is nothing to compare it against.

The read from the previous dataset is bounded: it stops after 10,000 items (500 URLs x 20 bot user agents, the largest one run can produce), even if the dataset holds more. If the previous dataset cannot be read at all — a bad ID, a permission error, or a network error — the run fails with a clear status message. It does not produce a partial or silently-empty comparison.

Limits and safety

• HTTP and HTTPS only. No browser, no proxy, no JavaScript rendering.
• Private, loopback, link-local, and other reserved IP targets are rejected, on the initial URL and on every redirect hop.
• Response bytes, redirect hops, concurrency, and timeouts are all bounded.
• DNS is resolved once per request, ahead of the connection. A DNS answer can still change between that check and the connection (DNS rebinding). This is a known residual risk, not a guarantee.

Billing

This Actor uses Apify's pay-per-event pricing model, with one charge event:

• Event name: url-audited.
• Billing unit: one charge per input URL that completes audit orchestration. The charge covers every selected bot user agent checked for that URL; individual bot records are never charged on their own. For example, one URL checked against 6 bot user agents produces 6 dataset records but is billed as 1 url-audited event.
• When it is charged: after the audit finishes and the URL's records are ready, before the records are pushed to the dataset.
• What is not charged: URLs rejected during input validation (for example, an empty urls list, or an input that fails the schema), and runs that fail before any URL result is produced. A URL rejected by the SSRF guard during the audit still counts, because it produces a BLOCKED_SERVER record and so completes audit orchestration.
• If a charge call fails (for example, a transient error reaching the Apify platform), the Actor logs the failure and still publishes the audit results. It does not discard correct, already-computed results over a billing error. The operator can reconcile a missed charge from the run logs.

The price for the url-audited event is not set in this codebase. It must be configured separately, in the Apify Console or through the Apify API, on the Actor's pay-per-event pricing page.

Development

uv sync
uv run pytest
uv run ruff check .