AI Crawler & Content-Signal Permission Delta Monitor
Pricing
from $15.00 / 1,000 ai crawler permission changes
Watches any domain and reports the instant its AI-crawler permissions change: a robots.txt Allow/Disallow flip for GPTBot, ClaudeBot and 15 other tracked tokens, a Cloudflare Content-Signal flip, or llms.txt appearing/changing. Pay-per-event: billed only for what changed, with a free baseline audit.
Timeout-budget fix from a fleet-wide audit: the default 300s run timeout was
too short for this Actor's own default worst-case retry workload on a large
domain list (real arithmetic in AGENTS.md). domainSource.ts now fetches
robots.txt/llms.txt/llms-full.txt concurrently instead of sequencing
robots.txt before the other two, halving the worst-case time per domain
(~182s to ~91s) with no behavior change. maxDomainsPerRun's default and
maximum were lowered from 200/2000 to 30 so the actor's own worst case
(~182s for 30 domains) leaves genuine margin under the 300s timeout;
domains beyond the cap are still deferred to the next run with no state
loss, as before. See AGENTS.md's 2026-09-19 entry for the full arithmetic
and the alternatives considered.
Initial build. Delta engine watching robots.txt AI-crawler directives (18
tracked tokens), Cloudflare Content-Signal categories, and llms.txt/
llms-full.txt presence and content, classified as BASELINE_SNAPSHOT /
ALLOWED / DISALLOWED / CHANGED / NO_CHANGE via SHA-256 status/content
fingerprints. Pay-Per-Event billing via the two-argument
pushData(record, eventName)cloudflare.com domain, including a
tamper-and-detect run proving the delta classification genuinely detects a
permission change, not just a mocked one. An internal adversarial audit pass
found and fixed three real issues before this version: an inconsistent
camelCase output field, a duplicate/case-variant domain double-charge risk,
and an imprecise description of the charge-limit halt's concurrency
behavior - see AGENTS.md for the full account of each. Not yet pushed to
Apify Console or configured for monetization.