AI Crawler & Content-Signal Permission Delta Monitor

Your robots.txt can quietly lock out GPTBot or ClaudeBot overnight — and nothing tells you until your AI-driven traffic just stops.

Most sites never watch their own robots.txt for AI-crawler permission changes — a single Allow/Disallow flip for GPTBot, ClaudeBot, or any of 17 tracked bot tokens can happen with a routine config push and go unnoticed for weeks. The same blind spot applies to Cloudflare's Content-Signal line and to llms.txt, both of which can appear, disappear, or change without any alert reaching the people who depend on that access. This Actor solves that: it watches any domain and reports the instant its AI-crawler permissions change, so you find out the moment it happens instead of the moment your traffic does.

---

Why this outperforms a standard scraper

• Delta tracking, not re-scraping. Every record is fingerprinted on every run. Unchanged records are never re-delivered — and never billed.
• Pay only for what's new. A robots.txt Allow/Disallow flip for a tracked bot token costs $0.015. A Cloudflare Content-Signal flip or an llms.txt appearing/changing costs $0.006. Nothing else is charged — a domain with no changes since your last check returns nothing billable.
• Free baseline audit, then true delta monitoring. The first run against a domain establishes its permission baseline at no charge for unchanged state; with onlyNew on, every run after that delivers only actual changes across all 17 tracked bot tokens — not a fresh full re-check you have to diff yourself.

See it before you trust it

{
  "domain": "example.com",
  "changeType": "robots_txt_permission_flip",
  "botToken": "GPTBot",
  "previousPermission": "Allow",
  "currentPermission": "Disallow",
  "detectedAt": "2026-09-15T08:12:00Z"
}

The previousPermission / currentPermission pair on the exact botToken that flipped is what turns this into an alert you can act on, not just a robots.txt dump.

Zero-risk trial

Unchanged results cost $0.00 — the first baseline audit of a domain is free. Run it once against real data before you commit to anything:

curl -X POST "https://api.apify.com/v2/acts/eWDx4XY54R5GXysFi/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"domains":["example.com"],"onlyNew":true}'

import requests

response = requests.post(
    "https://api.apify.com/v2/acts/eWDx4XY54R5GXysFi/run-sync-get-dataset-items",
    params={"token": "<YOUR_API_TOKEN>"},
    json={"domains": ["example.com"], "onlyNew": True},
)
records = response.json()
print(f"{len(records)} records returned")

const response = await fetch(
  "https://api.apify.com/v2/acts/eWDx4XY54R5GXysFi/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>",
  {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ domains: ["example.com"], onlyNew: true }),
  }
);
const records = await response.json();
console.log(records.length + " records returned");

Pricing

Event	What it means	Price
AI Crawler Permission Change	A robots.txt Allow/Disallow flip for a tracked bot token (e.g. GPTBot, ClaudeBot).	$0.015
Content-Signal / llms.txt Change	A Cloudflare Content-Signal flip, or an llms.txt appearing/changing.	$0.006

Actor-start fee: $0.00005/GB-memory (one-time per run, not per record).

What you get on every record

• Coverage of all 17 tracked robots.txt User-agent tokens, including GPTBot, ClaudeBot, Google-Extended, and PerplexityBot
• Cloudflare Content-Signal line parsing on every checked domain, when checkContentSignals is on
• llms.txt presence and content tracked across runs, when checkLlmsTxt is on
• With onlyNew enabled, a repeat run of an already-baselined domain returns only the bot tokens or signals that actually changed
• A free baseline audit on first check, so you see the full current permission state before paying for anything

Input parameters

Field	Type	Description	Default
domains	array	Bare hostnames to monitor — no protocol, no path (e.g. openai.com).	[]
trackedBots	array	The 17 robots.txt User-agent tokens watched: GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more.	17 tokens (see docs)
checkContentSignals	boolean	Parses any Cloudflare Content-Signal: line in robots.txt.	true
checkLlmsTxt	boolean	Tracks presence/absence and changes to a domain's llms.txt.	true
onlyNew	boolean	A repeat run of an already-baselined domain only delivers actual changes.	true

Source & reliability

This Actor reads directly from each monitored domain's own robots.txt and llms.txt — the same files any real AI crawler would fetch — rather than relying on a third-party permissions database, so what it reports is exactly what GPTBot, ClaudeBot, and the other 15 tracked tokens would actually see. Because it tracks all 17 bot tokens plus Cloudflare's Content-Signal line on every check, and only surfaces rows once a domain is baselined and onlyNew is on, repeat monitoring stays both comprehensive and cheap.