# AI Crawler & Content-Signal Permission Delta Monitor (`stefano_seggio/ai-crawler-content-signal-permission-monitor`) Actor

Watches any domain and reports the instant its AI-crawler permissions change: a robots.txt Allow/Disallow flip for GPTBot, ClaudeBot and 15 other tracked tokens, a Cloudflare Content-Signal flip, or llms.txt appearing/changing. Pay-per-event: billed only for what changed, with a free baseline audit.

- **URL**: https://apify.com/stefano\_seggio/ai-crawler-content-signal-permission-monitor.md
- **Developed by:** [Stefano Seggio](https://apify.com/stefano_seggio) (community)
- **Categories:** SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $15.00 / 1,000 ai crawler permission changes

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler & Content-Signal Permission Delta Monitor

#### Your robots.txt can quietly lock out GPTBot or ClaudeBot overnight — and nothing tells you until your AI-driven traffic just stops.

Most sites never watch their own robots.txt for AI-crawler permission changes — a single Allow/Disallow flip for GPTBot, ClaudeBot, or any of 17 tracked bot tokens can happen with a routine config push and go unnoticed for weeks. The same blind spot applies to Cloudflare's Content-Signal line and to llms.txt, both of which can appear, disappear, or change without any alert reaching the people who depend on that access. **This Actor solves that**: it watches any domain and reports the instant its AI-crawler permissions change, so you find out the moment it happens instead of the moment your traffic does.

***

### Why this outperforms a standard scraper

- **Delta tracking, not re-scraping.** Every record is fingerprinted on every run. Unchanged records are never re-delivered — and never billed.
- **Pay only for what's new.** A robots.txt Allow/Disallow flip for a tracked bot token costs $0.015. A Cloudflare Content-Signal flip or an llms.txt appearing/changing costs $0.006. Nothing else is charged — a domain with no changes since your last check returns nothing billable.
- **Free baseline audit, then true delta monitoring.** The first run against a domain establishes its permission baseline at no charge for unchanged state; with `onlyNew` on, every run after that delivers only actual changes across all 17 tracked bot tokens — not a fresh full re-check you have to diff yourself.

### See it before you trust it

```json
{
  "domain": "example.com",
  "changeType": "robots_txt_permission_flip",
  "botToken": "GPTBot",
  "previousPermission": "Allow",
  "currentPermission": "Disallow",
  "detectedAt": "2026-09-15T08:12:00Z"
}
```

The `previousPermission` / `currentPermission` pair on the exact `botToken` that flipped is what turns this into an alert you can act on, not just a robots.txt dump.

### Zero-risk trial

Unchanged results cost **$0.00** — the first baseline audit of a domain is free. Run it once against real data before you commit to anything:

```bash
curl -X POST "https://api.apify.com/v2/acts/eWDx4XY54R5GXysFi/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"domains":["example.com"],"onlyNew":true}'
```

```python
import requests

response = requests.post(
    "https://api.apify.com/v2/acts/eWDx4XY54R5GXysFi/run-sync-get-dataset-items",
    params={"token": "<YOUR_API_TOKEN>"},
    json={"domains": ["example.com"], "onlyNew": True},
)
records = response.json()
print(f"{len(records)} records returned")
```

```javascript
const response = await fetch(
  "https://api.apify.com/v2/acts/eWDx4XY54R5GXysFi/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>",
  {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ domains: ["example.com"], onlyNew: true }),
  }
);
const records = await response.json();
console.log(records.length + " records returned");
```

### Pricing

| Event | What it means | Price |
|---|---|---|
| AI Crawler Permission Change | A robots.txt Allow/Disallow flip for a tracked bot token (e.g. GPTBot, ClaudeBot). | $0.015 |
| Content-Signal / llms.txt Change | A Cloudflare Content-Signal flip, or an llms.txt appearing/changing. | $0.006 |

Actor-start fee: $0.00005/GB-memory (one-time per run, not per record).

### What you get on every record

- Coverage of all 17 tracked robots.txt User-agent tokens, including GPTBot, ClaudeBot, Google-Extended, and PerplexityBot
- Cloudflare Content-Signal line parsing on every checked domain, when `checkContentSignals` is on
- llms.txt presence and content tracked across runs, when `checkLlmsTxt` is on
- With `onlyNew` enabled, a repeat run of an already-baselined domain returns only the bot tokens or signals that actually changed
- A free baseline audit on first check, so you see the full current permission state before paying for anything

### Input parameters

| Field | Type | Description | Default |
|---|---|---|---|
| domains | array | Bare hostnames to monitor — no protocol, no path (e.g. `openai.com`). | `[]` |
| trackedBots | array | The 17 robots.txt User-agent tokens watched: GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more. | 17 tokens (see docs) |
| checkContentSignals | boolean | Parses any Cloudflare `Content-Signal:` line in robots.txt. | true |
| checkLlmsTxt | boolean | Tracks presence/absence and changes to a domain's llms.txt. | true |
| onlyNew | boolean | A repeat run of an already-baselined domain only delivers actual changes. | true |

### Source & reliability

This Actor reads directly from each monitored domain's own robots.txt and llms.txt — the same files any real AI crawler would fetch — rather than relying on a third-party permissions database, so what it reports is exactly what GPTBot, ClaudeBot, and the other 15 tracked tokens would actually see. Because it tracks all 17 bot tokens plus Cloudflare's Content-Signal line on every check, and only surfaces rows once a domain is baselined and `onlyNew` is on, repeat monitoring stays both comprehensive and cheap.

# Changelog

This Actor's version history is a separate document: https://apify.com/stefano\_seggio/ai-crawler-content-signal-permission-monitor/changelog.md

# Actor input Schema

## `domains` (type: `array`):

Bare hostnames to monitor - no protocol, no path, no trailing slash. Example: "cloudflare.com", "openai.com". Each domain is checked at its own https://{domain}/robots.txt (and, if enabled, /llms.txt and /llms-full.txt). Mix your own domains with competitor/vendor domains freely - each gets fully independent delta state.

## `trackedBots` (type: `array`):

The robots.txt User-agent tokens whose Allow/Disallow directives are watched for changes over time. Defaults to 18 currently-documented AI-crawler tokens spanning OpenAI, Anthropic, Google, Apple, Perplexity, Common Crawl, ByteDance, Amazon, Meta, Diffbot, and Cohere. Add a newly-published crawler token, or narrow this to one vendor family, by editing this list.

## `checkContentSignals` (type: `boolean`):

Parses any "Content-Signal: search=..., ai-input=..., ai-train=..." line found in robots.txt (Cloudflare's proposed extension to the standard, formalized as the IETF draft draft-romm-aipref-contentsignals) and tracks all three category values for change over time. Turn this off only if you exclusively care about classic robots.txt Allow/Disallow directives.

## `checkLlmsTxt` (type: `boolean`):

Fetches both well-known paths (llmstxt.org's proposed convention) and tracks presence/absence plus a content hash, so an added, removed, or edited llms.txt is reported as a CHANGED event. llms.txt is an informal community proposal, not a ratified standard - this Actor reports its presence and content as a signal, never as a compliance guarantee.

## `onlyNew` (type: `boolean`):

When true (default), a repeat run of an already-baselined domain only delivers ALLOWED, DISALLOWED, and CHANGED events - the free NO\_CHANGE row is suppressed when nothing differs. A domain's first-ever check always delivers a free BASELINE\_SNAPSHOT regardless of this setting, since that is the point of adding a domain to the watchlist. Set to false to also see an explicit "nothing changed" row on every run.

## `maxDomainsPerRun` (type: `integer`):

Hard cap on how many domains from the list are checked (and, where applicable, charged) in a single run. Domains beyond this cap are simply checked on the next run - nothing about their state is lost or marked as seen. Capped at 30 so this Actor's own worst case fits inside its 300-second default run timeout with real margin: with default concurrency (15) and default retry settings (requestTimeoutSecs=15, maxRetries=4), one domain's robots.txt/llms.txt/llms-full.txt fetches (run concurrently) can together take up to ~91s if every one of them needs its full 5-attempt retry budget, so a full run of up to 30 domains (2 concurrency waves) has a worst case of ~182s - about 61% of the 300s timeout. Raising this above 30 without also raising this Actor's run timeout (via the API's "timeout" run option or a Console/schedule override, roughly +91s per extra 15 domains) risks the run being killed mid-walk on a bad day; that is safe (no state is lost or double-committed) but wastes the retry compute already spent on the domains that did not finish.

## `concurrency` (type: `integer`):

Caps the total number of in-flight HTTP requests across the whole run. Each domain fans out to up to three independent resource fetches (robots.txt, llms.txt, llms-full.txt), so this bounds total concurrent load across every domain being checked, not per-domain concurrency. Raise cautiously - most of the wall-clock cost of a large watchlist run is waiting on slow or unresponsive third-party domains, not this Actor's own compute.

## `requestTimeoutSecs` (type: `integer`):

Aborts a single robots.txt/llms.txt/llms-full.txt request that takes longer than this, so one slow or unresponsive domain never stalls the whole run. Applies per individual attempt, not per domain overall.

## `maxRetries` (type: `integer`):

Retry attempts (exponential backoff with jitter, capped at 15 seconds between attempts) before a single request is given up on as failed. Applies only to retryable outcomes (429 and 5xx responses, or a request timeout) - any other non-2xx, non-404 response is treated as final immediately and does not consume retry budget. A 404 is never a failure: it is parsed as a valid "resource absent" signal.

## `deltaStateName` (type: `string`):

Names this run's persistent delta-state store. Use a distinct name per independent schedule (for example "own-sites" versus "competitor-watchlist") so their baselines never cross-contaminate each other.

## `resetState` (type: `boolean`):

Wipes all remembered state for every domain under this deltaStateName before the run starts, so every domain is re-baselined from scratch on this run (delivered as free BASELINE\_SNAPSHOT records, subject to onlyNew's usual free-baseline behavior).

## Actor input object example

```json
{
  "domains": [
    "cloudflare.com"
  ],
  "trackedBots": [
    "GPTBot",
    "ChatGPT-User",
    "OAI-SearchBot",
    "ClaudeBot",
    "Claude-User",
    "Claude-SearchBot",
    "Google-Extended",
    "Applebot-Extended",
    "PerplexityBot",
    "Perplexity-User",
    "CCBot",
    "Bytespider",
    "Amazonbot",
    "Meta-ExternalAgent",
    "Meta-ExternalFetcher",
    "Diffbot",
    "cohere-ai",
    "omgilibot"
  ],
  "checkContentSignals": true,
  "checkLlmsTxt": true,
  "onlyNew": true,
  "maxDomainsPerRun": 30,
  "concurrency": 15,
  "requestTimeoutSecs": 15,
  "maxRetries": 4,
  "deltaStateName": "default",
  "resetState": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "cloudflare.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("stefano_seggio/ai-crawler-content-signal-permission-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": ["cloudflare.com"] }

# Run the Actor and wait for it to finish
run = client.actor("stefano_seggio/ai-crawler-content-signal-permission-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "cloudflare.com"
  ]
}' |
apify call stefano_seggio/ai-crawler-content-signal-permission-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,stefano_seggio/ai-crawler-content-signal-permission-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/eWDx4XY54R5GXysFi/builds/rkWahJmh3T0mAifbS/openapi.json