# AI Crawler Policy Drift Monitor (`checksmithcats/ai-crawler-policy-drift-monitor`) Actor

Tracks changes in public robots.txt, noindex, sitemap, and llms.txt signals against a stored baseline. Public crawler-facing signals only.

- **URL**: https://apify.com/checksmithcats/ai-crawler-policy-drift-monitor.md
- **Developed by:** [Checksmith Cats](https://apify.com/checksmithcats) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.40 / ai crawler drift report generated

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Crawler Policy Drift Monitor

Track changes in the public files and directives that crawlers can read.

This Actor is the scheduled companion to AI Crawler Policy Preflight. A successful run stores a compact baseline in the Actor key-value store. Later runs compare the same public signals with that baseline and return explicit change rows.

### What it monitors

- `robots.txt` directives for monitored AI and search crawlers
- sitemap references
- homepage and sampled-page `noindex` signals
- `X-Robots-Tag`
- `/llms.txt` presence and basic file metadata
- `/.well-known/security.txt` presence and expiry metadata

### Outputs

- `REPORT.json`, `REPORT.html`, and `REPORT.md`
- `CURRENT_SNAPSHOT.json`
- `DRIFT.json`
- dataset rows for findings and detected changes

If no public HTTP response can be observed, the run returns an incomplete-scan report, does not replace the previous baseline, and does not attempt a pay-per-event charge.

### Good fit

- scheduled checks after deploys, CMS changes, or theme changes
- evidence of when a public crawler directive changed
- agency before/after records
- machine-readable drift rows for an existing QA workflow

### Boundaries

This Actor records detectable changes in public signals. It does not:

- predict indexing, ranking, traffic, citations, or AI answers
- recommend whether a crawler should be allowed or blocked
- reproduce crawler-operator systems
- provide legal, SEO, security, or policy advice
- guarantee notification delivery outside normal Apify run outputs
- inspect private, login-gated, CAPTCHA-gated, or intranet URLs

Checksmith Cats is not affiliated with Google, Microsoft, OpenAI, Anthropic, Perplexity, Common Crawl, or any crawler operator.

### Input

- `siteUrl`: public HTTP(S) site URL
- `sitemapUrl`: optional public sitemap URL
- `stateKey`: optional baseline key; keep the default for normal scheduled use

Changing `siteUrl` while reusing the same `stateKey` replaces the baseline without reporting cross-site drift.

### Pricing

Pay per event:

| Event | Price |
|---|---:|
| `ai-crawler-drift-report-generated` | $0.40 per completed report |
| `ai-crawler-drift-site-checked` | $0.25 per successfully observed site |

A normal completed run checks one site and costs **$0.65**. Input-validation failures and runs with no observable public HTTP response are not charged. Set an Apify max cost per run if you need a hard account-level limit.

### Support boundary

Support covers reproducible Actor defects, documented input-format questions, missing output files, and a mismatch between documented and recorded billing events.

Support does not cover crawler-policy selection, SEO or legal advice, indexing or citation analysis, managed monitoring, custom integrations, or interpretation of a site's business strategy.

### Related Actors

- **AI Crawler Policy Preflight**: one-time crawler-policy inspection without a stored drift baseline
- **AI Visibility Signal Report**: broader inventory across crawler, structured-data, canonical/hreflang, and site-trust signals

# Actor input Schema

## `siteUrl` (type: `string`):

Public http(s) site URL to check. The Actor reads public pages only.

## `sitemapUrl` (type: `string`):

Optional public sitemap URL. If omitted, the Actor uses Sitemap lines from robots.txt when present.

## `stateKey` (type: `string`):

Optional key-value store key for the previous snapshot. Leave unchanged for normal scheduled use.

## Actor input object example

```json
{
  "siteUrl": "https://checksmithcats.com/",
  "stateKey": "AI_CRAWLER_POLICY_LAST_SNAPSHOT.json"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `report` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteUrl": "https://checksmithcats.com/",
    "stateKey": "AI_CRAWLER_POLICY_LAST_SNAPSHOT.json"
};

// Run the Actor and wait for it to finish
const run = await client.actor("checksmithcats/ai-crawler-policy-drift-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "siteUrl": "https://checksmithcats.com/",
    "stateKey": "AI_CRAWLER_POLICY_LAST_SNAPSHOT.json",
}

# Run the Actor and wait for it to finish
run = client.actor("checksmithcats/ai-crawler-policy-drift-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteUrl": "https://checksmithcats.com/",
  "stateKey": "AI_CRAWLER_POLICY_LAST_SNAPSHOT.json"
}' |
apify call checksmithcats/ai-crawler-policy-drift-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=checksmithcats/ai-crawler-policy-drift-monitor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/FBOA9bUx8YmWh24Ev/builds/hwKcwpWJmm7ORJ41k/openapi.json