AI Crawler Policy Drift Monitor

Track changes in the public files and directives that crawlers can read.

This Actor is the scheduled companion to AI Crawler Policy Preflight. A successful run stores a compact baseline in the Actor key-value store. Later runs compare the same public signals with that baseline and return explicit change rows.

What it monitors

• robots.txt directives for monitored AI and search crawlers
• sitemap references
• homepage and sampled-page noindex signals
• X-Robots-Tag
• /llms.txt presence and basic file metadata
• /.well-known/security.txt presence and expiry metadata

Outputs

• REPORT.json, REPORT.html, and REPORT.md
• CURRENT_SNAPSHOT.json
• DRIFT.json
• dataset rows for findings and detected changes

If no public HTTP response can be observed, the run returns an incomplete-scan report, does not replace the previous baseline, and does not attempt a pay-per-event charge.

Good fit

• scheduled checks after deploys, CMS changes, or theme changes
• evidence of when a public crawler directive changed
• agency before/after records
• machine-readable drift rows for an existing QA workflow

Boundaries

This Actor records detectable changes in public signals. It does not:

• predict indexing, ranking, traffic, citations, or AI answers
• recommend whether a crawler should be allowed or blocked
• reproduce crawler-operator systems
• provide legal, SEO, security, or policy advice
• guarantee notification delivery outside normal Apify run outputs
• inspect private, login-gated, CAPTCHA-gated, or intranet URLs

Checksmith Cats is not affiliated with Google, Microsoft, OpenAI, Anthropic, Perplexity, Common Crawl, or any crawler operator.

Input

• siteUrl: public HTTP(S) site URL
• sitemapUrl: optional public sitemap URL
• stateKey: optional baseline key; keep the default for normal scheduled use

Changing siteUrl while reusing the same stateKey replaces the baseline without reporting cross-site drift.

Pricing

Pay per event:

Event	Price
ai-crawler-drift-report-generated	$0.40 per completed report
ai-crawler-drift-site-checked	$0.25 per successfully observed site

A normal completed run checks one site and costs $0.65. Input-validation failures and runs with no observable public HTTP response are not charged. Set an Apify max cost per run if you need a hard account-level limit.

Support boundary

Support covers reproducible Actor defects, documented input-format questions, missing output files, and a mismatch between documented and recorded billing events.

Support does not cover crawler-policy selection, SEO or legal advice, indexing or citation analysis, managed monitoring, custom integrations, or interpretation of a site's business strategy.

Related Actors

• AI Crawler Policy Preflight: one-time crawler-policy inspection without a stored drift baseline
• AI Visibility Signal Report: broader inventory across crawler, structured-data, canonical/hreflang, and site-trust signals

Common input errors

Start with the published example and keep siteUrl, stateKey in the documented JSON shape. Malformed JSON, unsupported URL schemes, missing required values, and inputs above the stated limits are rejected rather than repaired silently.