AI Crawler Policy Drift Monitor

Track changes in the public files and directives that crawlers can read.

This Actor is the scheduled companion to AI Crawler Policy Preflight. A successful run stores a compact baseline in the Actor key-value store. Later runs compare the same public signals with that baseline and return explicit change rows.

What it monitors

• robots.txt directives for monitored AI and search crawlers
• sitemap references
• homepage and sampled-page noindex signals
• X-Robots-Tag
• /llms.txt presence and basic file metadata
• /.well-known/security.txt presence and expiry metadata

Outputs

• REPORT.json, REPORT.html, and REPORT.md
• CURRENT_SNAPSHOT.json
• DRIFT.json
• dataset rows for findings and detected changes

If no public HTTP response can be observed, the run returns an incomplete-scan report, does not replace the previous baseline, and does not attempt a pay-per-event charge.

Good fit

• scheduled checks after deploys, CMS changes, or theme changes
• evidence of when a public crawler directive changed
• agency before/after records
• machine-readable drift rows for an existing QA workflow

Boundaries

This Actor records detectable changes in public signals. It does not:

• predict indexing, ranking, traffic, citations, or AI answers
• recommend whether a crawler should be allowed or blocked
• reproduce crawler-operator systems
• provide legal, SEO, security, or policy advice
• guarantee notification delivery outside normal Apify run outputs
• inspect private, login-gated, CAPTCHA-gated, or intranet URLs

Checksmith Cats is not affiliated with Google, Microsoft, OpenAI, Anthropic, Perplexity, Common Crawl, or any crawler operator.

Input

• siteUrl: public HTTP(S) site URL
• sitemapUrl: optional public sitemap URL
• stateKey: optional baseline key; keep the default for normal scheduled use

Changing siteUrl while reusing the same stateKey replaces the baseline without reporting cross-site drift.

Pricing

Pay per event:

Event	Price
ai-crawler-drift-report-generated	$0.40 per completed report
ai-crawler-drift-site-checked	$0.25 per successfully observed site

A normal completed run checks one site and costs $0.65. Input-validation failures and runs with no observable public HTTP response are not charged. Set an Apify max cost per run if you need a hard account-level limit.

Support boundary

Support covers reproducible Actor defects, documented input-format questions, missing output files, and a mismatch between documented and recorded billing events.

Support does not cover crawler-policy selection, SEO or legal advice, indexing or citation analysis, managed monitoring, custom integrations, or interpretation of a site's business strategy.

Related Actors

• AI Crawler Policy Preflight: one-time crawler-policy inspection without a stored drift baseline
• AI Visibility Signal Report: broader inventory across crawler, structured-data, canonical/hreflang, and site-trust signals