Detect robots and llms.txt policy changes

Created by

Checksmith Cats

Actor

AI Crawler Policy Drift Monitor

Track changes in public robots.txt, noindex, sitemap, and llms.txt signals against a stored baseline, with explicit drift rows and safe failed runs.

Try for free

AI Crawler Policy Drift Monitorchecksmithcats/ai-crawler-policy-drift-monitor

Product

Check id

Severity

Summary

+2 fields

Input

Site URL(required):https://checksmithcats.com/

State key:AI_CRAWLER_POLICY_LAST_SNAPSHOT.json

Output fields

Product

Check id

Severity

Summary

Subject id

Generated at

How it works

Sign up on Apify or log in

The AI Crawler Policy Drift Monitor Actor is ready to run with the inputs already set for "Detect robots and llms.txt policy changes". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Detect robots and llms.txt policy changes" task

The AI Crawler Policy Drift Monitor Actor will start running based on the input automatically. You can also change all the input fields, like site url and state key, to match your specific use case.

Receive the "Detect robots and llms.txt policy changes" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as product, check id, severity, and more.

Integrate into your workflow

The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow. You can also run the "Detect robots and llms.txt policy changes" task via API.