Monitor robots.txt changes to prevent site de-indexing
Created by
DatawinderLabs
Actor
Robots.txt Monitor
Track shifts in crawl directives, user-agent rules, and disallow flags. Prevent staging errors from blocking search bots and tanking organic rankings.
Robots.txt Monitordatawinder/robots-txt-monitor
Baseline Status
Unchanged Flag
Modification Type
Severity
+3 fieldsTextNumberBooleanListObject
Input
robots.txt URL(required):https://www.google.com/robots.txt
Follow Redirects:true
Timeout (ms):10000
Output fields
Baseline Status
Unchanged Flag
Modification Type
Severity
Change Details
Critical Alert Count
Warning
Sign up on Apify01
Create your Apify account to access the Robots.txt Monitor.
Start the run02
The Actor will start running based on the input automatically.
Receive the output03
Monitor the progress in real-time. You will be notified as soon as your dataset is complete and ready for review.
Integrate into your workflow04
The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow.
