Go to example tasks
Compare crawler requests with robots policy
Created by
Checksmith Cats
Summarize crawler-claim traffic, current robots-policy mismatch signals, status codes, and redacted HTTP error patterns from submitted access logs.
Crawler Traffic Log Reportchecksmithcats/crawler-traffic-log-report
Product
Check id
Title
Severity
+3 fieldsTextNumberBooleanListObject
Input
Log text:203.0.113.10 - - [07/Jul/2026:10:00:00 +0000] "GET /ai/private?token=secret HTTP/1.1" 200 123 "-" "GPTBot/1.0"
66.249.66.1 - - [07/Jul/2026:10:00:01 +0000] "GET /public HTTP/1.1" 200 456 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)"
198.51.100.2 - - [07/Jul/2026:10:00:02 +0000] "GET /missing HTTP/1.1" 404 12 "https://ref.example/path?utm=abc" "Mozilla/5.0"
Current robots.txt:User-agent: GPTBot
Disallow: /ai
User-agent: *
Disallow: /private
Mode:both
Line limit:5000
Perform DNS verification:false
Output fields
Product
Check id
Title
Severity
Subject id
Summary
Generated at
Sign up on Apify01
Create your Apify account to access the Crawler Traffic Log Report.
Start the run02
The Actor will start running based on the input automatically.
Receive the output03
Monitor the progress in real-time. You will be notified as soon as your dataset is complete and ready for review.
Integrate into your workflow04
The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow.
