Compare crawler requests with robots policy

Created by

Checksmith Cats

Actor

Crawler Traffic Log Report

Summarize crawler-claim traffic, current robots-policy mismatch signals, status codes, and redacted HTTP error patterns from submitted access logs.

Try for free

Crawler Traffic Log Reportchecksmithcats/crawler-traffic-log-report

Product

Check id

Title

Severity

+3 fields

Input

Log text:203.0.113.10 - - [07/Jul/2026:10:00:00 +0000] "GET /ai/private?token=secret HTTP/1.1" 200 123 "-" "GPTBot/1.0" 66.249.66.1 - - [07/Jul/2026:10:00:01 +0000] "GET /public HTTP/1.1" 200 456 "-" "Googlebot/2.1 (+http://www.google.com/bot.html)" 198.51.100.2 - - [07/Jul/2026:10:00:02 +0000] "GET /missing HTTP/1.1" 404 12 "https://ref.example/path?utm=abc" "Mozilla/5.0"

Current robots.txt:User-agent: GPTBot Disallow: /ai User-agent: * Disallow: /private

Mode:both

Line limit:5000

Perform DNS verification:false

Output fields

Product

Check id

Title

Severity

Subject id

Summary

Generated at

How it works

Sign up on Apify or log in

The Crawler Traffic Log Report Actor is ready to run with the inputs already set for "Compare crawler requests with robots policy". Sign up or log in and you'll land in Apify Console with a copy of this exact task.

Start the "Compare crawler requests with robots policy" task

The Crawler Traffic Log Report Actor will start running based on the input automatically. You can also change all the input fields, like log text, current robots.txt, and more, to match your specific use case.

Receive the "Compare crawler requests with robots policy" output

Monitor the progress in real time. You will be notified as soon as your dataset is complete and ready for review. Each result includes data such as product, check id, title, and more.

Integrate into your workflow

The final output is delivered in JSON, CSV, or Excel format, ready to be plugged into your workflow. You can also run the "Compare crawler requests with robots policy" task via API.