Crawler Traffic Log Report

Turn submitted access logs into a compact report on crawler-claim traffic and HTTP error patterns.

The Actor reads common, combined, or JSONL access logs and returns aggregate bot-taxonomy matches, current robots-policy mismatch signals, and error-surface patterns.

What it reports

• pinned crawler-taxonomy version
• bot, vendor, and category request counts
• explicit verification-status labels
• low-confidence mismatch observations against submitted current robots.txt
• status-code distribution
• top redacted or hashed error and redirect path groups
• counts of sensitive-looking patterns

Optional DNS verification is off by default, limited to ten public IP addresses per verification-capable bot, and never queries private, loopback, reserved, or other non-global addresses.

Privacy boundary

Raw IPs, raw query strings, emails, and token-like values are not copied into generated reports, dataset rows, or Evidence Packs.

The submitted Actor input itself is handled by Apify and may remain in run storage according to account and platform retention settings. Redact sensitive logs before submission, keep DNS verification off unless needed, and delete run storage when your retention policy requires it.

Boundaries

This Actor does not establish:

• AI-training activity
• historical robots-policy violations
• crawler identity from a user-agent string alone
• the robots.txt policy that existed when an old log line was recorded
• server-incident root cause
• security, SEO, or legal compliance

It reports machine-detectable signals from the submitted material.

Usage controls and pricing

Pay per event:

Event	Price
crawler-traffic-log-report-generated	$0.35 per completed report
crawler-log-line-parsed	$0.0001 per parsed line, capped at 10,000 billed lines

Examples:

• 1,000 parsed lines: $0.45
• default 5,000 parsed lines: $0.85
• 10,000 or more parsed lines: $1.35 maximum billed amount per completed run

lineLimit defaults to 5,000 and can be raised to 50,000. Lines above the 10,000-line billing cap may still be analyzed but are not sent as billing events. Input-validation and log-intake failures are not charged. Set an Apify max cost per run if you need a lower account-level limit.

Support boundary

Support covers reproducible Actor defects, documented input-format questions, supported archive formats, missing output files, and a mismatch between documented and recorded billing events.

Support does not cover incident response, crawler attribution, historical policy reconstruction, server configuration, log repair, custom parsers, custom integrations, or interpretation of business outcomes.

Related Actors

• AI Crawler Policy Preflight: inspect what a public site currently declares
• AI Crawler Policy Drift Monitor: track changes in crawler-facing public files
• AI Visibility Signal Report: combine crawler policy with broader public-site signals