# Crawler Traffic Log Report (`checksmithcats/crawler-traffic-log-report`) Actor

Summarizes crawler-claim traffic, current robots-policy mismatch signals, status codes, and redacted HTTP error patterns from submitted access logs.

- **URL**: https://apify.com/checksmithcats/crawler-traffic-log-report.md
- **Developed by:** [Checksmith Cats](https://apify.com/checksmithcats) (community)
- **Categories:** Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.35 / crawler traffic log report generated

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Crawler Traffic Log Report

Turn submitted access logs into a compact report on crawler-claim traffic and HTTP error patterns.

The Actor reads common, combined, or JSONL access logs and returns aggregate bot-taxonomy matches, current robots-policy mismatch signals, and error-surface patterns.

### What it reports

- pinned crawler-taxonomy version
- bot, vendor, and category request counts
- explicit verification-status labels
- low-confidence mismatch observations against submitted current `robots.txt`
- status-code distribution
- top redacted or hashed error and redirect path groups
- counts of sensitive-looking patterns

Optional DNS verification is off by default, limited to ten public IP addresses per verification-capable bot, and never queries private, loopback, reserved, or other non-global addresses.

### Privacy boundary

Raw IPs, raw query strings, emails, and token-like values are not copied into generated reports, dataset rows, or Evidence Packs.

The submitted Actor input itself is handled by Apify and may remain in run storage according to account and platform retention settings. Redact sensitive logs before submission, keep DNS verification off unless needed, and delete run storage when your retention policy requires it.

### Boundaries

This Actor does not establish:

- AI-training activity
- historical robots-policy violations
- crawler identity from a user-agent string alone
- the robots.txt policy that existed when an old log line was recorded
- server-incident root cause
- security, SEO, or legal compliance

It reports machine-detectable signals from the submitted material.

### Usage controls and pricing

Pay per event:

| Event | Price |
|---|---:|
| `crawler-traffic-log-report-generated` | $0.35 per completed report |
| `crawler-log-line-parsed` | $0.0001 per parsed line, capped at 10,000 billed lines |

Examples:

- 1,000 parsed lines: **$0.45**
- default 5,000 parsed lines: **$0.85**
- 10,000 or more parsed lines: **$1.35 maximum billed amount per completed run**

`lineLimit` defaults to 5,000 and can be raised to 50,000. Lines above the 10,000-line billing cap may still be analyzed but are not sent as billing events. Input-validation and log-intake failures are not charged. Set an Apify max cost per run if you need a lower account-level limit.

### Support boundary

Support covers reproducible Actor defects, documented input-format questions, supported archive formats, missing output files, and a mismatch between documented and recorded billing events.

Support does not cover incident response, crawler attribution, historical policy reconstruction, server configuration, log repair, custom parsers, custom integrations, or interpretation of business outcomes.

### Related Actors

- **AI Crawler Policy Preflight**: inspect what a public site currently declares
- **AI Crawler Policy Drift Monitor**: track changes in crawler-facing public files
- **AI Visibility Signal Report**: combine crawler policy with broader public-site signals

# Actor input Schema

## `logText` (type: `string`):

Access log text in common, combined, or JSONL format. Provide either this field or logBase64.

## `logBase64` (type: `string`):

Base64-encoded plain, .gz, or .zip log file. Provide either this field or logText.

## `fileName` (type: `string`):

Optional original file name. Used only to detect .gz or .zip containers.

## `robotsTxt` (type: `string`):

Optional current robots.txt text. Mismatch findings are low confidence because historical policy at log time is unknown.

## `mode` (type: `string`):

Crawler policy report, error-surface report, or both.

## `lineLimit` (type: `integer`):

Maximum log lines to parse from the submitted input. Billing is capped at 10000 parsed-line events per run.

## `performDnsVerification` (type: `boolean`):

Optional bounded reverse-DNS/forward-DNS check for verification-capable crawler claims. Off by default.

## `verificationIpLimit` (type: `integer`):

Maximum distinct IPs per verification-capable bot to check when DNS verification is enabled.

## Actor input object example

```json
{
  "logText": "203.0.113.10 - - [07/Jul/2026:10:00:00 +0000] \"GET /docs HTTP/1.1\" 200 123 \"-\" \"GPTBot/1.0\"\n198.51.100.2 - - [07/Jul/2026:10:00:02 +0000] \"GET /missing HTTP/1.1\" 404 12 \"-\" \"Mozilla/5.0\"",
  "fileName": "access.log",
  "mode": "both",
  "lineLimit": 5000,
  "performDnsVerification": false,
  "verificationIpLimit": 5
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `report` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "logText": `203.0.113.10 - - [07/Jul/2026:10:00:00 +0000] "GET /docs HTTP/1.1" 200 123 "-" "GPTBot/1.0"
198.51.100.2 - - [07/Jul/2026:10:00:02 +0000] "GET /missing HTTP/1.1" 404 12 "-" "Mozilla/5.0"`
};

// Run the Actor and wait for it to finish
const run = await client.actor("checksmithcats/crawler-traffic-log-report").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "logText": """203.0.113.10 - - [07/Jul/2026:10:00:00 +0000] \"GET /docs HTTP/1.1\" 200 123 \"-\" \"GPTBot/1.0\"
198.51.100.2 - - [07/Jul/2026:10:00:02 +0000] \"GET /missing HTTP/1.1\" 404 12 \"-\" \"Mozilla/5.0\"""" }

# Run the Actor and wait for it to finish
run = client.actor("checksmithcats/crawler-traffic-log-report").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "logText": "203.0.113.10 - - [07/Jul/2026:10:00:00 +0000] \\"GET /docs HTTP/1.1\\" 200 123 \\"-\\" \\"GPTBot/1.0\\"\\n198.51.100.2 - - [07/Jul/2026:10:00:02 +0000] \\"GET /missing HTTP/1.1\\" 404 12 \\"-\\" \\"Mozilla/5.0\\""
}' |
apify call checksmithcats/crawler-traffic-log-report --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=checksmithcats/crawler-traffic-log-report",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/WdqLhmorpEwKNlhJY/builds/Di6fXH2A6IyN1wOOe/openapi.json