# AI Agent Traffic Detector (`seeb/ai-agent-traffic-detector`) Actor Detect AI crawler and bot traffic from access-log lines or public bot policy pages. - **URL**: https://apify.com/seeb/ai-agent-traffic-detector.md - **Developed by:** [Techionik](https://apify.com/seeb) (community) - **Categories:** Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $18.00 / 1,000 dataset items This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## AI Agent Traffic Detector Spot AI crawlers, answer-engine bots, and suspicious user agents before they quietly consume your content. ### Who Uses It Security teams, SEO teams, publishers, SaaS operators, and growth teams reviewing access logs. ### Why It Is Useful This actor does not save a generic web-scrape dump. It saves a compact dataset shaped around one business job, so the output is easier to review, export, and plug into research workflows. ### Input - `startUrls` - Public pages to inspect. - `logLines` - Pasted snippets for fast tests or sources that do not need crawling. - `maxPagesPerStartUrl` - Same-domain pages to inspect from each URL. - `maxResults` - Maximum clean rows to save, capped at 20. - `proxyConfiguration` - Optional Apify proxy settings. ### Output Fields - `crawlerName` - Detected crawler or bot family from the user-agent or page text. - `crawlerFamily` - Useful classification such as AI crawler or generic crawler. - `userAgentPattern` - The user-agent pattern buyers can compare against logs. - `requestedPath` - The path the crawler requested, useful for seeing what content was targeted. - `seenAt` - Date found in the log or page text. - `botConfidence` - A simple confidence score based on matched crawler signals. - `recommendedBlockRule` - Practical next step for robots.txt or traffic review. - `trafficEvidence` - Where the signal came from. ### Example Output ```json { "crawlerName": "GPTBot", "crawlerFamily": "AI crawler", "userAgentPattern": "Mozilla/5.0 AppleWebKit GPTBot/1.2", "requestedPath": "/pricing", "seenAt": "2026-07-10", "botConfidence": "85", "recommendedBlockRule": "Review robots.txt AI crawler rules", "trafficEvidence": "pasted access log" } ```` ### Use Cases - Audit AI crawler traffic before changing robots.txt. - Create a short list of suspicious user agents for review. - Compare crawl activity around pricing, docs, blog, and product pages. ### Limits The actor works best with public pages or focused pasted snippets. Login walls, CAPTCHA, heavy anti-bot protection, and very noisy pages can reduce result quality. ### Pricing Recommended pricing is paid per saved dataset item, so users pay for useful rows instead of empty page visits. # Actor input Schema ## `startUrls` (type: `array`): Paste access-log or user-agent lines, or add public crawler policy pages. ## `logLines` (type: `array`): Paste access-log or user-agent lines, or add public crawler policy pages. ## `maxPagesPerStartUrl` (type: `integer`): How many same-domain pages to inspect from each start URL. ## `maxResults` (type: `integer`): Maximum clean rows to save. Capped at 20 to keep outputs focused. ## `proxyConfiguration` (type: `object`): Optional Apify proxy settings. Most public pages work without proxy. ## Actor input object example ```json { "startUrls": [ { "url": "https://platform.openai.com/docs/bots" }, { "url": "https://developers.cloudflare.com/bots/" } ], "logLines": [ "2026-07-10 10:24:18 GET /pricing 200 \"Mozilla/5.0 AppleWebKit GPTBot/1.2\"", "2026-07-10 10:26:11 GET /docs 200 \"ClaudeBot/1.0\"", "2026-07-10 10:29:04 GET /blog 200 \"PerplexityBot/1.0\"" ], "maxPagesPerStartUrl": 1, "maxResults": 20, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `results` (type: `string`): No description ## `runSummary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://platform.openai.com/docs/bots" }, { "url": "https://developers.cloudflare.com/bots/" } ] }; // Run the Actor and wait for it to finish const run = await client.actor("seeb/ai-agent-traffic-detector").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [ { "url": "https://platform.openai.com/docs/bots" }, { "url": "https://developers.cloudflare.com/bots/" }, ] } # Run the Actor and wait for it to finish run = client.actor("seeb/ai-agent-traffic-detector").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://platform.openai.com/docs/bots" }, { "url": "https://developers.cloudflare.com/bots/" } ] }' | apify call seeb/ai-agent-traffic-detector --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=seeb/ai-agent-traffic-detector", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "AI Agent Traffic Detector", "description": "Detect AI crawler and bot traffic from access-log lines or public bot policy pages.", "version": "1.0", "x-build-id": "jJwMbjJftGSh8gv26" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/seeb~ai-agent-traffic-detector/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-seeb-ai-agent-traffic-detector", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/seeb~ai-agent-traffic-detector/runs": { "post": { "operationId": "runs-sync-seeb-ai-agent-traffic-detector", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/seeb~ai-agent-traffic-detector/run-sync": { "post": { "operationId": "run-sync-seeb-ai-agent-traffic-detector", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "Paste access-log or user-agent lines, or add public crawler policy pages.", "items": { "type": "object", "properties": { "url": { "title": "URL", "type": "string", "description": "Public page URL." } }, "required": [ "url" ] } }, "logLines": { "title": "Log lines", "type": "array", "description": "Paste access-log or user-agent lines, or add public crawler policy pages.", "items": { "type": "string" }, "default": [ "2026-07-10 10:24:18 GET /pricing 200 \"Mozilla/5.0 AppleWebKit GPTBot/1.2\"", "2026-07-10 10:26:11 GET /docs 200 \"ClaudeBot/1.0\"", "2026-07-10 10:29:04 GET /blog 200 \"PerplexityBot/1.0\"" ] }, "maxPagesPerStartUrl": { "title": "Pages per URL", "minimum": 1, "maximum": 3, "type": "integer", "description": "How many same-domain pages to inspect from each start URL.", "default": 1 }, "maxResults": { "title": "Max results", "minimum": 1, "maximum": 20, "type": "integer", "description": "Maximum clean rows to save. Capped at 20 to keep outputs focused.", "default": 20 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Optional Apify proxy settings. Most public pages work without proxy.", "default": { "useApifyProxy": false } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```