AI Agent Traffic Detector

Spot AI crawlers, answer-engine bots, and suspicious user agents before they quietly consume your content.

Who Uses It

Security teams, SEO teams, publishers, SaaS operators, and growth teams reviewing access logs.

Why It Is Useful

This actor does not save a generic web-scrape dump. It saves a compact dataset shaped around one business job, so the output is easier to review, export, and plug into research workflows.

Input

• startUrls - Public pages to inspect.
• logLines - Pasted snippets for fast tests or sources that do not need crawling.
• maxPagesPerStartUrl - Same-domain pages to inspect from each URL.
• maxResults - Maximum clean rows to save, capped at 20.
• proxyConfiguration - Optional Apify proxy settings.

Output Fields

• crawlerName - Detected crawler or bot family from the user-agent or page text.
• crawlerFamily - Useful classification such as AI crawler or generic crawler.
• userAgentPattern - The user-agent pattern buyers can compare against logs.
• requestedPath - The path the crawler requested, useful for seeing what content was targeted.
• seenAt - Date found in the log or page text.
• botConfidence - A simple confidence score based on matched crawler signals.
• recommendedBlockRule - Practical next step for robots.txt or traffic review.
• trafficEvidence - Where the signal came from.

Example Output

{
    "crawlerName": "GPTBot",
    "crawlerFamily": "AI crawler",
    "userAgentPattern": "Mozilla/5.0 AppleWebKit GPTBot/1.2",
    "requestedPath": "/pricing",
    "seenAt": "2026-07-10",
    "botConfidence": "85",
    "recommendedBlockRule": "Review robots.txt AI crawler rules",
    "trafficEvidence": "pasted access log"
}

Use Cases

• Audit AI crawler traffic before changing robots.txt.
• Create a short list of suspicious user agents for review.
• Compare crawl activity around pricing, docs, blog, and product pages.

Limits

The actor works best with public pages or focused pasted snippets. Login walls, CAPTCHA, heavy anti-bot protection, and very noisy pages can reduce result quality.

Pricing

Recommended pricing is paid per saved dataset item, so users pay for useful rows instead of empty page visits.