AI Bot Analytics & AI Crawler Checker (GPTBot, llms.txt)

Google Analytics for AI agents. Find out which AI bots visit your website, how often, and which pages they want: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Meta-ExternalAgent, Bytespider, CCBot, Amazonbot and 25+ more. The Actor reads your server or CDN access logs and flags fake bots by checking each hit against the official IP lists published by OpenAI, Perplexity, Google, Bing, Apple and others.

It also runs an AI-readiness audit on any website. You get the robots.txt verdict for 37 AI and search bots, a live test for firewall/CDN rules that block AI bots, plus llms.txt / llms-full.txt validation, sitemap, schema.org structured data, markdown support, text visible without JavaScript, and an AI-readiness score (0-100) with a prioritized list of fixes.

What you can do with it

• AI bot traffic analytics: hits per AI bot and per company (OpenAI, Anthropic, Perplexity, Google, Meta...). Each hit is classed as training, AI search or user-triggered assistant fetch, with daily trend, status codes and bandwidth.
• "What do AI agents want?": the pages AI bots request most, with a per-bot breakdown.
• Fake bot detection: hits that claim to be GPTBot, PerplexityBot, Googlebot and similar bots but come from IPs outside the operator's official ranges.
• Unknown bots: the most active user agents that look like bots but are not in the catalog yet.
• AI crawler checker: whether robots.txt allows or blocks each AI bot on your homepage and on the exact page, and which rule matched (RFC 9309 matching with wildcards and Allow/Disallow precedence). Reads Cloudflare Content Signals (search, ai-input, ai-train).
• Firewall / CDN check: catches Cloudflare "Block AI bots", WAF rules and 403s that silently stop AI search bots even when robots.txt allows them.
• llms.txt checker: checks presence, H1 title, summary, sections, link count and llmstxt.org format problems, and also detects llms-full.txt.
• AI SEO / GEO audit: sitemap, JSON-LD types, meta robots noindex/noai, X-Robots-Tag, word count without JavaScript, JS app-shell detection, and markdown served via Accept: text/markdown.

How to use

Audit websites (no setup):

1. Add domains or page URLs to Websites to audit, e.g. example.com or https://example.com/pricing.
2. Click Start. Each site returns one result with a score, a grade, the bots that are blocked, and the issues to fix.

Analyze your AI bot traffic:

1. Export your access logs. Examples: Nginx/Apache access.log, Cloudflare Logpush (JSON), Vercel log drains, CloudFront standard logs, IIS logs. .gz files are fine.
2. Put direct download links in Access log file URLs (signed S3/GCS/R2 URLs work), or paste lines into Or paste log lines.
3. Click Start. You get one overview row, one row per bot and the top pages AI bots requested.

With an empty input, the Actor audits two example sites (stripe.com and nytimes.com) so you can see the output.

Input example

{
  "startUrls": ["example.com", "https://example.com/blog/"],
  "probeBotAccess": true,
  "logUrls": ["https://my-bucket.s3.amazonaws.com/logs/access.log.gz?X-Amz-Signature=..."],
  "verifyBotIps": true,
  "maxPathRows": 25
}

Output

Every row has a type: siteAudit, logOverview, botReport or pathReport. The Output tab has three views: Site audits, AI bot traffic and Pages AI bots want.

Site audit (shortened):

{
  "type": "siteAudit",
  "domain": "nytimes.com",
  "aiReadinessScore": 48,
  "grade": "D",
  "aiPolicySummary": "Partly blocks AI training bots; partly blocks AI search & assistant bots",
  "blockedBots": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "PerplexityBot", "Google-Extended", "CCBot"],
  "firewallBlockedBots": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "PerplexityBot"],
  "issues": [
    { "severity": "high", "issue": "robots.txt blocks AI search / assistant bots that can cite you: OAI-SearchBot, ChatGPT-User, ...", "fix": "Allow them explicitly ..." },
    { "severity": "medium", "issue": "No /llms.txt file", "fix": "Publish /llms.txt ..." }
  ],
  "robotsTxt": { "state": "found", "status": 200, "sitemaps": ["https://www.nytimes.com/sitemaps/new/news.xml.gz"], "contentSignals": null },
  "llmsTxt": { "found": false, "status": 404 },
  "sitemap": { "found": true, "type": "index", "entries": 767 },
  "servesMarkdown": false,
  "page": { "title": "The New York Times - Breaking News...", "wordCount": 4033, "jsonLdTypes": ["NewsMediaOrganization", "WebSite"], "noindex": false, "noai": false },
  "bots": [
    { "bot": "GPTBot", "operator": "OpenAI", "purpose": "training", "allowedOnHomepage": false, "matchedRule": "Disallow: /", "probeHttpStatus": 403, "probeBlocked": true }
  ]
}

Bot report (from logs):

{
  "type": "botReport",
  "bot": "GPTBot", "operator": "OpenAI", "purposeLabel": "AI training crawler",
  "hits": 18452, "sharePctOfAiBotHits": 41.3,
  "verifiedHits": 18210, "suspectedSpoofedHits": 242,
  "uniquePaths": 3120, "topPaths": [{ "path": "/docs/api", "hits": 812 }],
  "statusCodes": { "200": 17001, "404": 1210, "301": 241 }, "successRatePct": 93.4,
  "firstSeen": "2026-10-01T00:03:11.000Z", "lastSeen": "2026-10-23T23:58:40.000Z",
  "hitsByDay": { "2026-10-22": 790, "2026-10-23": 845 }
}

The overview row adds total requests, AI share of traffic, hits by purpose and by company, suspected fake-bot requests, the top AI-requested pages and unclassified bots. Each page row gives a path with its AI bot hits split by bot.

How the AI-readiness score works (0-100)

Check	Points
robots.txt allows the bots that can cite you (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Googlebot, Bingbot)	30
Firewall/CDN does not block AI bot user agents (live test)	15
Valid llms.txt	15
XML sitemap	10
Structured data (JSON-LD / microdata)	10
Content readable without JavaScript (150+ words in raw HTML)	10
No noindex / noai	5
Title and meta description	5

Blocking training-only bots (GPTBot, ClaudeBot, Google-Extended, CCBot...) is a legitimate choice and does not lower the score. It shows up in trainingBotsBlocked and in the policy summary.

How much does it cost?

Pay per event: $0.005 per run + $0.003 per result. A result is one website audit or one report row. No compute charges.

You get	You pay (Starter/Bronze plan)
10 website audits	$0.035
100 website audits	$0.305
Log report with 15 bots + 25 top pages (41 rows)	$0.128

Apify's free plan includes $5 of usage every month, and free-plan users pay half price. Paid plans get volume discounts (Silver -10%, Gold and above -20%). Log size does not change the price: a 5 GB log costs the same as a small one.

Use cases

• SEO and GEO / AEO agencies: audit client sites for AI search visibility and send the fix list.
• Publishers: measure how much AI companies crawl you and which content they take, for licensing talks or bot policy.
• SaaS and docs teams: confirm ChatGPT, Claude and Perplexity can read your docs and pricing pages, and check llms.txt.
• Security and DevOps: spot fake AI bots and noisy crawlers, and confirm the CDN rules work as intended.
• Monitoring: schedule it weekly on your logs or your competitors' sites and watch the trend.

Run it from code (API, Python, JavaScript)

curl -X POST "https://api.apify.com/v2/acts/api-co~ai-bot-analytics/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" -d '{"startUrls": ["example.com"]}'

from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("api-co/ai-bot-analytics").call(run_input={"logUrls": ["https://.../access.log.gz"]})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["type"] == "botReport":
        print(row["bot"], row["hits"], row["suspectedSpoofedHits"])

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('api-co/ai-bot-analytics').call({ startUrls: ['example.com', 'competitor.com'] });
const { items } = await client.dataset(run.defaultDatasetId).listItems();

FAQ

Which log formats are supported? Formats are detected per line: Apache/Nginx combined and common, JSON lines (Cloudflare Logpush ClientRequestUserAgent, Vercel proxy.userAgent, Caddy, Fastly, Netlify and generic ip/user_agent/path/status/time keys), and W3C extended logs with a #Fields: header (AWS CloudFront, IIS). Gzip is detected automatically. If fewer than 20% of lines can be parsed, you get an error row that says so.

Are my logs stored or shared? Logs are streamed and only aggregated counts are saved to your own dataset. Raw lines and IPs are not stored.

How is a "fake" bot detected? OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Perplexity, Google, Bing, Apple, Common Crawl, DuckDuckGo and Mistral publish the IP ranges their bots use. A hit whose user agent says GPTBot but whose IP is outside OpenAI's list is counted as suspectedSpoofedHits. Bots without a published list (for example ClaudeBot) show null. If your logs record a proxy/CDN IP instead of the visitor IP, use the CDN's own logs (Cloudflare ClientIP) for accurate results.

How does the live firewall test work, and is it exact? The page is requested with each AI bot's user agent and the answer is compared with a normal browser request. A 403, 429 or 503 status, or a challenge page, means your server, CDN or WAF rejects that user agent. Real bots come from their own verified IPs, and some CDNs treat verified bots differently, so read this as a strong signal rather than a guarantee. robots.txt results are exact.

Why is blocking a bot in robots.txt not always enough? robots.txt is voluntary. To enforce a policy, use your CDN's bot rules and check the result with the log report.

Why are Google-Extended and Applebot-Extended not in my logs? They are robots.txt tokens only. Google and Apple crawl with Googlebot and Applebot and use the token to decide whether your content can train their AI models.

Can it crawl a whole site? It audits the URLs you give it, and robots.txt, llms.txt and the sitemap are site-wide. To check many pages, list them in Websites to audit.

Related Actors

• SEO Site Auditor: technical SEO crawl of a whole site.
• Website to Markdown: turn pages into clean markdown for LLMs.
• Tech Stack Detector: see which CDN and frameworks a site uses.