# AI Bot Analytics & AI Crawler Checker (GPTBot, llms.txt) (`api-co/ai-bot-analytics`) Actor

Google Analytics for AI agents: see which AI bots (GPTBot, ClaudeBot, PerplexityBot, ChatGPT-User...) hit your site and what pages they want, from your access logs, with fake-bot detection. Plus an AI-readiness audit: robots.txt per AI bot, firewall blocks, llms.txt, sitemap, schema, score.

- **URL**: https://apify.com/api-co/ai-bot-analytics.md
- **Developed by:** [Emilio Cisneros](https://apify.com/api-co) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.40 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Bot Analytics & AI Crawler Checker (GPTBot, llms.txt)

**Google Analytics for AI agents.** Find out which AI bots visit your website, how often, and which pages they want: **GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, Meta-ExternalAgent, Bytespider, CCBot, Amazonbot** and 25+ more. The Actor reads your **server or CDN access logs** and flags **fake bots** by checking each hit against the official IP lists published by OpenAI, Perplexity, Google, Bing, Apple and others.

It also runs an **AI-readiness audit** on any website. You get the **robots.txt** verdict for 37 AI and search bots, a **live test** for firewall/CDN rules that block AI bots, plus **llms.txt / llms-full.txt** validation, sitemap, **schema.org** structured data, markdown support, text visible without JavaScript, and an **AI-readiness score (0-100)** with a prioritized list of fixes.

### What you can do with it

- **AI bot traffic analytics**: hits per AI bot and per company (OpenAI, Anthropic, Perplexity, Google, Meta...). Each hit is classed as training, AI search or user-triggered assistant fetch, with daily trend, status codes and bandwidth.
- **"What do AI agents want?"**: the pages AI bots request most, with a per-bot breakdown.
- **Fake bot detection**: hits that claim to be GPTBot, PerplexityBot, Googlebot and similar bots but come from IPs outside the operator's official ranges.
- **Unknown bots**: the most active user agents that look like bots but are not in the catalog yet.
- **AI crawler checker**: whether robots.txt allows or blocks each AI bot on your homepage and on the exact page, and which rule matched (RFC 9309 matching with wildcards and Allow/Disallow precedence). Reads Cloudflare **Content Signals** (`search`, `ai-input`, `ai-train`).
- **Firewall / CDN check**: catches Cloudflare "Block AI bots", WAF rules and 403s that silently stop AI search bots even when robots.txt allows them.
- **llms.txt checker**: checks presence, H1 title, summary, sections, link count and llmstxt.org format problems, and also detects llms-full.txt.
- **AI SEO / GEO audit**: sitemap, JSON-LD types, meta robots `noindex`/`noai`, X-Robots-Tag, word count without JavaScript, JS app-shell detection, and markdown served via `Accept: text/markdown`.

### How to use

**Audit websites (no setup):**

1. Add domains or page URLs to **Websites to audit**, e.g. `example.com` or `https://example.com/pricing`.
2. Click **Start**. Each site returns one result with a score, a grade, the bots that are blocked, and the issues to fix.

**Analyze your AI bot traffic:**

1. Export your access logs. Examples: Nginx/Apache `access.log`, Cloudflare Logpush (JSON), Vercel log drains, CloudFront standard logs, IIS logs. `.gz` files are fine.
2. Put direct download links in **Access log file URLs** (signed S3/GCS/R2 URLs work), or paste lines into **Or paste log lines**.
3. Click **Start**. You get one overview row, one row per bot and the top pages AI bots requested.

With an empty input, the Actor audits two example sites (stripe.com and nytimes.com) so you can see the output.

### Input example

```json
{
  "startUrls": ["example.com", "https://example.com/blog/"],
  "probeBotAccess": true,
  "logUrls": ["https://my-bucket.s3.amazonaws.com/logs/access.log.gz?X-Amz-Signature=..."],
  "verifyBotIps": true,
  "maxPathRows": 25
}
```

### Output

Every row has a `type`: `siteAudit`, `logOverview`, `botReport` or `pathReport`. The Output tab has three views: **Site audits**, **AI bot traffic** and **Pages AI bots want**.

**Site audit** (shortened):

```json
{
  "type": "siteAudit",
  "domain": "nytimes.com",
  "aiReadinessScore": 48,
  "grade": "D",
  "aiPolicySummary": "Partly blocks AI training bots; partly blocks AI search & assistant bots",
  "blockedBots": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "PerplexityBot", "Google-Extended", "CCBot"],
  "firewallBlockedBots": ["GPTBot", "OAI-SearchBot", "ChatGPT-User", "ClaudeBot", "PerplexityBot"],
  "issues": [
    { "severity": "high", "issue": "robots.txt blocks AI search / assistant bots that can cite you: OAI-SearchBot, ChatGPT-User, ...", "fix": "Allow them explicitly ..." },
    { "severity": "medium", "issue": "No /llms.txt file", "fix": "Publish /llms.txt ..." }
  ],
  "robotsTxt": { "state": "found", "status": 200, "sitemaps": ["https://www.nytimes.com/sitemaps/new/news.xml.gz"], "contentSignals": null },
  "llmsTxt": { "found": false, "status": 404 },
  "sitemap": { "found": true, "type": "index", "entries": 767 },
  "servesMarkdown": false,
  "page": { "title": "The New York Times - Breaking News...", "wordCount": 4033, "jsonLdTypes": ["NewsMediaOrganization", "WebSite"], "noindex": false, "noai": false },
  "bots": [
    { "bot": "GPTBot", "operator": "OpenAI", "purpose": "training", "allowedOnHomepage": false, "matchedRule": "Disallow: /", "probeHttpStatus": 403, "probeBlocked": true }
  ]
}
```

**Bot report** (from logs):

```json
{
  "type": "botReport",
  "bot": "GPTBot", "operator": "OpenAI", "purposeLabel": "AI training crawler",
  "hits": 18452, "sharePctOfAiBotHits": 41.3,
  "verifiedHits": 18210, "suspectedSpoofedHits": 242,
  "uniquePaths": 3120, "topPaths": [{ "path": "/docs/api", "hits": 812 }],
  "statusCodes": { "200": 17001, "404": 1210, "301": 241 }, "successRatePct": 93.4,
  "firstSeen": "2026-10-01T00:03:11.000Z", "lastSeen": "2026-10-23T23:58:40.000Z",
  "hitsByDay": { "2026-10-22": 790, "2026-10-23": 845 }
}
```

The **overview** row adds total requests, AI share of traffic, hits by purpose and by company, suspected fake-bot requests, the top AI-requested pages and unclassified bots. Each **page** row gives a path with its AI bot hits split by bot.

### How the AI-readiness score works (0-100)

| Check | Points |
|---|---|
| robots.txt allows the bots that can cite you (OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User, Googlebot, Bingbot) | 30 |
| Firewall/CDN does not block AI bot user agents (live test) | 15 |
| Valid llms.txt | 15 |
| XML sitemap | 10 |
| Structured data (JSON-LD / microdata) | 10 |
| Content readable without JavaScript (150+ words in raw HTML) | 10 |
| No noindex / noai | 5 |
| Title and meta description | 5 |

Blocking **training-only** bots (GPTBot, ClaudeBot, Google-Extended, CCBot...) is a legitimate choice and **does not lower the score**. It shows up in `trainingBotsBlocked` and in the policy summary.

### How much does it cost?

Pay per event: **$0.005 per run + $0.003 per result**. A result is one website audit or one report row. No compute charges.

| You get | You pay (Starter/Bronze plan) |
|---|---|
| 10 website audits | $0.035 |
| 100 website audits | $0.305 |
| Log report with 15 bots + 25 top pages (41 rows) | $0.128 |

Apify's **free plan includes $5 of usage every month**, and free-plan users pay **half price**. Paid plans get volume discounts (Silver -10%, Gold and above -20%). Log size does not change the price: a 5 GB log costs the same as a small one.

### Use cases

- **SEO and GEO / AEO agencies**: audit client sites for AI search visibility and send the fix list.
- **Publishers**: measure how much AI companies crawl you and which content they take, for licensing talks or bot policy.
- **SaaS and docs teams**: confirm ChatGPT, Claude and Perplexity can read your docs and pricing pages, and check llms.txt.
- **Security and DevOps**: spot fake AI bots and noisy crawlers, and confirm the CDN rules work as intended.
- **Monitoring**: schedule it weekly on your logs or your competitors' sites and watch the trend.

### Run it from code (API, Python, JavaScript)

```bash
curl -X POST "https://api.apify.com/v2/acts/api-co~ai-bot-analytics/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" -d '{"startUrls": ["example.com"]}'
```

```python
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("api-co/ai-bot-analytics").call(run_input={"logUrls": ["https://.../access.log.gz"]})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    if row["type"] == "botReport":
        print(row["bot"], row["hits"], row["suspectedSpoofedHits"])
```

```javascript
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('api-co/ai-bot-analytics').call({ startUrls: ['example.com', 'competitor.com'] });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

### FAQ

**Which log formats are supported?**
Formats are detected per line: Apache/Nginx *combined* and *common*, JSON lines (Cloudflare Logpush `ClientRequestUserAgent`, Vercel `proxy.userAgent`, Caddy, Fastly, Netlify and generic `ip`/`user_agent`/`path`/`status`/`time` keys), and W3C extended logs with a `#Fields:` header (AWS CloudFront, IIS). Gzip is detected automatically. If fewer than 20% of lines can be parsed, you get an error row that says so.

**Are my logs stored or shared?**
Logs are streamed and only aggregated counts are saved to your own dataset. Raw lines and IPs are not stored.

**How is a "fake" bot detected?**
OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User), Perplexity, Google, Bing, Apple, Common Crawl, DuckDuckGo and Mistral publish the IP ranges their bots use. A hit whose user agent says GPTBot but whose IP is outside OpenAI's list is counted as `suspectedSpoofedHits`. Bots without a published list (for example ClaudeBot) show `null`. If your logs record a proxy/CDN IP instead of the visitor IP, use the CDN's own logs (Cloudflare `ClientIP`) for accurate results.

**How does the live firewall test work, and is it exact?**
The page is requested with each AI bot's user agent and the answer is compared with a normal browser request. A 403, 429 or 503 status, or a challenge page, means your server, CDN or WAF rejects that user agent. Real bots come from their own verified IPs, and some CDNs treat verified bots differently, so read this as a strong signal rather than a guarantee. robots.txt results are exact.

**Why is blocking a bot in robots.txt not always enough?**
robots.txt is voluntary. To enforce a policy, use your CDN's bot rules and check the result with the log report.

**Why are Google-Extended and Applebot-Extended not in my logs?**
They are robots.txt tokens only. Google and Apple crawl with Googlebot and Applebot and use the token to decide whether your content can train their AI models.

**Can it crawl a whole site?**
It audits the URLs you give it, and robots.txt, llms.txt and the sitemap are site-wide. To check many pages, list them in **Websites to audit**.

### Related Actors

- [SEO Site Auditor](https://apify.com/api-co/seo-site-auditor): technical SEO crawl of a whole site.
- [Website to Markdown](https://apify.com/api-co/website-to-markdown): turn pages into clean markdown for LLMs.
- [Tech Stack Detector](https://apify.com/api-co/tech-stack-detector): see which CDN and frameworks a site uses.

# Actor input Schema

## `startUrls` (type: `array`):

Domains or page URLs, e.g. 'example.com' or 'https://example.com/pricing'. Each gets an AI-readiness audit: robots.txt rules for 37 AI and search bots, a live firewall test, llms.txt, sitemap, structured data and text visible without JavaScript. Leave empty (and add no logs) to audit two example sites.

## `probeBotAccess` (type: `boolean`):

Requests each page with the GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot and PerplexityBot user agents and compares the answer with a normal browser. Catches Cloudflare 'Block AI bots', WAF rules and 403s that robots.txt does not show.

## `logUrls` (type: `array`):

Direct links to server/CDN access logs (plain or .gz): a signed S3/GCS/R2 URL, a Dropbox/Drive direct link, or an Apify key-value store record URL. Formats are detected automatically: Apache/Nginx combined or common, JSON lines (Cloudflare Logpush, Vercel, Caddy, Fastly, Netlify and generic keys) and W3C logs with a #Fields: header (CloudFront, IIS).

## `logText` (type: `string`):

Paste access-log lines directly (good for a quick look; use log URLs for big files).

## `verifyBotIps` (type: `boolean`):

Checks each hit against the IP ranges published by OpenAI, Perplexity, Google, Bing, Apple, Common Crawl, DuckDuckGo and Mistral, and counts hits from other IPs as suspected fake bots.

## `includeSearchEngines` (type: `boolean`):

Add Googlebot and Bingbot to the per-bot report for comparison (they are never counted as AI bot traffic).

## `maxPathRows` (type: `integer`):

How many 'most-requested by AI bots' page rows to output.

## `maxLogLines` (type: `integer`):

Safety limit on lines read across all log files.

## `maxConcurrency` (type: `integer`):

How many websites to audit at the same time.

## `maxItems` (type: `integer`):

Stop after this many result rows (site audits + report rows). You are only charged per row.

## Actor input object example

```json
{
  "startUrls": [
    "https://stripe.com"
  ],
  "probeBotAccess": true,
  "verifyBotIps": true,
  "includeSearchEngines": true,
  "maxPathRows": 25,
  "maxLogLines": 20000000,
  "maxConcurrency": 4,
  "maxItems": 1000
}
```

# Actor output Schema

## `results` (type: `string`):

Site audits and AI bot traffic reports.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://stripe.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("api-co/ai-bot-analytics").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": ["https://stripe.com"] }

# Run the Actor and wait for it to finish
run = client.actor("api-co/ai-bot-analytics").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://stripe.com"
  ]
}' |
apify call api-co/ai-bot-analytics --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,api-co/ai-bot-analytics"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/drZoCbRfwSzceScy0/builds/ZujzCnzEMhdxabKkg/openapi.json