# AI Crawler Checker: GPTBot, robots.txt & llms.txt Policy Audit (`adastra-connect/ai-crawler-policy-auditor`) Actor

Check which AI crawlers each website blocks in robots.txt (GPTBot, ClaudeBot, Google-Extended, PerplexityBot), plus llms.txt and TDM opt-out signals.

- **URL**: https://apify.com/adastra-connect/ai-crawler-policy-auditor.md
- **Developed by:** [Ad Astra Connect](https://apify.com/adastra-connect) (community)
- **Categories:** SEO tools, AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 domains

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler Checker: does a website block GPTBot, ClaudeBot or Google-Extended?

**Check, for a list of websites, which AI crawlers each one blocks in robots.txt, and whether it publishes llms.txt or text-and-data-mining (TDM) opt-out signals.** Enter domains and get one row per domain with a verdict for 30 user agents, 29 AI crawlers plus Googlebot as a search reference (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot and more), plus llms.txt, TDMRep, `noai` meta tags and ai.txt. Schedule it in monitoring mode and each run returns only the policy changes. These are observed machine-readable signals for review, not legal advice or a rights determination.

### Use cases

- **AI dataset and RAG teams:** document which sources opt out of AI training before you crawl, with a dated snapshot per domain.
- **Publishers and media groups:** compare your robots.txt AI crawler policy with competitors and peers.
- **SEO and GEO agencies:** audit client sites for accidental blocks of AI search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) that can keep them out of AI answers.
- **Brand and legal ops:** track when a site starts or stops blocking AI crawlers, with monitor mode and a schedule.

### Sample output: AI crawler robots.txt audit

Real rows from a run on October 22, 2026, shortened:

| domain | robotsTxt.found | blocksAnyAiBot | blocksAllTrainingBots | tdmOptOutSignal | llmsTxt.found | GPTBot verdict |
| --- | --- | --- | --- | --- | --- | --- |
| example.com | false | false | false | false | false | allowed (no robots.txt) |
| www.w3.org | true | true | false | true | false | partial |

Each row also has `bots` (token, operator, purpose, verdict for `/` and your test path), `blockedBots`, `allowedBots`, `aiPolicyScore`, `tdmrep`, `metaRobots`, `xRobotsTag`, `aiTxt`, `resources`, `errors` and `checkedAt`.

### What data can you get: robots.txt, llms.txt and TDM signals

| Fields | Data |
| --- | --- |
| `input`, `domain`, `finalUrl`, `checkedAt` | Normalized input host, host, final homepage URL (or robots URL if the homepage was skipped), and UTC observation time. |
| `robotsTxt` | HTTP status, file presence, bytes read, up to 20 sitemaps, and wildcard full-block flag. |
| `bots` | Token, operator, purpose, official documentation URL, named-group flag, verdict, effective verdict, and decisions for `/` and your optional path. |
| `blockedBots`, `allowedBots` | Tokens effectively fully blocked or unrestricted. Partially restricted tokens remain in `bots`. |
| `blocksAllTrainingBots`, `blocksAnyAiBot` | Selected training tokens all fully blocked, or any selected AI bot fully or partly restricted. Googlebot is a search reference and excluded from the second flag. |
| `aiPolicyScore` | An informational 0-100 published-policy score, explained below. |
| `llmsTxt`, `llmsFullTxt` | Text-file presence and bytes read; llms.txt also has a link count. The title is null to avoid retaining names. |
| `tdmrep` | Observed TDMRep entries, whether any parsed tdm-reservation value is 1, and first policy URL signal. Nonstandard location paths are fingerprinted. |
| `tdmReservationHeader`, `tdmPolicyHeader`, `tdmMeta` | Homepage TDM headers and meta declarations. |
| `xRobotsTag`, `metaRobots`, `noaiMeta` | Homepage robots headers, generic and bot-specific meta directives, and noai/noimageai detection. |
| `linkHints`, `aiTxt` | Relevant published link relations and Spawning-style ai.txt presence and full-disallow signal. |
| `tdmOptOutSignal` | Any selected training bot restriction, TDM reservation, or noai/noimageai meta signal. |
| `resources`, `homepage`, `errors`, `status` | Per-resource final URLs, final hosts, statuses, byte caps, skipped-homepage reason, and errors. |
| `rawRobotsTxt`, `botsAsOf` | Optional reconstructed robots directives capped at 50 KB; comments are omitted and nonstandard paths fingerprinted, plus registry date. |
| `changeType`, `field`, `previousValue`, `currentValue`, `detectedAt` | One changed field per row in monitoring mode. |

Use the dataset's **overview** and **changes** table views. Apify supports JSON, CSV, Microsoft Excel and other dataset exports. Failed or missing resources are visible in `resources` and `errors`; an unavailable policy is never represented as a confirmed allow decision.

### How to check which AI crawlers a website blocks

1. Enter bare domains or public HTTP/HTTPS URLs. Paths in domain inputs are ignored; hosts are deduplicated. Subdomains remain separate.
2. Optionally select bot tokens and a `testPath`. Unknown tokens are evaluated as custom bots with mixed purpose. The test path is evaluated locally and never fetched.
3. Set `maxResults` and run. Each normal output row is one billable `domain` event, including an unreachable-domain report.
4. Open the dataset and `RUN_SUMMARY` in the default key-value store for per-target statuses, counts and errors.
5. Enable monitoring and keep the same `monitorKey` for scheduled comparisons.

Empty API input `{}` uses the three domains below, a default maximum of 100 rows and concurrency 5. The input editor prefills `maxResults` with 10. Each host has one active request at a time and a default 250 ms minimum request interval. A larger published Crawl-delay for this auditor takes precedence.

```json
{
  "domains": ["www.nytimes.com", "www.theguardian.com", "en.wikipedia.org"],
  "testPath": "/news/",
  "bots": ["GPTBot", "ClaudeBot", "Google-Extended"],
  "includeRaw": false,
  "maxResults": 10,
  "concurrency": 5,
  "requestDelayMs": 250,
  "monitorMode": false
}
```

Example of a shortened domain row. Policy values vary by date and region:

```json
{
  "input": "publisher.example",
  "domain": "publisher.example",
  "checkedAt": "2026-10-21T12:00:00.000Z",
  "robotsTxt": {
    "status": 200,
    "found": true,
    "sizeBytes": 87,
    "sitemaps": [],
    "hasWildcardDisallowAll": false
  },
  "bots": [
    {"token": "GPTBot", "operator": "OpenAI", "purpose": "training", "verdict": "blocked", "effectiveVerdict": "blocked", "namedGroup": true, "rootAllowed": false, "testPathAllowed": null}
  ],
  "blockedBots": ["GPTBot"],
  "allowedBots": [],
  "blocksAllTrainingBots": true,
  "blocksAnyAiBot": true,
  "aiPolicyScore": 100,
  "llmsTxt": {"found": false, "sizeBytes": 0, "title": null, "linkCount": 0},
  "llmsFullTxt": {"found": false, "sizeBytes": 0},
  "tdmrep": {"found": true, "reservation": 1, "policyUrl": null, "entries": [{"location": "/", "tdm-reservation": 1, "tdm-policy": null}]},
  "tdmReservationHeader": null,
  "xRobotsTag": [],
  "metaRobots": [],
  "noaiMeta": false,
  "aiTxt": {"found": false, "disallowsAll": false},
  "tdmOptOutSignal": true,
  "errors": [],
  "status": "ok",
  "botsAsOf": "2026-10-21"
}
```

#### Observed robots signals and score

`allowed` means no matching disallow signal was found in the observed rules. `blocked` means the observed rules indicate a full disallow; `partial` means some tested paths have disallow signals. These technical labels describe parsed rules, not legal permission or actual crawler behavior. `no_rule` means no group names the token and the wildcard group, if present, is used. `effectiveVerdict` records the resulting allowed/blocked/partial policy even for wildcard fallback. If robots.txt cannot be interpreted, effective decisions are null. `rootAllowed` and `testPathAllowed` are explicit path decisions. Partial-policy classification tests rule-derived representative paths; unusual interacting wildcard patterns can need manual inspection.

Summary arrays and flags use effective decisions. The score is `round(80 * fullyBlockedTrainingTokens / selectedTrainingTokens + 20 * hasTdmReservation)`. If no training tokens are selected, the first term is zero and `blocksAllTrainingBots` is false. TDM reservation is 1 in any parsed TDMRep entry or a homepage TDM header/meta declaration. Multiple reservation surfaces still contribute only 20 points. Partial blocks and noai alone contribute no score points, but can set `tdmOptOutSignal`. Legacy training tokens are included in the registry and denominator. This score measures observed signals, not compliance, protection, crawler behavior, or permission to reuse content.

### Pricing

This Actor uses **pay per event** pricing: **$2.50 per 1,000 domain rows** returned (Apify Store discount tiers pay less). The price on the Store page is authoritative. Platform compute is included, so you pay for the rows you receive plus Apify's small per-run start fee, nothing else.

Normal mode bills one `domain` event per domain row. Monitoring bills one `domain` event per changed-field row. A silent first baseline and an unchanged monitor run emit no billable rows. The Actor honors the spending limit and `maxResults`, and writes batches of up to 100 rows.

### Monitor robots.txt AI crawler policy changes

Enable `monitorMode` and choose a stable `monitorKey`. State persists per input host in the named `ai-crawler-policy-auditor-state` key-value store. The first successful observation saves a baseline and emits nothing, unless `emitBaselineOnFirstRun` is enabled. Baseline events carry `baseline: true`.

Snapshots retain bot verdicts, robots availability, llms.txt presence and 1 KB size buckets, TDMRep entries and reservation, and homepage header/meta values. Monitoring emits `bot_blocked`, `bot_unblocked`, `llms_txt_added`, `llms_txt_removed`, `tdm_reservation_changed`, `meta_changed`, or `robots_unreachable`. Increasing/decreasing partial restrictions count as bot blocked/unblocked. llms.txt size-bucket changes and robots recovery use `meta_changed`; their `field` identifies what changed.

An unavailable resource keeps its previous values, so a transient error does not generate removal or unblock alerts. A robots outage emits its availability change. If a run stops partway through delivering a host's changes, its snapshot is not advanced and some delivered events can repeat next run. Use `domain`, `field`, values and timestamps for downstream deduplication. Bot selection, test path or registry date changes reset that host's comparison scope. Run one schedule at a time per namespace to avoid concurrent snapshot updates.

### Scheduling and integrations

Use Apify schedules for periodic audits, webhooks for run completion, or the Apify API for on-demand checks. Connect datasets to Make, Zapier or n8n workflows. AI agents can invoke the Actor through Apify MCP integrations. An unchanged monitor run produces an empty change feed; inspect `RUN_SUMMARY` to distinguish silence from failures.

### Is it legal?

The Actor reads public policy endpoints and the public homepage without login, cookies, browser automation or proxies. It identifies itself and respects homepage robots rules for its own user agent. A homepage disallowed by robots is skipped; the public machine-policy files are still fetched to audit published instructions. No pages beyond these resources and their redirects are crawled. The output omits source prose and llms.txt titles, retains only recognized header/meta directives, removes input paths and URL queries, and fingerprints nonstandard source paths. Optional raw output reconstructs directives without comments or unselected source bot tokens. Submit only organizational domains and bot product tokens, without personal information. Fingerprinted URL paths are comparison identifiers, not working page links.

This Actor provides observed machine-readable signals for review. It does not give legal advice, a compliance verdict, a rights-reservation determination, or a license. A value of 0, 1, absent, allowed or blocked describes only an observed signal or parser result. Consult qualified counsel for legal questions about your intended use, applicable law and site terms.

### Limitations

- A snapshot reports what this user agent received at one time. Regional, CDN, firewall and bot-specific responses can differ. HTTP 403 and other failures are reported without bypass attempts.
- Only the root policy files and homepage are inspected. testPath is not fetched. Bot-specific meta tags and TDM scopes are observations, not a complete site-wide rights decision.
- Requests have a 20-second maximum timeout, at most five redirects and a 1 MB body cap. The homepage is capped at 512 KB and llms-full.txt at 64 KB. `sizeBytes` is bytes read, including an error body if present; check `found`, status and `resources.*.truncated`. Counts describe only the read prefix. llms.txt titles are omitted.
- Localhost, literal IP addresses, internal hostnames and DNS answers in private, metadata or reserved address ranges are blocked, including redirects. Responses are streamed with byte caps and a 16 KB header cap. Robots matching avoids backtracking regular expressions; oversized policy rule sets are reported as errors.
- Each domain has a 25-second audit budget, including retries and delays, to keep the small prefill fast. Remaining observations can be incomplete and have errors. Runs stop gracefully 45 seconds before their platform timeout.
- HTML soft-404 detection checks HTML content types and the html tag. Some other soft-404 responses can evade it. Regex metadata extraction does not execute scripts or fully parse malformed HTML.
- robots.txt is voluntary; a bot's actual behavior can differ. User-request fetchers may not apply robots.txt. Google-Extended and similar tokens can govern use rather than a separate network crawler.
- llms.txt is a discovery convention and its presence grants no training permission. ai.txt conventions vary. This Actor recognizes full-disallow lines in the Spawning convention.
- The registry is dated 2026-10-21, includes historical tokens, and does not cover every AI system. Custom tokens use mixed purpose and do not affect the training denominator.
- If every target is unreachable, invalid or has no usable observations, the run fails after saving its summary. Individual target failures do not fail otherwise useful runs. Missing policy files are valid observations.

### FAQ

**Does `no_rule` mean a bot is allowed?**
Read `effectiveVerdict`. A wildcard policy can block a bot that has no named group.

**Does a high score determine rights or permissions?**
No. The score summarizes selected published opt-out signals. Interpret scopes and errors and obtain any necessary rights separately.

**Why did a monitoring run produce zero rows?**
It may have saved its first baseline or found no changes. Check the summary for resource failures, which do not clear the last known policy.

**Can I test a URL path or add another bot?**
Use `testPath` and add its product token to `bots`. No request is sent to the test path.

### Trademarks and non-affiliation

Ad Astra Connect is the publisher. Apify, Microsoft Excel, OpenAI, GPTBot, ChatGPT, OAI-SearchBot, Anthropic, Claude, ClaudeBot, Google, Googlebot, Google-Extended, Spawning, Make, Zapier, n8n, The New York Times, The Guardian and Wikipedia are trademarks of their respective owners. This Actor is not affiliated with, endorsed or sponsored by those brands or their owners. Bot names identify the policies being checked. All other operator and bot brands appearing in generated registry fields belong to their respective owners; the same non-affiliation disclaimer applies to every listed brand.

### Support

Use the Actor's Issues tab on Apify. Include the run ID, a non-sensitive input example and relevant error fields.

# Actor input Schema

## `domains` (type: `array`):

Organizational public domains or URLs without personal data. Input paths are omitted. Private/internal destinations and literal IPs are blocked.

## `testPath` (type: `string`):

Evaluate this path in addition to /. This path is never fetched. Example: /news/.

## `bots` (type: `array`):

Leave empty for all registered tokens. Unknown tokens are evaluated as custom bots with mixed purpose.

## `includeRaw` (type: `boolean`):

Include up to 50 KB of reconstructed robots directives. Comments are omitted; nonstandard paths are fingerprinted.

## `maxResults` (type: `integer`):

Hard cap on domain rows or individual change rows across the run.

## `concurrency` (type: `integer`):

Different hosts in parallel; one active request at a time per destination host.

## `requestDelayMs` (type: `integer`):

Minimum spacing between requests to one host. Default 250 ms; a larger Crawl-delay for this auditor takes precedence.

## `monitorMode` (type: `boolean`):

Compare with compact snapshots in the named ai-crawler-policy-auditor-state store. First run saves baselines.

## `monitorKey` (type: `string`):

Keep the same value across scheduled runs. Separate values create independent baselines. Avoid overlapping runs in one namespace.

## `emitBaselineOnFirstRun` (type: `boolean`):

Emit one change-shaped row for each meaningful initial field. Baseline rows carry baseline: true.

## Actor input object example

```json
{
  "domains": [
    "www.nytimes.com",
    "www.theguardian.com",
    "en.wikipedia.org"
  ],
  "testPath": "",
  "bots": [],
  "includeRaw": false,
  "maxResults": 10,
  "concurrency": 5,
  "requestDelayMs": 250,
  "monitorMode": false,
  "monitorKey": "default",
  "emitBaselineOnFirstRun": false
}
```

# Actor output Schema

## `results` (type: `string`):

All rows in the default dataset.

## `overview` (type: `string`):

Domain snapshot table.

## `changes` (type: `string`):

Individual policy change table.

## `runSummary` (type: `string`):

Statuses, counts and errors in RUN_SUMMARY.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "www.nytimes.com",
        "www.theguardian.com",
        "en.wikipedia.org"
    ],
    "maxResults": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("adastra-connect/ai-crawler-policy-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": [
        "www.nytimes.com",
        "www.theguardian.com",
        "en.wikipedia.org",
    ],
    "maxResults": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("adastra-connect/ai-crawler-policy-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "www.nytimes.com",
    "www.theguardian.com",
    "en.wikipedia.org"
  ],
  "maxResults": 10
}' |
apify call adastra-connect/ai-crawler-policy-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,adastra-connect/ai-crawler-policy-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/VjOGxzDePIXwQc4yb/builds/hVkd0MYIk1f9VWoi8/openapi.json