AI Crawler Checker: does a website block GPTBot, ClaudeBot or Google-Extended?

Check, for a list of websites, which AI crawlers each one blocks in robots.txt, and whether it publishes llms.txt or text-and-data-mining (TDM) opt-out signals. Enter domains and get one row per domain with a verdict for 30 user agents, 29 AI crawlers plus Googlebot as a search reference (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Google-Extended, Applebot-Extended, PerplexityBot, CCBot, Meta-ExternalAgent, Bytespider, Amazonbot and more), plus llms.txt, TDMRep, noai meta tags and ai.txt. Schedule it in monitoring mode and each run returns only the policy changes. These are observed machine-readable signals for review, not legal advice or a rights determination.

Use cases

• AI dataset and RAG teams: document which sources opt out of AI training before you crawl, with a dated snapshot per domain.
• Publishers and media groups: compare your robots.txt AI crawler policy with competitors and peers.
• SEO and GEO agencies: audit client sites for accidental blocks of AI search bots (OAI-SearchBot, PerplexityBot, Claude-SearchBot) that can keep them out of AI answers.
• Brand and legal ops: track when a site starts or stops blocking AI crawlers, with monitor mode and a schedule.

Sample output: AI crawler robots.txt audit

Real rows from a run on October 22, 2026, shortened:

domain	robotsTxt.found	blocksAnyAiBot	blocksAllTrainingBots	tdmOptOutSignal	llmsTxt.found	GPTBot verdict
example.com	false	false	false	false	false	allowed (no robots.txt)
www.w3.org	true	true	false	true	false	partial

Each row also has bots (token, operator, purpose, verdict for / and your test path), blockedBots, allowedBots, aiPolicyScore, tdmrep, metaRobots, xRobotsTag, aiTxt, resources, errors and checkedAt.

What data can you get: robots.txt, llms.txt and TDM signals

Fields	Data
input, domain, finalUrl, checkedAt	Normalized input host, host, final homepage URL (or robots URL if the homepage was skipped), and UTC observation time.
robotsTxt	HTTP status, file presence, bytes read, up to 20 sitemaps, and wildcard full-block flag.
bots	Token, operator, purpose, official documentation URL, named-group flag, verdict, effective verdict, and decisions for / and your optional path.
blockedBots, allowedBots	Tokens effectively fully blocked or unrestricted. Partially restricted tokens remain in bots.
blocksAllTrainingBots, blocksAnyAiBot	Selected training tokens all fully blocked, or any selected AI bot fully or partly restricted. Googlebot is a search reference and excluded from the second flag.
aiPolicyScore	An informational 0-100 published-policy score, explained below.
llmsTxt, llmsFullTxt	Text-file presence and bytes read; llms.txt also has a link count. The title is null to avoid retaining names.
tdmrep	Observed TDMRep entries, whether any parsed tdm-reservation value is 1, and first policy URL signal. Nonstandard location paths are fingerprinted.
tdmReservationHeader, tdmPolicyHeader, tdmMeta	Homepage TDM headers and meta declarations.
xRobotsTag, metaRobots, noaiMeta	Homepage robots headers, generic and bot-specific meta directives, and noai/noimageai detection.
linkHints, aiTxt	Relevant published link relations and Spawning-style ai.txt presence and full-disallow signal.
tdmOptOutSignal	Any selected training bot restriction, TDM reservation, or noai/noimageai meta signal.
resources, homepage, errors, status	Per-resource final URLs, final hosts, statuses, byte caps, skipped-homepage reason, and errors.
rawRobotsTxt, botsAsOf	Optional reconstructed robots directives capped at 50 KB; comments are omitted and nonstandard paths fingerprinted, plus registry date.
changeType, field, previousValue, currentValue, detectedAt	One changed field per row in monitoring mode.

Use the dataset's overview and changes table views. Apify supports JSON, CSV, Microsoft Excel and other dataset exports. Failed or missing resources are visible in resources and errors; an unavailable policy is never represented as a confirmed allow decision.

How to check which AI crawlers a website blocks

1. Enter bare domains or public HTTP/HTTPS URLs. Paths in domain inputs are ignored; hosts are deduplicated. Subdomains remain separate.
2. Optionally select bot tokens and a testPath. Unknown tokens are evaluated as custom bots with mixed purpose. The test path is evaluated locally and never fetched.
3. Set maxResults and run. Each normal output row is one billable domain event, including an unreachable-domain report.
4. Open the dataset and RUN_SUMMARY in the default key-value store for per-target statuses, counts and errors.
5. Enable monitoring and keep the same monitorKey for scheduled comparisons.

Empty API input {} uses the three domains below, a default maximum of 100 rows and concurrency 5. The input editor prefills maxResults with 10. Each host has one active request at a time and a default 250 ms minimum request interval. A larger published Crawl-delay for this auditor takes precedence.

{
  "domains": ["www.nytimes.com", "www.theguardian.com", "en.wikipedia.org"],
  "testPath": "/news/",
  "bots": ["GPTBot", "ClaudeBot", "Google-Extended"],
  "includeRaw": false,
  "maxResults": 10,
  "concurrency": 5,
  "requestDelayMs": 250,
  "monitorMode": false
}

Example of a shortened domain row. Policy values vary by date and region:

{
  "input": "publisher.example",
  "domain": "publisher.example",
  "checkedAt": "2026-10-21T12:00:00.000Z",
  "robotsTxt": {
    "status": 200,
    "found": true,
    "sizeBytes": 87,
    "sitemaps": [],
    "hasWildcardDisallowAll": false
  },
  "bots": [
    {"token": "GPTBot", "operator": "OpenAI", "purpose": "training", "verdict": "blocked", "effectiveVerdict": "blocked", "namedGroup": true, "rootAllowed": false, "testPathAllowed": null}
  ],
  "blockedBots": ["GPTBot"],
  "allowedBots": [],
  "blocksAllTrainingBots": true,
  "blocksAnyAiBot": true,
  "aiPolicyScore": 100,
  "llmsTxt": {"found": false, "sizeBytes": 0, "title": null, "linkCount": 0},
  "llmsFullTxt": {"found": false, "sizeBytes": 0},
  "tdmrep": {"found": true, "reservation": 1, "policyUrl": null, "entries": [{"location": "/", "tdm-reservation": 1, "tdm-policy": null}]},
  "tdmReservationHeader": null,
  "xRobotsTag": [],
  "metaRobots": [],
  "noaiMeta": false,
  "aiTxt": {"found": false, "disallowsAll": false},
  "tdmOptOutSignal": true,
  "errors": [],
  "status": "ok",
  "botsAsOf": "2026-10-21"
}

Observed robots signals and score

allowed means no matching disallow signal was found in the observed rules. blocked means the observed rules indicate a full disallow; partial means some tested paths have disallow signals. These technical labels describe parsed rules, not legal permission or actual crawler behavior. no_rule means no group names the token and the wildcard group, if present, is used. effectiveVerdict records the resulting allowed/blocked/partial policy even for wildcard fallback. If robots.txt cannot be interpreted, effective decisions are null. rootAllowed and testPathAllowed are explicit path decisions. Partial-policy classification tests rule-derived representative paths; unusual interacting wildcard patterns can need manual inspection.

Summary arrays and flags use effective decisions. The score is round(80 * fullyBlockedTrainingTokens / selectedTrainingTokens + 20 * hasTdmReservation). If no training tokens are selected, the first term is zero and blocksAllTrainingBots is false. TDM reservation is 1 in any parsed TDMRep entry or a homepage TDM header/meta declaration. Multiple reservation surfaces still contribute only 20 points. Partial blocks and noai alone contribute no score points, but can set tdmOptOutSignal. Legacy training tokens are included in the registry and denominator. This score measures observed signals, not compliance, protection, crawler behavior, or permission to reuse content.

Pricing

This Actor uses pay per event pricing: $2.50 per 1,000 domain rows returned (Apify Store discount tiers pay less). The price on the Store page is authoritative. Platform compute is included, so you pay for the rows you receive plus Apify's small per-run start fee, nothing else.

Normal mode bills one domain event per domain row. Monitoring bills one domain event per changed-field row. A silent first baseline and an unchanged monitor run emit no billable rows. The Actor honors the spending limit and maxResults, and writes batches of up to 100 rows.

Monitor robots.txt AI crawler policy changes

Enable monitorMode and choose a stable monitorKey. State persists per input host in the named ai-crawler-policy-auditor-state key-value store. The first successful observation saves a baseline and emits nothing, unless emitBaselineOnFirstRun is enabled. Baseline events carry baseline: true.

Snapshots retain bot verdicts, robots availability, llms.txt presence and 1 KB size buckets, TDMRep entries and reservation, and homepage header/meta values. Monitoring emits bot_blocked, bot_unblocked, llms_txt_added, llms_txt_removed, tdm_reservation_changed, meta_changed, or robots_unreachable. Increasing/decreasing partial restrictions count as bot blocked/unblocked. llms.txt size-bucket changes and robots recovery use meta_changed; their field identifies what changed.

An unavailable resource keeps its previous values, so a transient error does not generate removal or unblock alerts. A robots outage emits its availability change. If a run stops partway through delivering a host's changes, its snapshot is not advanced and some delivered events can repeat next run. Use domain, field, values and timestamps for downstream deduplication. Bot selection, test path or registry date changes reset that host's comparison scope. Run one schedule at a time per namespace to avoid concurrent snapshot updates.

Scheduling and integrations

Use Apify schedules for periodic audits, webhooks for run completion, or the Apify API for on-demand checks. Connect datasets to Make, Zapier or n8n workflows. AI agents can invoke the Actor through Apify MCP integrations. An unchanged monitor run produces an empty change feed; inspect RUN_SUMMARY to distinguish silence from failures.

Is it legal?

The Actor reads public policy endpoints and the public homepage without login, cookies, browser automation or proxies. It identifies itself and respects homepage robots rules for its own user agent. A homepage disallowed by robots is skipped; the public machine-policy files are still fetched to audit published instructions. No pages beyond these resources and their redirects are crawled. The output omits source prose and llms.txt titles, retains only recognized header/meta directives, removes input paths and URL queries, and fingerprints nonstandard source paths. Optional raw output reconstructs directives without comments or unselected source bot tokens. Submit only organizational domains and bot product tokens, without personal information. Fingerprinted URL paths are comparison identifiers, not working page links.

This Actor provides observed machine-readable signals for review. It does not give legal advice, a compliance verdict, a rights-reservation determination, or a license. A value of 0, 1, absent, allowed or blocked describes only an observed signal or parser result. Consult qualified counsel for legal questions about your intended use, applicable law and site terms.

Limitations

• A snapshot reports what this user agent received at one time. Regional, CDN, firewall and bot-specific responses can differ. HTTP 403 and other failures are reported without bypass attempts.
• Only the root policy files and homepage are inspected. testPath is not fetched. Bot-specific meta tags and TDM scopes are observations, not a complete site-wide rights decision.
• Requests have a 20-second maximum timeout, at most five redirects and a 1 MB body cap. The homepage is capped at 512 KB and llms-full.txt at 64 KB. sizeBytes is bytes read, including an error body if present; check found, status and resources.*.truncated. Counts describe only the read prefix. llms.txt titles are omitted.
• Localhost, literal IP addresses, internal hostnames and DNS answers in private, metadata or reserved address ranges are blocked, including redirects. Responses are streamed with byte caps and a 16 KB header cap. Robots matching avoids backtracking regular expressions; oversized policy rule sets are reported as errors.
• Each domain has a 25-second audit budget, including retries and delays, to keep the small prefill fast. Remaining observations can be incomplete and have errors. Runs stop gracefully 45 seconds before their platform timeout.
• HTML soft-404 detection checks HTML content types and the html tag. Some other soft-404 responses can evade it. Regex metadata extraction does not execute scripts or fully parse malformed HTML.
• robots.txt is voluntary; a bot's actual behavior can differ. User-request fetchers may not apply robots.txt. Google-Extended and similar tokens can govern use rather than a separate network crawler.
• llms.txt is a discovery convention and its presence grants no training permission. ai.txt conventions vary. This Actor recognizes full-disallow lines in the Spawning convention.
• The registry is dated 2026-10-21, includes historical tokens, and does not cover every AI system. Custom tokens use mixed purpose and do not affect the training denominator.
• If every target is unreachable, invalid or has no usable observations, the run fails after saving its summary. Individual target failures do not fail otherwise useful runs. Missing policy files are valid observations.

FAQ

Does no_rule mean a bot is allowed? Read effectiveVerdict. A wildcard policy can block a bot that has no named group.

Does a high score determine rights or permissions? No. The score summarizes selected published opt-out signals. Interpret scopes and errors and obtain any necessary rights separately.

Why did a monitoring run produce zero rows? It may have saved its first baseline or found no changes. Check the summary for resource failures, which do not clear the last known policy.

Can I test a URL path or add another bot? Use testPath and add its product token to bots. No request is sent to the test path.

Trademarks and non-affiliation

Ad Astra Connect is the publisher. Apify, Microsoft Excel, OpenAI, GPTBot, ChatGPT, OAI-SearchBot, Anthropic, Claude, ClaudeBot, Google, Googlebot, Google-Extended, Spawning, Make, Zapier, n8n, The New York Times, The Guardian and Wikipedia are trademarks of their respective owners. This Actor is not affiliated with, endorsed or sponsored by those brands or their owners. Bot names identify the policies being checked. All other operator and bot brands appearing in generated registry fields belong to their respective owners; the same non-affiliation disclaimer applies to every listed brand.

Support

Use the Actor's Issues tab on Apify. Include the run ID, a non-sensitive input example and relevant error fields.