robots.txt AI Checker | GPTBot, ClaudeBot & AI Crawl Rules

Track how publishers and arbitrary websites handle AI crawlers, LLM training bots, and search indexing tools with this specialized web scraper. As data collection for generative AI models becomes a massive point of contention, tracking policy shifts across target domains is critical for maintaining compliance and understanding the evolving web ecosystem. This tool automatically fetches and parses robots.txt pages, allowing you to extract detailed bot policies specifically targeting agents like GPTBot, ClaudeBot, and Google-Extended. Users rely on this scraper to audit thousands of URLs effortlessly, substituting manual website checks with automated, scheduled runs. Set up daily or weekly monitoring workflows to immediately detect when a domain updates its scraping rules or imposes new restrictions on AI data collection. The system extracts structured data directly from the raw text, returning concrete details about which specific bots are explicitly allowed or disallowed. Output fields include the exact user-agent string, the restricted directory paths, and crawl-delay directives. By identifying exactly what changed since your last run, you can securely build web datasets, respect publisher boundaries, and integrate compliance checks directly into your broader data engineering pipelines.

Run the next report

• Generate an indexability and AI crawler readiness report when robots.txt findings need canonical, sitemap, noindex, and schema evidence.
• Measure public AI brand visibility after crawler policy changes need an observable search-assistant outcome.

Store Quickstart

• Start with store-input.example.json. It uses demoMode=true so the first Store run is safe, cheap, and easy to understand.
• If the compact output is useful, switch to store-input.templates.json and pick one of:
• Demo Quickstart for a trial run
• Production Monitor for recurring dataset snapshots
• Webhook Alert for policy-change notifications

Monitoring Contract

• datasetMode: "all" is the default and preserves one-shot behavior by emitting every result.
• datasetMode: "changes_only" emits the first observed baseline, then emits only changed domains. An unchanged run emits 0 dataset rows and 0 result charges.
• Snapshots are persisted in the fixed named Apify Key-Value Store robotstxt-ai-checker-state; snapshotKey selects only the record inside that store.
• A pending state reservation is saved before delivery and committed only after delivery succeeds. Failed delivery stays blocked instead of being charged again automatically.
• Every emitted Dataset row is pushed individually through the Apify Actor SDK and must receive chargedCount >= 1. The run writes PHASE89_DELIVERY_AUDIT after delivery.
• Source requests allow only public HTTP(S) hosts. Local, private, link-local, metadata, credential-bearing, and non-HTTP(S) URLs are rejected. Redirects are rechecked and limited to 3 hops; responses are capped at 1 MiB and requests time out after 10 seconds.

Key Features

• 🛡️ Compliance-first — Produces audit-ready reports mapping findings to standards (WCAG, GDPR, SOC2)
• 🔒 Non-invasive scanning — Uses only observable public signals — no intrusive probing
• 📊 Severity-scored output — Each finding rated for criticality with remediation guidance
• 📡 Delta-alerting — Flag new findings since last run via webhook delivery
• 📋 Evidence export — Raw headers/responses captured for compliance documentation

Use Cases

Who	Why
Developers	Automate recurring data fetches without building custom scrapers
Data teams	Pipe structured output into analytics warehouses
Ops teams	Monitor changes via webhook alerts
Product managers	Track competitor/market signals without engineering time

Input

Field	Type	Default	Description
domains	array	prefilled	List of domains to analyze robots.txt for AI crawler policies. Max 500.
delivery	string	"dataset"	How to deliver results. 'dataset' saves to Apify Dataset, 'webhook' sends to a URL. In demoMode, delivery is always data
webhookUrl	string	—	Webhook URL to send results to (only used when delivery is 'webhook'). Works with Slack, Discord, or any HTTP endpoint.
snapshotKey	string	"robotstxt-snapshots"	Record key for snapshots inside the fixed robotstxt-ai-checker-state Key-Value Store.
datasetMode	string	"all"	all emits every result; changes_only emits only changed results and charges nothing when unchanged.
concurrency	integer	5	Maximum number of parallel requests. Higher = faster but may trigger rate limits.
dryRun	boolean	false	If true, runs without saving results or sending webhooks. Useful for testing.
demoMode	boolean	false	If true, checks only 1 domain, returns compact policy fields, and disables webhook/snapshot writes.

Input Example

{
  "domains": [
    "google.com",
    "github.com",
    "nytimes.com",
    "openai.com"
  ],
  "delivery": "dataset",
  "snapshotKey": "robotstxt-snapshots",
  "datasetMode": "all",
  "concurrency": 5,
  "dryRun": false,
  "demoMode": false
}

Input Examples

Example: Single domain AI bot audit

{
  "domains": [
    "example.com"
  ],
  "bots": [
    "GPTBot",
    "ClaudeBot",
    "anthropic-ai",
    "PerplexityBot"
  ]
}

Example: Bulk competitor sites

{
  "domains": [
    "competitor1.com",
    "competitor2.com",
    "competitor3.com"
  ],
  "bots": [
    "GPTBot",
    "ClaudeBot"
  ],
  "emitPerBotDisposition": true
}

Example: All-AI-bot policy snapshot

{
  "domains": [
    "nytimes.com",
    "wsj.com",
    "ft.com"
  ],
  "detectAllAiBots": true
}

Output

Field	Type	Description
meta	object
results	array
results[].domain	string
results[].status	string
results[].summary	object
results[].aiPolicies	array
results[].changes	array
results[].checkedAt	timestamp
results[].actorName	string	Stable actor identifier.
results[].rowType	string	robots_txt_policy_snapshot, robots_txt_missing, robots_txt_error, or robots_txt_invalid_domain.
results[].sourceUrl	string/null	Validated final robots.txt URL.
results[].fetchedAt	timestamp	Fetch completion timestamp.
results[].warnings	array	Non-fatal warnings or error context.
results[].chargedEvent	string or null	apify-default-dataset-item for billed Dataset rows; otherwise null.
results[].billingEventName	string	apify-default-dataset-item.
results[].demoApplied	boolean
results[].detailsMasked	boolean
results[].error	null

Output Example

{
  "meta": {
    "generatedAt": "2026-02-22T17:50:20.909Z",
    "totals": {
      "total": 1,
      "requestedDomains": 2,
      "processedDomains": 1,
      "withRobotsTxt": 1,
      "noRobotsTxt": 0,
      "invalidDomains": 0,
      "blockingAi": 0,
      "errors": 0
    },
    "demoApplied": true,
    "limits": {
      "maxDomains": 1,
      "compactPolicies": true,
      "webhookEnabled": false,
      "snapshotWriteEnabled": false
    },
    "upgradeHint": "Demo mode checks 1 domain, disables webhook delivery, and returns a compact policy view. Set demoMode=false to unlock bulk checks and full policy details."
  },
  "results": [
    {
      "domain": "openai.com",
      "status": "ok",
      "summary": {
        "totalCrawlers": 16,
        "blocked": 0,
        "partialBlock": 16,
        "allowed": 0,
        "changed": 0
      },
      "aiPolicies": [
        {
          "crawler": "GPTBot",
          "company": "OpenAI",
          "blocked": false,
          "partialBlock": true,
          "allowed": false

API Usage

Run this actor programmatically using the Apify API. Replace YOUR_API_TOKEN with your token from Apify Console → Settings → Integrations.

cURL

curl -X POST "https://api.apify.com/v2/acts/taroyamada~robotstxt-ai-checker/run-sync-get-dataset-items?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "domains": [ "google.com", "github.com", "nytimes.com", "openai.com" ], "delivery": "dataset", "snapshotKey": "robotstxt-snapshots", "concurrency": 5, "dryRun": false, "demoMode": false }'

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("taroyamada/robotstxt-ai-checker").call(run_input={
  "domains": [
    "google.com",
    "github.com",
    "nytimes.com",
    "openai.com"
  ],
  "delivery": "dataset",
  "snapshotKey": "robotstxt-snapshots",
  "concurrency": 5,
  "dryRun": false,
  "demoMode": false
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

JavaScript / Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('taroyamada/robotstxt-ai-checker').call({
  "domains": [
    "google.com",
    "github.com",
    "nytimes.com",
    "openai.com"
  ],
  "delivery": "dataset",
  "snapshotKey": "robotstxt-snapshots",
  "concurrency": 5,
  "dryRun": false,
  "demoMode": false
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Tips & Limitations

• Schedule weekly runs against your production domains to catch config drift.
• Use webhook delivery to pipe findings into your SIEM (Splunk, Datadog, Elastic).
• For CI integration, block releases on critical severity findings using exit codes.
• Combine with ssl-certificate-monitor for layered cert + headers coverage.
• Findings include links to official remediation docs — share with dev teams via the webhook payload.

See also (AI bot policy + visibility cluster)

• AI Brand Visibility Scraper — Track whether AI assistants actually cite your brand after the policy changes you make here.

FAQ

Is running this against a third-party site legal?

Passive public-header scanning is generally permitted, but follow your own compliance policies. Only scan sites you have authorization for.

How often should I scan?

Weekly for production domains; daily if you have high config-change velocity.

Can I export to a compliance tool?

Use webhook delivery or Dataset API — formats map well to Drata, Vanta, OneTrust import templates.

Is this a penetration test?

No — this actor performs passive compliance scanning only. No exploitation, fuzzing, or auth bypass.

Does this qualify as a SOC2 control?

This actor produces evidence artifacts suitable for SOC2 CC7.1 (continuous monitoring). It is not itself a SOC2 certification.

Related Actors

Security & Compliance cluster — explore related Apify tools:

• Privacy & Cookie Compliance Scanner | GDPR / CCPA Banner Audit — Scan public privacy pages and cookie banners for GDPR/CCPA compliance signals.
• Security Headers Checker API | OWASP Audit — Bulk-audit websites for OWASP security headers, grade each response, and monitor header changes between runs.
• SSL Certificate Monitor API | Expiry + Issuer Changes — Check SSL/TLS certificates in bulk, detect expiry and issuer changes, and emit alert-ready rows for ops and SEO teams.
• DNS / SPF / DKIM / DMARC Audit API — Bulk-audit domains for SPF, DKIM, DMARC, MX, and email-auth posture with grades and fix-ready recommendations.
• Data Breach Disclosure Monitor | HIPAA Breach Watch — Monitor the HHS OCR Breach Portal for new HIPAA data breach disclosures.
• WCAG Accessibility Checker API | ADA & EAA Compliance Audit — Audit websites for WCAG 2.
• 📜 Open-Source License & Dependency Audit API — Audit npm packages for license risk, dependency depth, maintainer activity, and compliance posture.
• Trust Center & Subprocessor Monitor API — Monitor vendor trust centers, subprocessor lists, DPA updates, and security posture changes.

Cost

Pay Per Event:

• Current transition pricing may include a temporary legacy start fee until the scheduled pricing update activates.
• apify-default-dataset-item: $0.011 per delivered result

Example after the scheduled start-fee removal: 1,000 delivered items = $11.00

changes_only runs with no changed domains emit zero rows and incur zero result charges. The fixed snapshot write is not a Dataset event.

No subscription required — you only pay for what you use.

⭐ Was this helpful?

If this actor saved you time, please leave a ★ rating on Apify Store. It takes 10 seconds, helps other developers discover it, and keeps updates free.

Bug report or feature request? Open an issue on the Issues tab of this actor.