# AI Crawler Robots.txt Audit (`davidbenittah/ai-crawler-robots-audit`) Actor Check whether ChatGPT, Claude, Perplexity, Google AI and other major AI crawlers can access each website, then separate search visibility risks from training protection. - **URL**: https://apify.com/davidbenittah/ai-crawler-robots-audit.md - **Developed by:** [David](https://apify.com/davidbenittah) (community) - **Categories:** SEO tools, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## AI Crawler Robots.txt Audit Audit whether **ChatGPT, Claude, Perplexity, Google AI, Apple and major model-training crawlers** may access each website according to its live `robots.txt`. The result separates AI answer visibility from model-training access, because blocking those are two different business decisions. A single old rule can quietly block an answer crawler and reduce the chance that a brand appears or gets cited in AI search. The opposite mistake also happens: a publisher wants answer-engine visibility but unintentionally leaves every training crawler open. This Actor turns the live policy into a plain recommendation instead of making someone compare bot names by hand. ### What it checks For each public domain, the Actor: 1. Downloads the live `robots.txt` without requiring a login, cookies, or credentials. 2. Applies specific user-agent groups before wildcard rules, including `Allow` and `Disallow` precedence at the site root. 3. Reports answer/search crawlers separately from training crawlers. 4. Checks whether `/llms.txt` exists as an additional readiness signal. 5. Returns the exact matched rule for every crawler so an SEO specialist can verify the result. Search and answer crawlers include OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-SearchBot. Training or extended-use controls include GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Amazonbot and FacebookBot. Google-Extended does not control inclusion or ranking in Google Search. Bot behavior and vendor naming can change; the output is an audit of the current public file, not a promise of indexing or citation. ### Ready-to-run example ```json { "domains": ["example.com", "your-client-site.com"], "includeTrainingBots": true, "blockedOnly": false, "requestTimeoutSecs": 15 } ```` Every required input has a working default. Enter up to 50 domains or full public URLs. No account or API key is needed. ### Output example ```json { "domain": "your-client-site.com", "robotsUrl": "https://your-client-site.com/robots.txt", "robotsStatus": 200, "llmsTxtFound": true, "searchVisibilityRisk": true, "blockedSearchBots": ["OAI-SearchBot", "PerplexityBot"], "blockedTrainingBots": ["GPTBot", "ClaudeBot"], "allowedSearchBots": ["ChatGPT-User", "Claude-SearchBot"], "recommendation": "Review robots.txt: OAI-SearchBot, PerplexityBot cannot crawl the site root, which may reduce visibility in AI answers.", "checkedAt": "2026-07-19T10:00:00.000Z" } ``` The full `crawlerResults` array includes each crawler's company, purpose, allowed status, exact matched rule, and whether the decision came from a specific or wildcard group. A `SUMMARY` record reports websites checked and sites with an AI search visibility risk. ### Who uses it - **SEO and GEO agencies:** audit every client after a migration, CDN change, or robots.txt deployment. Use `blockedOnly: true` on a schedule to keep the result list actionable. - **B2B SaaS teams:** verify that an inherited robots.txt file does not block answer engines while normal Google SEO still looks healthy. - **Publishers:** distinguish protecting content from model training from disappearing from AI-assisted discovery. - **Web agencies:** add an AI crawler access check to launch and maintenance reports without manually tracking every bot name. ### Reading the result `searchVisibilityRisk: true` means at least one listed answer/search crawler is blocked at `/`. It does **not** prove the website is absent from an AI product: vendors can use other indexes and systems. A healthy result also does not guarantee citation. The Actor reports the access policy accurately and leaves the business decision to you. A missing `robots.txt` normally means crawling is allowed. A network error is different: the Actor returns status `0`, includes the error, and recommends rerunning before changing policy. A missing `llms.txt` is informational, not automatically a defect. ### Data, limits, and responsible use The Actor reads only public `robots.txt` and `llms.txt` files. It collects no personal data, does not bypass access controls, and does not log in. Runs are capped at 50 websites to keep cost and load predictable. Schedule weekly or after deployments rather than polling aggressively. ### Unofficial tool disclaimer This Actor is an **unofficial, independent audit tool**. It is not affiliated with or endorsed by OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, or Common Crawl. Product names identify public crawler user agents only. Always confirm policy changes against the vendors' current documentation before editing a production `robots.txt` file. # Actor input Schema ## `domains` (type: `array`): Domains or public website URLs. Hard maximum: 50 per run. ## `includeTrainingBots` (type: `boolean`): Also report crawlers used primarily for model training, separately from AI search and answer crawlers. ## `blockedOnly` (type: `boolean`): Useful for scheduled agency audits: healthy websites are omitted from the dataset. ## `requestTimeoutSecs` (type: `integer`): Maximum wait for robots.txt, from 3 to 30 seconds. ## Actor input object example ```json { "domains": [ "example.com" ], "includeTrainingBots": true, "blockedOnly": false, "requestTimeoutSecs": 15 } ``` # Actor output Schema ## `results` (type: `string`): Structured AI crawler access audit items for every checked website. ## `summary` (type: `string`): Counts of checked websites, visibility risks, and produced result items. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "domains": [ "example.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("davidbenittah/ai-crawler-robots-audit").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "domains": ["example.com"] } # Run the Actor and wait for it to finish run = client.actor("davidbenittah/ai-crawler-robots-audit").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "domains": [ "example.com" ] }' | apify call davidbenittah/ai-crawler-robots-audit --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=davidbenittah/ai-crawler-robots-audit", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "AI Crawler Robots.txt Audit", "description": "Check whether ChatGPT, Claude, Perplexity, Google AI and other major AI crawlers can access each website, then separate search visibility risks from training protection.", "version": "1.0", "x-build-id": "DCRGHPL6G8K1wArsa" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/davidbenittah~ai-crawler-robots-audit/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-davidbenittah-ai-crawler-robots-audit", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/davidbenittah~ai-crawler-robots-audit/runs": { "post": { "operationId": "runs-sync-davidbenittah-ai-crawler-robots-audit", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/davidbenittah~ai-crawler-robots-audit/run-sync": { "post": { "operationId": "run-sync-davidbenittah-ai-crawler-robots-audit", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "domains" ], "properties": { "domains": { "title": "Websites to audit", "type": "array", "description": "Domains or public website URLs. Hard maximum: 50 per run.", "default": [ "example.com" ], "items": { "type": "string" } }, "includeTrainingBots": { "title": "Include model-training crawlers", "type": "boolean", "description": "Also report crawlers used primarily for model training, separately from AI search and answer crawlers.", "default": true }, "blockedOnly": { "title": "Output only websites with blocked AI search crawlers", "type": "boolean", "description": "Useful for scheduled agency audits: healthy websites are omitted from the dataset.", "default": false }, "requestTimeoutSecs": { "title": "Request timeout in seconds", "minimum": 3, "maximum": 30, "type": "integer", "description": "Maximum wait for robots.txt, from 3 to 30 seconds.", "default": 15 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```