# AI Crawler Robots.txt Auditor (`phoenix2810/ai-crawler-robots-auditor`) Actor

Fetch a public website's robots.txt and report which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, and more) are blocked, allowed, or defaulted. Returns an AI-visibility readiness score with recommendations for GEO and AI-SEO strategy.

- **URL**: https://apify.com/phoenix2810/ai-crawler-robots-auditor.md
- **Developed by:** [Sanskar Jaiswal](https://apify.com/phoenix2810) (community)
- **Categories:** SEO tools, AI, Open source
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Crawler Robots.txt Auditor

AI Crawler Robots.txt Auditor fetches a public website's `robots.txt` and reports which AI crawlers are blocked, allowed, or have no explicit rule. It checks ~20 known AI crawler user-agent tokens (GPTBot, OAI-SearchBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, Meta-ExternalAgent, AppleBot, and more), computes an AI-visibility readiness score, and returns recommendations for GEO and AI-SEO strategy.

### Use cases

- SEO and GEO teams deciding which AI products may use their content for training and retrieval.
- AI-SEO consultants auditing client robots.txt files for ChatGPT, Claude, Perplexity, and Google AI Overviews visibility.
- Site migration QA teams confirming robots.txt AI crawler policy did not regress.
- Content licensing teams documenting which AI crawlers a publisher allows or blocks.
- Developer-tool builders adding AI crawler policy checks to monitoring workflows.

### Input

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| `startUrl` | string | Yes | Public website URL to audit. The actor fetches `/robots.txt`. HTTP and HTTPS are supported. Private IP ranges and URL credentials are blocked. |
| `timeoutSeconds` | integer | No | Timeout for the robots.txt HTTP request. Default: 10. Minimum: 3. Maximum: 30. |

### Output

Each run pushes one dataset item.

| Field | Type | Description |
| --- | --- | --- |
| `inputUrl` | string | The URL provided as input. |
| `siteOrigin` | string | Normalized website origin checked by the actor. |
| `robotsUrl` | string | URL of robots.txt that was fetched. |
| `robotsFound` | boolean | Whether a parseable robots.txt was returned. |
| `status` | integer or null | HTTP status of the robots.txt response. |
| `contentType` | string | Response content type for robots.txt. |
| `checkedAt` | string | ISO timestamp for the audit. |
| `score` | integer | AI-visibility readiness score from 0 to 100. |
| `grade` | string | Letter grade: A, B, C, D, or F. |
| `findings` | array | Structured checks with name, status, and points. |
| `recommendations` | array | Plain-language fixes for missing or unclear AI crawler policy. |
| `sitemaps` | array | Sitemap URLs declared in robots.txt. |
| `crawlers` | array | Per-crawler results: name, operator, userAgent, purpose, matchedToken, status, disallowPaths, allowPaths. |
| `blocked` | array | Names of crawlers explicitly blocked. |
| `allowed` | array | Names of crawlers explicitly allowed. |
| `noRule` | array | Names of crawlers with no explicit rule in robots.txt. |
| `partial` | array | Names of crawlers with partial Disallow rules. |
| `error` | string or null | Fetch error message, if any. |

#### Per-crawler status values

| Status | Meaning |
| --- | --- |
| `blocked` | An explicit `User-agent` group for this crawler contains `Disallow: /` (or equivalent root block). |
| `allowed` | An explicit `User-agent` group for this crawler exists and does not block the root. |
| `partial` | An explicit `User-agent` group for this crawler contains non-root `Disallow` rules. |
| `no-rule` | No explicit `User-agent` group for this crawler was found. |

### Example input

```json
{
  "startUrl": "https://example.com",
  "timeoutSeconds": 10
}
```

### Example output

```json
{
  "inputUrl": "https://example.com",
  "siteOrigin": "https://example.com",
  "robotsUrl": "https://example.com/robots.txt",
  "robotsFound": true,
  "status": 200,
  "contentType": "text/plain",
  "checkedAt": "2026-08-05T00:00:00.000Z",
  "score": 62,
  "grade": "D",
  "findings": [
    { "name": "robots.txt found", "status": "pass", "points": 20 },
    { "name": "AI crawler coverage", "status": "warn", "points": 42 }
  ],
  "recommendations": [
    "GPTBot is explicitly blocked. Unblock it to allow inclusion in OpenAI AI products if your GEO/AI-SEO strategy favors visibility."
  ],
  "sitemaps": ["https://example.com/sitemap.xml"],
  "crawlers": [
    {
      "name": "GPTBot",
      "operator": "OpenAI",
      "userAgent": "GPTBot",
      "purpose": "Powers ChatGPT training and retrieval.",
      "matchedToken": "GPTBot",
      "status": "blocked",
      "disallowPaths": ["/"],
      "allowPaths": []
    },
    {
      "name": "ClaudeBot",
      "operator": "Anthropic",
      "userAgent": "ClaudeBot",
      "purpose": "Powers Claude model training.",
      "matchedToken": "ClaudeBot",
      "status": "allowed",
      "disallowPaths": [],
      "allowPaths": ["/"]
    }
  ],
  "blocked": ["GPTBot", "Bytespider"],
  "allowed": ["ClaudeBot"],
  "noRule": ["PerplexityBot"],
  "partial": [],
  "error": null
}
```

### Security

The actor only fetches public HTTP and HTTPS URLs. It rejects URL credentials, private IPv4 literals, private IPv6 literals, and hostnames that resolve to private IP ranges. Redirects are followed manually and revalidated before each follow-up request. Response bodies are capped before parsing.

### Pricing

Suggested pay-per-event pricing:

| Event | Price |
| --- | ---: |
| Actor start | `$0.005` |
| Site audited | `$0.01` |

This keeps single-domain AI crawler policy checks inexpensive while charging proportionally for successful dataset results.

### FAQ

#### Does this actor block or unblock crawlers on my site?

No. It only reads and reports the existing robots.txt policy. It does not modify any files or submit any changes.

#### Which AI crawlers does it check?

It checks ~20 known AI crawler user-agent tokens including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, anthropic-ai, Google-Extended, Googlebot-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Meta-ExternalAgent, AppleBot, Amazonbot, YouBot, Diffbot, Omgilibot, ImagesiftBot, and Timpibot.

#### Does it require login or credentials?

No. It only fetches the public robots.txt file.

#### Does it enforce robots.txt semantics for crawling?

No. It is an audit and reporting tool. It surfaces directives so GEO and AI-SEO teams can make explicit policy choices. It is not a robots.txt enforcement engine.

#### What keep metric should be used?

Keep or improve this actor if it receives organic Store impressions, runs, revenue, or inbound questions within 60 to 90 days. Archive it if there is no discovery and no usage.

# Actor input Schema

## `startUrl` (type: `string`):

Public website URL to audit. The actor fetches /robots.txt. HTTP and HTTPS only. Private IP ranges and URL credentials are blocked.

## `timeoutSeconds` (type: `integer`):

Timeout for the robots.txt HTTP request.

## Actor input object example

```json
{
  "startUrl": "https://example.com",
  "timeoutSeconds": 10
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrl": "https://example.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("phoenix2810/ai-crawler-robots-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrl": "https://example.com" }

# Run the Actor and wait for it to finish
run = client.actor("phoenix2810/ai-crawler-robots-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrl": "https://example.com"
}' |
apify call phoenix2810/ai-crawler-robots-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=phoenix2810/ai-crawler-robots-auditor",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/mbGzicWpKykbakVr5/builds/7YXTJwvR253LFd6TK/openapi.json