# AI Crawler Policy Audit API | 🤖 $1/1K (`webdata_labs/ai-crawler-policy-audit-api`) Actor

\[💵 $1.00 / 1K] Audit robots.txt and llms.txt to see whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers are allowed.

- **URL**: https://apify.com/webdata\_labs/ai-crawler-policy-audit-api.md
- **Developed by:** [WebData Labs](https://apify.com/webdata_labs) (community)
- **Categories:** SEO tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 ai crawler policy audit rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## AI Crawler Policy Audit API

Audit any website's `robots.txt` and `llms.txt` to understand whether major AI crawlers are allowed, blocked, or not explicitly configured. Turn a manual policy review into a structured dataset for AI governance, SEO, legal operations, content licensing, and competitive research.

The Actor checks **GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot** and returns one clean row per domain.

### 🤖 What you get

- AI crawler access status for five major bots
- `robots.txt` and `llms.txt` HTTP status codes
- Direct URLs to both policy files
- A clear `allowed`, `blocked`, `not_specified`, or `unknown` classification
- A short recommendation for every audited domain
- Explicit error rows for unreachable websites instead of silently missing results

Results are ready to download as JSON, CSV, Excel, XML, or RSS and can be consumed through the Apify API.

### 🎯 Common use cases

- Audit your own website before changing AI crawler policy
- Compare how competitors treat generative-AI crawlers
- Monitor a portfolio of media, ecommerce, SaaS, or publisher domains
- Detect policy changes with scheduled recurring runs
- Build an evidence trail for AI-governance and compliance reviews
- Enrich an SEO or domain-intelligence dataset with crawler-access metadata

### 🚀 Ready-to-run examples

- [Check AI crawler access for your website](https://apify.com/webdata_labs/ai-crawler-policy-audit-api/examples/check-ai-crawler-access-for-your-site) — inspect GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot policy for a single domain.
- [Audit competitor AI crawler policies](https://apify.com/webdata_labs/ai-crawler-policy-audit-api/examples/audit-competitor-ai-crawler-policies) — compare crawler rules across OpenAI, Anthropic, and Google.
- [Monitor AI bot robots.txt policies](https://apify.com/webdata_labs/ai-crawler-policy-audit-api/examples/monitor-ai-bot-robots-txt-policies) — create a reusable portfolio check for scheduled monitoring.

### ⚙️ Input

Provide domains, full URLs, or a mixture of both. The Actor normalizes and deduplicates them automatically.

```json
{
  "domains": [
    "openai.com",
    "https://www.anthropic.com",
    "perplexity.ai"
  ],
  "maxDomains": 100,
  "timeoutSecs": 15
}
````

#### Input fields

| Field | Description | Default |
|---|---|---|
| `domains` | Domains or URLs to audit | Required |
| `maxDomains` | Maximum unique domains processed in one run, up to 1,000 | `100` |
| `timeoutSecs` | Maximum wait for each policy-file request | `15` |

### 📤 Output

```json
{
  "domain": "example.com",
  "robotsUrl": "https://example.com/robots.txt",
  "robotsStatus": 200,
  "llmsTxtUrl": "https://example.com/llms.txt",
  "llmsTxtStatus": 404,
  "llmsTxtFound": false,
  "gptbot": "allowed",
  "claudebot": "not_specified",
  "perplexitybot": "blocked",
  "googleExtended": "allowed",
  "commonCrawl": "allowed",
  "recommendation": "Review blocked AI crawler policies."
}
```

#### Policy values

| Value | Meaning |
|---|---|
| `blocked` | The applicable user-agent group contains an all-site `Disallow: /` rule |
| `allowed` | An applicable group exists without an all-site block |
| `not_specified` | No matching bot-specific or wildcard group was found |
| `unknown` | The policy file could not be retrieved or interpreted |

This is a technical policy check, not legal advice. It reports published crawler directives; it does not determine whether a crawler actually obeyed them.

### 🔌 API usage

Start a synchronous run and receive dataset rows directly:

```bash
curl "https://api.apify.com/v2/acts/webdata_labs~ai-crawler-policy-audit-api/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"domains":["openai.com","anthropic.com"],"maxDomains":100,"timeoutSecs":15}'
```

You can also schedule recurring audits, trigger runs from Make or Zapier, receive webhooks, or use the official Apify JavaScript and Python clients.

### 💵 Pricing

The Actor costs **$1 per 1,000 audited domains** ($0.001 per exported row). You pay for delivered audit rows; Apify platform usage is included in the event price shown on the Actor page.

### 🌍 Coverage and limitations

- The Actor audits publicly accessible `robots.txt` and `llms.txt` files.
- Websites can temporarily fail, redirect unexpectedly, or block automated requests; these cases are returned with an error or non-200 status.
- A missing `llms.txt` is common and is not automatically a problem.
- Policy classifications focus on all-site blocking rules and are intentionally conservative.
- Results can change whenever a website updates its crawler policy, so schedule recurring runs when change detection matters.

### ❓ FAQ

**Does this prove that an AI company used my content?**\
No. It only audits the crawler directives currently published by the website.

**Is `llms.txt` the same as `robots.txt`?**\
No. `robots.txt` provides crawler directives. `llms.txt` is an emerging convention for presenting site information to language models and tools.

**Can I audit domains without typing `https://`?**\
Yes. Plain domains and full URLs are both accepted and normalized.

**Can I monitor policy changes?**\
Yes. Save your portfolio as an Apify Task, run it on a schedule, and compare datasets or send results through a webhook.

### 🛠️ Support

If a policy format is classified incorrectly or a domain cannot be fetched, open an issue on the Actor page and include the run URL, affected domain, and expected result.

# Actor input Schema

## `domains` (type: `array`):

Sites to audit. Domains are accepted without https://.

## `maxDomains` (type: `integer`):

Safety cap for domains audited in one run.

## `timeoutSecs` (type: `integer`):

Maximum wait for each policy file request.

## Actor input object example

```json
{
  "domains": [
    "openai.com",
    "anthropic.com"
  ],
  "maxDomains": 100,
  "timeoutSecs": 15
}
```

# Actor output Schema

## `audit` (type: `string`):

No description

## `OUTPUT` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "openai.com",
        "anthropic.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("webdata_labs/ai-crawler-policy-audit-api").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "openai.com",
        "anthropic.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("webdata_labs/ai-crawler-policy-audit-api").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "openai.com",
    "anthropic.com"
  ]
}' |
apify call webdata_labs/ai-crawler-policy-audit-api --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=webdata_labs/ai-crawler-policy-audit-api",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "AI Crawler Policy Audit API | 🤖 $1/1K",
        "description": "[💵 $1.00 / 1K] Audit robots.txt and llms.txt to see whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers are allowed.",
        "version": "0.1",
        "x-build-id": "4OFCPbJhMtkj7r3Jj"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/webdata_labs~ai-crawler-policy-audit-api/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-webdata_labs-ai-crawler-policy-audit-api",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/webdata_labs~ai-crawler-policy-audit-api/runs": {
            "post": {
                "operationId": "runs-sync-webdata_labs-ai-crawler-policy-audit-api",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/webdata_labs~ai-crawler-policy-audit-api/run-sync": {
            "post": {
                "operationId": "run-sync-webdata_labs-ai-crawler-policy-audit-api",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "required": [
                    "domains"
                ],
                "properties": {
                    "domains": {
                        "title": "Domains or URLs",
                        "type": "array",
                        "description": "Sites to audit. Domains are accepted without https://.",
                        "default": [
                            "openai.com",
                            "anthropic.com"
                        ],
                        "items": {
                            "type": "string"
                        }
                    },
                    "maxDomains": {
                        "title": "Maximum domains",
                        "minimum": 1,
                        "maximum": 1000,
                        "type": "integer",
                        "description": "Safety cap for domains audited in one run.",
                        "default": 100
                    },
                    "timeoutSecs": {
                        "title": "Request timeout (seconds)",
                        "minimum": 3,
                        "maximum": 60,
                        "type": "integer",
                        "description": "Maximum wait for each policy file request.",
                        "default": 15
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```