# Robots AI Crawler Policy Checker (`seeb/robots-ai-crawler-policy-checker`) Actor

Analyze robots.txt and crawler policy text for AI bot access rules, disallow paths, sitemap references, and blocked model-training crawlers.

- **URL**: https://apify.com/seeb/robots-ai-crawler-policy-checker.md
- **Developed by:** [Techionik](https://apify.com/seeb) (community)
- **Categories:** SEO tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$25.00 / 1,000 crawler policies

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Robots AI Crawler Policy Checker

Analyze robots.txt and crawler policy text for AI bot access rules, disallow paths, sitemap references, and blocked model-training crawlers.

### Why This Actor Is Useful

This actor audits robots.txt and crawler-policy text for AI search and model-training crawler rules. It is useful for publishers, SEO teams, and AI visibility consultants who need to know which bots can access which paths.

It reads public robots/crawler-policy text and does not bypass private access controls.

### What It Extracts

- `siteName` - Site.
- `crawlerName` - Crawler.
- `policyAction` - Action.
- `disallowedPath` - Disallowed path.
- `allowedPath` - Allowed path.
- `sitemapUrl` - Sitemap.
- `crawlDelay` - Crawl delay.
- `aiTrainingSignal` - AI training signal.
- `riskLevel` - Risk level.
- `sourceUrl` - Source URL.
- `evidenceText` - Evidence.

### Practical Workflows

- Check whether GPTBot, Google-Extended, ClaudeBot, CCBot, or other AI crawlers are blocked.
- Collect sitemap URLs and crawl-delay rules.
- Prepare AI visibility and crawler-access audits.
- Compare policy differences across publisher domains.

### Input

Add robots.txt URLs or AI crawler policy pages in **Pages to scan**. If you already have the robots.txt content, paste it into **Paste robots.txt text instead**. Keep the default result limit for most crawler-policy checks.

- `startUrls` - Add robots.txt URLs or public AI crawler policy pages. Example: https://example.com/robots.txt
- `sourceText` - Optional: paste robots.txt lines or AI crawler policy text that mentions GPTBot, Google-Extended, ClaudeBot, CCBot, or sitemap rules.
- `maxResults` - Stop after this many useful rows. Most users can leave this at 100.
- `requestTimeoutSecs` - How long to wait for each URL. Increase only for slow public pages.

### Example Input

```json
{
    "sourceText": [
        "ExampleNews robots.txt User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: /premium Sitemap: https://examplenews.com/sitemap.xml Crawl-delay: 5",
        "Publisher policy states AI training crawlers are not permitted. User-agent: CCBot Disallow: /archive/"
    ],
    "maxResults": 10,
    "requestTimeoutSecs": 20
}
```

### Example Output

```json
{
    "siteName": "ExampleNews",
    "crawlerName": "GPTBot",
    "policyAction": "Disallow",
    "disallowedPath": "/",
    "allowedPath": null,
    "sitemapUrl": "https://examplenews.com/sitemap.xml",
    "crawlDelay": "5",
    "aiTrainingSignal": null,
    "riskLevel": "disallow: /",
    "sourceUrl": "pasted://source-1",
    "evidenceText": "ExampleNews robots.txt User-agent: GPTBot Disallow: / User-agent: Google-Extended Disallow: /premium Sitemap: https://examplenews.com/sitemap.xml Crawl-delay: 5"
}
```

### Reliability Notes

The actor is intentionally lightweight and fast. It does not try to bypass captcha systems, private pages, paywalls, or login walls. When a site blocks direct fetching, paste the public visible text into `sourceText`; the same actor-specific parser will still produce structured rows.

### Limitations

- The actor extracts only information present in the supplied source.
- Missing source details remain blank instead of being guessed.
- Highly unusual wording may require pasted text or cleaner source snippets for best results.

### Output

Rows are written to the default Apify dataset with actor-specific fields and source evidence. The output is designed for spreadsheets, dashboards, client audits, procurement notes, SEO reports, developer research, and other marketplace buyer workflows.

# Actor input Schema

## `startUrls` (type: `array`):

Paste one or more robots.txt URLs or public AI crawler policy pages. Example: https://example.com/robots.txt

## `maxResults` (type: `integer`):

Stop after this many useful rows. Leave this at 100 for normal runs.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://example.com/robots.txt"
    }
  ],
  "maxResults": 100
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://example.com/robots.txt"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("seeb/robots-ai-crawler-policy-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [{ "url": "https://example.com/robots.txt" }] }

# Run the Actor and wait for it to finish
run = client.actor("seeb/robots-ai-crawler-policy-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://example.com/robots.txt"
    }
  ]
}' |
apify call seeb/robots-ai-crawler-policy-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,seeb/robots-ai-crawler-policy-checker"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/u18UElcKgOaNhJ58g/builds/qbPROgAzPVQ6MPVN1/openapi.json