# AI Crawler & Agent Access Checker (robots.txt, llms.txt) (`fast_data_asap/ai-crawler-access-checker`) Actor

Check which AI crawlers and agents a website lets in: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and more. Detects bot walls (Cloudflare, Akamai), llms.txt, sitemap and JSON-LD. Bulk audits for SEO and GEO. $5 per 1,000 sites.

- **URL**: https://apify.com/fast_data_asap/ai-crawler-access-checker.md
- **Developed by:** [Fast Data](https://apify.com/fast_data_asap) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 website checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler Checker – Can GPTBot, ClaudeBot & AI Agents Access Your Site?

Which AI systems can actually read a website? This Actor checks **robots.txt rules for 17 AI crawlers, AI agents and search bots**, detects bot walls (HTTP 403 for automated clients) and looks for llms.txt, sitemap and structured data – in bulk.

### Who uses it

- **SEO / GEO agencies:** audit AI visibility for clients and prospects in one run.
- **E-commerce teams:** make sure ChatGPT and AI shopping agents can read your product pages.
- **Researchers & journalists:** measure how a whole market treats AI crawlers.

### What you get

- Status per agent: `allowed`, `partially_blocked` (e.g. product paths) or `blocked`, plus the matching rule group
- Agents: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Googlebot, Bingbot, Applebot, Applebot-Extended, meta-externalagent, Bytespider, CCBot, Amazonbot
- `aiAgentsBlocked` list and count, `searchEnginesBlocked`
- Homepage HTTP status for bots (detects Cloudflare/Akamai bot walls)
- `llms.txt` present, sitemap URL, JSON-LD types on the homepage
- Shop platform detection

### How to use it

1. Click **Try for free** (or Start).
2. Paste your list into the input (see the example below).
3. Run it and download the results as JSON, CSV or Excel, or get them via API.

#### Example input

```json
{
  "shops": [
    "example.com",
    "another-shop.de"
  ]
}
```

robots.txt shows the stated policy; the homepage check shows whether bots are actually let in.

#### Example output (shortened, illustrative values)

```json
{
  "shop": "https://another-shop.de",
  "platform": "Shopify",
  "robotsTxtFound": true,
  "aiAgentsBlocked": [
    "GPTBot",
    "CCBot"
  ],
  "aiAgentsBlockedCount": 2,
  "searchEnginesBlocked": [],
  "homepageHttpStatus": 200,
  "homepageReachableForBots": true,
  "llmsTxtFound": false,
  "homepageJsonLdTypes": [
    "Organization",
    "WebSite"
  ]
}
```

### Pricing

Pay only for results: **$5.00 per 1,000 sites** (pay per event, no monthly fee). Apify's free plan includes monthly credits, so you can test it for free.

| Volume | Price |
|---|---|
| 100 websites | $0.50 |
| 1,000 websites | $5.00 |

### Use it via API, n8n, Make or as an AI tool

- **API (one call, results back):**

```bash
curl -X POST "https://api.apify.com/v2/acts/fast_data_asap~ai-crawler-access-checker/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"shops": ["example.com", "another-shop.de"]}'
```

- **n8n / Make / Zapier:** use the Apify integration (Run Actor → Get dataset items) or an HTTP request with the URL above.
- **Claude, ChatGPT, Cursor (MCP):** add `https://mcp.apify.com?tools=fast_data_asap/ai-crawler-access-checker` as an MCP server, then just ask in plain language.
- **RapidAPI:** also available as an endpoint of the [DACH E-Commerce Data API](https://rapidapi.com/fastdataapi-fastdataapi-default/api/dach-e-commerce-data-api).

### FAQ

**What did you find on 100 German shops?**\
In our test of 100 large DACH shops (Nov 2026), only 9 blocked AI agents in robots.txt, but 12 returned HTTP 403 to automated clients, and 33 had an llms.txt (20 of 24 Shopify stores). Write-up on dev.to (fast_data_asap).

**Does "allowed" mean the AI can read my site?**\
Only if the homepage check also passes. A bot wall (403) blocks crawlers even when robots.txt allows them.

**How do I add an llms.txt?**\
For Shopify stores use the Shopify llms.txt Generator.

### Related Fast Data Actors

- [Shopify llms.txt Generator](https://apify.com/fast_data_asap/shopify-llms-txt-generator)
- [German Shop Compliance Checker](https://apify.com/fast_data_asap/german-shop-compliance-checker)
- [DACH Shop Lead Profiler](https://apify.com/fast_data_asap/dach-shop-lead-profiler)
- [All Fast Data Actors](https://apify.com/fast_data_asap)

### 🇩🇪 Auf Deutsch: KI-Crawler-Check für Websites

Prüft, ob GPTBot, ClaudeBot, PerplexityBot, Google-Extended und andere KI-Crawler und KI-Agenten eine Website lesen dürfen (robots.txt) und ob sie technisch überhaupt durchkommen (Bot-Sperre/403). Dazu: llms.txt vorhanden?, Sitemap, strukturierte Daten. Für SEO-/GEO-Agenturen und Shops, die in ChatGPT & Co. sichtbar sein wollen.

*Stichworte: GPTBot blockieren prüfen, KI Crawler robots.txt Check, Sichtbarkeit in ChatGPT prüfen.*

***

Built and maintained by **Fast Data**. Questions or a missing feature? Open an issue on this page.

# Actor input Schema

## `shops` (type: `array`):

Domains or homepages to check.

## `useApifyProxy` (type: `boolean`):

Route requests through Apify Proxy (helps with blocking).

## `timeoutSecs` (type: `integer`):

Timeout per HTTP request.

## Actor input object example

```json
{
  "shops": [
    "ankerkraut.de",
    "zalando.de"
  ],
  "useApifyProxy": false,
  "timeoutSecs": 20
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "shops": [
        "ankerkraut.de",
        "zalando.de"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("fast_data_asap/ai-crawler-access-checker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "shops": [
        "ankerkraut.de",
        "zalando.de",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("fast_data_asap/ai-crawler-access-checker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "shops": [
    "ankerkraut.de",
    "zalando.de"
  ]
}' |
apify call fast_data_asap/ai-crawler-access-checker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fast_data_asap/ai-crawler-access-checker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0uQbiBpL4NjP8C7ts/builds/ufm6gdCME0RVihT2G/openapi.json