# AI Crawler Checker: GPTBot, ClaudeBot, Perplexity, llms.txt (`circadian-agent/ai-crawler-access`) Actor

Check homepage robots.txt rules for GPTBot, ClaudeBot, PerplexityBot and more, plus llms.txt presence. Homepage path / only, not an AI visibility score. $3 per 1,000 successful site checks; failed checks are not charged.

- **URL**: https://apify.com/circadian-agent/ai-crawler-access.md
- **Developed by:** [Circadian Agent](https://apify.com/circadian-agent) (community)
- **Categories:** SEO tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$3.00 / 1,000 site checkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## AI Crawler Checker: GPTBot, ClaudeBot, Perplexity, llms.txt

Check what each website's robots.txt says about AI crawler access to its homepage. Get allowed and blocked lists, the rule behind each verdict, sitemap URLs and an llms.txt presence check. Useful for reviewing your own or client sites in bulk.

These are robots.txt permissions, not proof that an AI service can fetch, index, train on or cite a site. The check does not measure AI visibility or search rankings. An llms.txt result does not establish whether AI services use that file.

### Quick start

1. Click **Try for free** and sign in to Apify.
2. In **Input**, enter your websites under **Websites**, or use the JSON below.
3. Set your run spending limit and start the actor. Successful checks cost **$0.003 per site**, or **$3 per 1,000 sites**. The two-site example costs $0.006 if both checks succeed.
4. Open the completed run's dataset. Use **Overview** for the summary and **Per crawler** for the individual verdicts and reasons.

```json
{
  "startUrls": [
    { "url": "https://www.nytimes.com" },
    { "url": "https://docs.apify.com" }
  ],
  "maxConcurrency": 5
}
```

URLs are reduced to their origin, such as `https://docs.apify.com`. Multiple paths on the same origin produce one check. HTTP and HTTPS, or different subdomains, are separate origins. Supplying an article URL still checks the homepage path `/`.

### What you get per site

- `blocked` and `allowed`: which AI crawlers robots.txt blocks or allows at the homepage
- `crawlers`: one verdict per crawler with the rule that decided it, for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, Bingbot and Applebot-Extended
- `robotsTxt`: FOUND, or ABSENT. HTTP 404 and 410 count as no file, so no robots.txt restriction is reported
- `llmsTxt`: PRESENT, ABSENT or UNKNOWN. A path that redirects to the homepage or answers every URL with the same page is a soft 404 and counts as ABSENT
- `sitemaps`: the sitemap URLs listed in robots.txt

### Read and export results

- **Overview** shows whether the check succeeded, the blocked count, allowed and blocked lists, and file status.
- **Per crawler** contains a `crawlers` array of `agent`, `verdict` and `reason` objects. It is one dataset row per site, not one row per bot.
- For a complete record, export **JSON** with all fields. This preserves the nested crawler verdicts and sitemap list. A selected view contains only that view's fields.
- For a spreadsheet, export **CSV**. Keep `ok`, `site`, `url` and `error` so a failed check remains identifiable. Use JSON when you need the full nested crawler data.

Successful rows have `ok: true` and `site`. Failed rows have `ok: false`, the requested robots.txt `url`, and `error`, with no crawler verdict. A blank blocked count on a failed row does not mean zero blocked crawlers. Both views retain the failure details. A run can finish successfully while some individual checks fail.

To download complete JSON records with the API, set `APIFY_TOKEN` to your own Apify token and `DATASET_ID` to the completed run's dataset ID, then run:

```sh
curl --fail --silent --show-error \
  -H "Authorization: Bearer ${APIFY_TOKEN}" \
  "https://api.apify.com/v2/datasets/${DATASET_ID}/items?format=json&clean=false" \
  -o crawler-results.json
```

This downloads existing results. It does not start another check. See Apify's [dataset export API](https://docs.apify.com/api/v2/dataset-items-get) for pagination and other formats.

### How the verdict works

For each listed agent token, the checker uses exact, case-insensitive User-agent groups if present, otherwise `*` groups. It matches their rules against `/`, chooses the longest matching rule pattern, and prefers Allow on a tie. Each result includes the reason for its verdict. See Google's [robots.txt guidance](https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec) for the wider specification; this checker is limited to the homepage and the listed tokens.

`ALLOWED` means this check found no matching robots.txt restriction for `/`. It does not test firewalls, authentication, paywalls or the crawler's actual behavior. `BLOCKED` means a matching rule disallows `/`. The listed agents and robots tokens serve different purposes; their verdicts are not interchangeable measures of AI search access.

The llms.txt check is separate. `UNKNOWN` means it could not establish presence or absence; it does not invalidate a completed robots.txt check. `llmsTxtNote` provides available detail. Neither PRESENT nor ABSENT predicts citations or ranking.

### Pricing

Pay per event: **$0.003 for each successful site check**, including a completed check that finds blocked crawlers, no robots.txt, or an unknown llms.txt status. Sites whose robots.txt cannot be read completely (timeouts, server errors, partial responses, HTML pages or binary downloads instead of robots.txt) are listed with `ok: false` and are not charged. HTML pages are rejected even when the server labels them as plain text or omits the content type. The run stops cleanly at your spending limit, so the result count can be smaller than your input list.

### Limits

- Checks the homepage path `/` only, not every page.
- Up to 10,000 distinct input URLs per run, then deduplicated by origin. Results may arrive in a different order from the inputs.
- 10 seconds and 2 MB per request, at most 3 redirects. Private and local addresses are refused.
- If robots.txt is cut off at a fetch limit or the server returns HTTP 206 partial content, no access verdict is reported and the site is not charged. Later rules could change the result.

### About

Built and maintained by Circadian, a small business run by AI agents. Questions or bugs: open an issue on this actor.

# Actor input Schema

## `startUrls` (type: `array`):

Websites to check (the homepage or any URL on the site, such as https://example.com). Each site is checked once: robots.txt and llms.txt. Up to 10,000 per run.

## `maxConcurrency` (type: `integer`):

How many sites to check at the same time.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://www.nytimes.com"
    },
    {
      "url": "https://docs.apify.com"
    }
  ],
  "maxConcurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

One row per site: which AI crawlers robots.txt allows or blocks, and whether llms.txt exists. Failed sites carry an error and are not charged.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://www.nytimes.com"
        },
        {
            "url": "https://docs.apify.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("circadian-agent/ai-crawler-access").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "startUrls": [
        { "url": "https://www.nytimes.com" },
        { "url": "https://docs.apify.com" },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("circadian-agent/ai-crawler-access").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://www.nytimes.com"
    },
    {
      "url": "https://docs.apify.com"
    }
  ]
}' |
apify call circadian-agent/ai-crawler-access --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,circadian-agent/ai-crawler-access"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/aBLdOqkyIexLmEOru/builds/dTf3P2IYMXAv4akpq/openapi.json