# AI Crawler Access Auditor (`kingii98/ai-crawler-access-auditor`) Actor

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.

- **URL**: https://apify.com/kingii98/ai-crawler-access-auditor.md
- **Developed by:** [kingii98](https://apify.com/kingii98) (community)
- **Categories:** SEO tools, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$3.00 / 1,000 site auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## AI Crawler Access Auditor

Audit how a website treats AI crawlers. Given public website domains or root URLs, the Actor fetches `robots.txt`, `llms.txt`, and the sitemaps declared in robots.txt — safely, with every redirect target validated as a public address — and deterministically evaluates whether each named AI crawler user agent may access each test path. Every decision cites the exact matched `Allow`/`Disallow` rule and the robots group it came from. There is no opaque score and no AI-generated recommendation: you get the rules, the match, and the fetch statuses.

The package reuses the SSRF-safe HTTP and URL validation core of the `sitemap_health_monitor` Actor in this repository; that package and its tests are unchanged.

### What it checks

- `robots.txt` fetch status and content, evaluated per crawler per test path with RFC 9309 semantics: case-insensitive substring user-agent matching, most-specific group selection, merging of equally specific groups, `*` wildcards, `$` end anchors, longest-rule match, and allow-on-tie precedence
- The exact matched rule (`matchedRule`, `matchedRuleType`) and the user-agent token of the selected group (`matchedUserAgent`)
- `llms.txt` presence and fetch status
- Sitemap declarations from robots.txt and their reachability (bounded probe of the first 10 declarations)

Default crawler user agents: `GPTBot`, `ChatGPT-User`, `OAI-SearchBot`, `ClaudeBot`, `Claude-SearchBot`, `PerplexityBot`, `Google-Extended`, `CCBot`. You can replace them with a custom list.

### Input

```json
{
  "startUrls": ["example.com", "https://docs.example.org"],
  "crawlerUserAgents": ["GPTBot", "ClaudeBot"],
  "testPaths": ["/", "/pricing"],
  "maxSites": 100,
  "concurrency": 10,
  "timeoutSecs": 20,
  "maxRedirects": 5
}
```

| Field | Description |
|---|---|
| `startUrls` | 1-100 public website domains or root URLs. Each entry is normalized to its origin (`https://example.com/blog` becomes `https://example.com`); a missing scheme defaults to `https`. Duplicates are removed. Required. |
| `crawlerUserAgents` | Optional 1-30 crawler user-agent tokens. Defaults to the eight AI crawlers listed above. Tokens are 1-64 characters of letters, numbers, dot, underscore, hyphen; duplicates (case-insensitive) are removed. |
| `testPaths` | Optional 1-10 site-relative paths (starting with `/`, up to 200 characters) evaluated for every crawler. Default `["/"]`. |
| `maxSites` | Hard cap on distinct sites audited. Default 100; range 1-100. Inputs with more distinct sites than `maxSites` are rejected before any network work. |
| `concurrency` | Concurrent site audits. Default 10; range 1-30. |
| `timeoutSecs` | Per-request timeout. Default 20 seconds; range 2-60. |
| `maxRedirects` | Maximum redirect hops followed per fetch. Default 5; range 0-5. Every redirect target is validated as public before it is followed. |

URLs with credentials, non-HTTP(S) schemes, empty hosts, or targets resolving to non-public addresses are rejected before or during the run.

### Output

Every run writes one summary record, one `site-result` record per audited site, and one `crawler-result` record per site × crawler × test path to the default dataset.

Summary:

```json
{
  "recordType": "summary",
  "checkedAt": "2026-08-07T09:15:00+00:00",
  "siteCount": 2,
  "crawlerCount": 8,
  "pathCount": 1,
  "checkCount": 16,
  "allowedCount": 10,
  "disallowedCount": 5,
  "unknownCount": 1,
  "sitesWithRobots": 2,
  "sitesWithLlmsTxt": 1,
  "sitesWithReachableSitemap": 1,
  "errors": 1
}
```

Site result:

```json
{
  "recordType": "site-result",
  "siteUrl": "https://example.com",
  "robotsStatus": "ok",
  "robotsPresent": true,
  "llmsTxtStatus": "ok",
  "llmsTxtPresent": true,
  "sitemapUrls": ["https://example.com/sitemap.xml"],
  "reachableSitemapCount": 1,
  "issueCodes": [],
  "error": null,
  "checkedAt": "2026-08-07T09:15:00+00:00"
}
```

Crawler result:

```json
{
  "recordType": "crawler-result",
  "siteUrl": "https://example.com",
  "crawler": "GPTBot",
  "path": "/",
  "allowed": false,
  "matchedRule": "Disallow: /",
  "matchedRuleType": "disallow",
  "matchedUserAgent": "gptbot",
  "robotsStatus": "ok",
  "robotsPresent": true,
  "llmsTxtStatus": "ok",
  "llmsTxtPresent": true,
  "sitemapUrls": ["https://example.com/sitemap.xml"],
  "reachableSitemapCount": 1,
  "issueCodes": [],
  "error": null,
  "checkedAt": "2026-08-07T09:15:00+00:00"
}
```

`robotsStatus` and `llmsTxtStatus` are one of `ok`, `not-found`, `blocked` (robots.txt only, HTTP 401/403), or `error` (HTTP 5xx, 429, redirect cap, or fetch failure). `robotsPresent` is true only when a robots.txt document was successfully retrieved.

Access decisions follow the robots.txt fetch state:

- `ok` — `allowed` is the deterministic rule evaluation, with `matchedRule`, `matchedRuleType`, and `matchedUserAgent` set when a rule matched.
- `not-found` (404 and other 4xx) — `allowed` is `true` with no matched rule: there are no robots restrictions, stated explicitly via `robotsStatus`.
- `blocked` (401/403) — `allowed` is conservatively `false` with issue `robots-access-denied`.
- `error` (5xx, 429, or fetch failure) — `allowed` is `null` (unknown). The Actor never fabricates an allowed or blocked verdict when the rules could not be read.

Issue codes: `robots-not-found`, `robots-access-denied`, `robots-unreachable`, `llms-txt-error`, `sitemap-unreachable`.

### Pricing

The Actor uses Apify pay-per-event pricing with the `site-audited` charge event. When monetization is enabled, users are charged **$0.003 per site audited** ($3 per 1,000 sites). One `site-audited` event covers one site: its robots.txt, llms.txt, bounded sitemap reachability probes, and the deterministic evaluation of every crawler × test path.

Apify platform usage (compute units and other resources consumed by the run) may still be shown to users according to their plan and Apify's pricing rules, as described in the Actor's listing.

The Actor respects the run's maximum total charge: before any network work it trims the site list to the chargeable prefix; if no site can be charged, it stops with a clear error.

Final pricing is configured in the Apify Store listing and may change subject to Apify's pricing-change notice rules.

### Security and privacy

- Only public HTTP(S) targets are accepted.
- URL credentials, localhost, and non-public, loopback, link-local, multicast, unspecified, or reserved addresses are rejected.
- Every redirect target is resolved and validated before it is followed; a redirect to a private address fails that fetch with an error state instead of being fetched.
- Site counts, crawler counts, path counts, concurrency, redirects, response bytes, sitemap probes, and timeouts are all capped before or during network work.
- The Actor does not use a browser, proxy, LLM, external database, or third-party analytics service.
- Each run is stateless; results live only in the run's default dataset, subject to the retention and access settings of the Apify account running the Actor.

Do not place secrets, private URLs, or personal data in any input field.

### Limitations

- Access evaluation reflects robots.txt rules as served at audit time; it does not execute JavaScript, render pages, or verify crawler behavior in practice.
- Only sitemaps declared in robots.txt are probed; a missing or unreadable robots.txt yields no sitemap declarations.
- At most 20 sitemap declarations are recorded and the first 10 are probed for reachability.
- Network failures and rate limits are reported as per-site error states; they are not automatically retried indefinitely.
- The Actor does not send notifications itself. Use Apify schedules, webhooks, or an automation platform.

### Support

For reproducible issues, open an issue from the Actor page and include the Apify run ID, sanitized input, expected result, and affected public URL. Do not include API tokens or private data.

This Actor reports robots.txt access rules; it does not provide legal advice, crawler-compliance guarantees, or uptime guarantees.

# Actor input Schema

## `startUrls` (type: `array`):

Public website domains or root URLs to audit (e.g. "example.com" or "https://example.com"). Each entry is normalized to its origin; robots.txt, llms.txt, and declared sitemaps are fetched from there.

## `crawlerUserAgents` (type: `array`):

AI crawler user-agent tokens to evaluate. Defaults to GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, and CCBot.

## `testPaths` (type: `array`):

Site-relative paths to evaluate against robots.txt rules for every crawler. Defaults to the root path.

## `maxSites` (type: `integer`):

Hard cap on the number of distinct sites audited per run.

## `concurrency` (type: `integer`):

Maximum number of sites audited concurrently.

## `timeoutSecs` (type: `integer`):

Per-request timeout applied to robots.txt, llms.txt, and sitemap fetches.

## `maxRedirects` (type: `integer`):

Maximum redirect hops followed per fetch. Every redirect target is validated as a public address before it is followed.

## Actor input object example

```json
{
  "startUrls": [
    "https://example.com/"
  ],
  "testPaths": [
    "/"
  ],
  "maxSites": 100,
  "concurrency": 10,
  "timeoutSecs": 20,
  "maxRedirects": 5
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("kingii98/ai-crawler-access-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("kingii98/ai-crawler-access-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call kingii98/ai-crawler-access-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kingii98/ai-crawler-access-auditor"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/UawiTdQ81oRMxxbnz/builds/QHc33d9qRFnSrmAFZ/openapi.json