# Robots.txt and AI Crawler Auditor (`jetlaunch/robots-ai-crawl-auditor`) Actor

Audit robots.txt, AI crawler access, sitemaps, llms.txt, security.txt and ads.txt for a list of domains.

- **URL**: https://apify.com/jetlaunch/robots-ai-crawl-auditor.md
- **Developed by:** [Chris O'Byrne](https://apify.com/jetlaunch) (community)
- **Categories:** SEO tools, Developer tools, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 domain audits

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### Robots.txt and AI Crawler Auditor

This Actor audits a list of domains and tells you, in one table, **who is allowed to crawl each site and what the site publishes for crawlers**. For every domain it reads only public well-known files: `robots.txt`, the sitemaps listed there (or `/sitemap.xml`), `llms.txt`, `/.well-known/security.txt` and `ads.txt`. It does not load any web page content.

The robots.txt rules are evaluated the way [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309) describes: group selection by user-agent token, longest matching rule wins, Allow wins a tie, `*` and `$` wildcards, percent-encoding normalization, and the rules for missing or failing files. You get an allowed or blocked verdict for **15 named bots plus the wildcard agent**, tested against a path you choose (default `/`).

#### Who is it for

- Publishers and marketers who want to check whether GPTBot, ClaudeBot, PerplexityBot and other AI crawlers can reach their site, or confirm that a block is working.
- SEO and GEO agencies that audit many client domains and need a repeatable, scheduled check.
- Developers and researchers who want machine-readable robots.txt and sitemap facts through the Apify API.

#### What it checks

| Check | Result fields |
|---|---|
| robots.txt (RFC 9309 parsing) | `robots_found`, `robots_status`, `robots_mode`, `rules_per_agent`, `ai_bots_blocked`, `ai_bots_allowed`, `search_bots_blocked` |
| Sitemaps | `sitemap_urls_listed`, `sitemap_count`, `sitemap_checks`, plus `url_count`, `lastmod_min`, `lastmod_max` in deep mode |
| llms.txt | `llms_txt_present`, `llms_txt_title` |
| security.txt (RFC 9116) | `security_txt_present`, `security_txt_fields` |
| ads.txt | `ads_txt_present` |
| Findings and score | `issues`, `score`, `score_breakdown` |

Bots tested: GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Google-Extended, Googlebot, Bingbot, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, and the wildcard agent `*`. Each entry carries a short purpose label (for example `ai-training`, `ai-search`, `search-engine`). The labels are descriptive and the vendors' own documentation is the authority. Google-Extended and Applebot-Extended are robots.txt control tokens, not separate crawlers. You can add up to 20 more tokens in the input.

#### How to use it

1. Open the Actor and enter your domains, one per line (a full URL is reduced to its host).
2. Optionally change the **test path**, for example `/blog/post` to test one section.
3. Switch on **Crawl sitemaps in depth** if you want URL counts and lastmod dates.
4. Click Start. The results appear in the Overview table, and the Issues view lists the findings in plain language.

#### Input

| Field | Meaning |
|---|---|
| `domains` | Up to 1,000 domains or URLs. |
| `testPath` | Path used for every verdict. A query string is allowed. Default `/`. |
| `deepSitemaps` | Follow nested sitemap indexes and count URLs. Default off. |
| `maxSitemapUrls` | Stop counting at this many URLs per domain, at most 50,000. |
| `maxSitemapFiles` | Read at most this many sitemap files per domain. Default 25. |
| `customUserAgents` | Extra user-agent tokens to test. |

Example input:

```json
{
    "domains": ["apify.com", "nytimes.com", "bbc.co.uk"],
    "testPath": "/",
    "deepSitemaps": false
}
```

#### Output

One record per domain. This abridged record was produced by a real run for apify.com with deep sitemaps on, capped at 2,000 URLs:

```json
{
    "domain": "apify.com",
    "status": "ok",
    "test_path": "/",
    "robots_found": true,
    "robots_status": 200,
    "robots_mode": "parsed",
    "rules_per_agent": [
        { "agent": "GPTBot", "purpose": "ai-training", "verdict": "allowed", "allowed": true, "matched_group": "wildcard", "matched_rule": "Allow: /", "crawl_delay": null }
    ],
    "ai_bots_blocked": [],
    "search_bots_blocked": [],
    "sitemap_urls_listed": ["https://apify.com/sitemap.xml"],
    "sitemap_count": 3,
    "url_count": 2000,
    "url_count_capped": true,
    "llms_txt_present": true,
    "security_txt_present": true,
    "security_txt_fields": { "Contact": ["mailto:***@apify.com"], "Expires": ["2035-01-01T00:00:00.000Z"] },
    "ads_txt_present": false,
    "issues": ["INFO: None of the 13 AI crawlers tested are named in robots.txt, so each one follows the wildcard group (which allows them)."],
    "score": 83
}
```

A blocked verdict from the same kind of run (nytimes.com, GPTBot) looks like this:

```json
{ "agent": "GPTBot", "purpose": "ai-training", "verdict": "blocked", "allowed": false, "matched_group": "named", "matched_rule": "Disallow: /", "crawl_delay": null }
```

`status` is `ok`, `robots_unavailable` (server error or refusal), `unreachable`, `invalid_domain`, `skipped_budget` or `error`. Only `ok` records are charged.

#### How robots.txt problems are handled

- 2xx: the file is parsed (the first 512 KiB, RFC 9309 requires at least 500 KiB).
- 404 and 410: treated as "no rules", so everything is allowed. This is reported as a finding.
- 5xx or network failure after retries: RFC 9309 says to assume the whole site is off limits, and the verdicts show that. The record is `robots_unavailable` and is not charged.
- Any other 4xx (for example 401, 403, 418, 429): the server refused this Actor, often because of bot protection. No verdicts are produced and the domain is not charged.
- An HTML page served at `/robots.txt` is reported as a soft 404.

#### Score

The score from 0 to 100 measures hygiene and readiness. It does not judge whether blocking AI crawlers is right for you.

| Part | Points |
|---|---|
| robots.txt present and clean | 20 |
| Sitemap declared in robots.txt (12, or 6 if only found at /sitemap.xml) and reachable (8) | 20 |
| AI policy clarity: a wildcard group (8) plus the share of the 13 AI bots named explicitly (up to 17) | 25 |
| Googlebot and Bingbot not blocked on the test path | 15 |
| llms.txt present | 10 |
| security.txt present with Contact and a valid Expires | 10 |

#### Pricing

This Actor uses pay-per-event pricing. Platform usage is included in the event price.

| Event | Price | When it is charged |
|---|---|---|
| `audit-domain` | $0.01 | Once per domain with a definitive robots.txt result (found, or a 404 or 410). Covers robots.txt verdicts, sitemap availability, llms.txt, security.txt and ads.txt. |
| `sitemap-deep` | $0.03 | Once per domain, only when **Crawl sitemaps in depth** is on and at least one sitemap file was read. |

You are not charged for invalid domains, unreachable domains, refused or failing robots.txt requests, or domains skipped because your maximum cost per run was reached.

#### Politeness and data source

The Actor sends at most one request per second to any host and identifies itself in the User-Agent: `JetlaunchRobotsAuditor/1.0 (+https://apify.com/jetlaunch/robots-ai-crawl-auditor; contact chris@jetlaunch.net)`. It retries politely on 429 and 5xx, never solves CAPTCHAs, and does not try to get around blocks. The files it reads are the ones sites publish for automated clients, so it fetches them (and sitemaps the site lists) even if the site's own robots.txt would disallow its user agent. The one exception is the guessed `/sitemap.xml`, which is skipped when robots.txt disallows it. Only sitemaps on the same site are followed. The data in your results is whatever the audited sites publish, and any rights in it belong to those sites.

Email addresses and phone numbers in `Contact` lines of security.txt are redacted, so the output contains no individual contact details.

#### FAQ

**Does it check whether a bot really obeys robots.txt?** No. It reads what a site publishes, not what any crawler does.

**Why is a verdict "allowed" when the site has no robots.txt?** Under RFC 9309 a missing file means no restrictions.

**Why do the sitemap numbers differ between modes?** Without deep mode, `sitemap_count` is the number of listed sitemaps that responded with a valid sitemap, and `url_count` is `null`. With deep mode it counts every file read, including nested ones, and the URL entries inside them. Entries are counted, not de-duplicated.

**Does Content-Signal or meta robots matter?** Content-Signal lines are reported but not used for verdicts. Meta robots tags and X-Robots-Tag headers are not checked, because that would need page requests.

**Can I run it on a schedule or call it from code?** Yes, use Apify schedules or the API.

#### Limitations

- A site's firewall may treat this Actor differently from a real crawler, so a refusal does not prove how Googlebot or GPTBot is treated.
- Verdicts follow RFC 9309. Some crawlers interpret edge cases differently, and some ignore directives such as Crawl-delay.
- Sitemap crawling stops at your limits (default 25 files, at most 50,000 URLs) and sitemaps on other domains are not followed.
- Bot names and labels may change over time as vendors update their crawlers.

This Actor is not affiliated with or endorsed by any of the companies whose crawler names appear in its results.

#### Support

Found a problem or want another file or bot added? Open a ticket in the **Issues** tab of this Actor.

# Actor input Schema

## `domains` (type: `array`):

Domains to audit, one per line, for example example.com. A full URL is accepted and reduced to its host. Up to 1,000 per run.

## `testPath` (type: `string`):

The URL path that every bot is tested against, for example / for the whole site or /blog/my-post to test one page. A query string is allowed.

## `deepSitemaps` (type: `boolean`):

Follow nested sitemap indexes and count URLs and the oldest and newest lastmod dates. Without it, the Actor only checks that each listed sitemap responds. Billed as a separate 'sitemap-deep' event per domain.

## `maxSitemapUrls` (type: `integer`):

Stop counting after this many sitemap URLs per domain. Applies only when deep sitemap crawling is on. The upper limit is 50,000.

## `maxSitemapFiles` (type: `integer`):

Stop after reading this many sitemap files per domain. Applies only when deep sitemap crawling is on. The Actor makes at most one request per second to each host, so large numbers take longer.

## `customUserAgents` (type: `array`):

Optional. Additional robots.txt user-agent tokens to test, for example Applebot or Diffbot. Letters, digits, hyphens and underscores only. Up to 20.

## Actor input object example

```json
{
  "domains": [
    "apify.com",
    "nytimes.com",
    "bbc.co.uk"
  ],
  "testPath": "/",
  "deepSitemaps": false,
  "maxSitemapUrls": 50000,
  "maxSitemapFiles": 25
}
```

# Actor output Schema

## `results` (type: `string`):

One record per domain with bot verdicts, sitemap stats, file checks, issues and score.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "apify.com",
        "nytimes.com",
        "bbc.co.uk"
    ],
    "testPath": "/"
};

// Run the Actor and wait for it to finish
const run = await client.actor("jetlaunch/robots-ai-crawl-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "domains": [
        "apify.com",
        "nytimes.com",
        "bbc.co.uk",
    ],
    "testPath": "/",
}

# Run the Actor and wait for it to finish
run = client.actor("jetlaunch/robots-ai-crawl-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "apify.com",
    "nytimes.com",
    "bbc.co.uk"
  ],
  "testPath": "/"
}' |
apify call jetlaunch/robots-ai-crawl-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,jetlaunch/robots-ai-crawl-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/v4V2MjZY2SH1VXa3N/builds/B1CRJuzUHMoxvdcjO/openapi.json