# Crawlability & AI Crawler Access Auditor (`lwsdjfls/crawlability-ai-access-auditor`) Actor

AI crawler access audit and robots.txt checker for any domain: rules per crawler (GPTBot, ClaudeBot, CCBot, Google-Extended and 25 more), Content-Signal and Content-Usage, sitemaps with URL counts, llms.txt, meta robots on sample pages, TDM opt-out detection and the anti-bot service in front.

- **URL**: https://apify.com/lwsdjfls/crawlability-ai-access-auditor.md
- **Developed by:** [Leyten Data](https://apify.com/lwsdjfls) (community)
- **Categories:** Developer tools, SEO tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 domain auditeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Crawlability & AI Crawler Access Auditor do?

An **AI crawler access** audit and robots.txt checker for any list of domains. One run tells you, per domain, what generic crawlers and every AI crawler may do (GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended and 25 more), whether the site has opted out of text and data mining in a machine-readable way, what the sitemaps cover, whether an **llms.txt** exists, what the pages say in meta robots and X-Robots-Tag, and which bot-management service (Cloudflare, Akamai, DataDome, …) answers the door. It is written for two readers at once.

#### If you crawl: is this target feasible, and is there an opt-out?

Before you build a scraper, or an agent that fetches pages, you want three answers: will the site let a crawler in, will the edge block a declared bot, and has the owner reserved text and data mining rights. The third one is the **TDM opt-out detection** that Article 4(3) of the DSM Directive (2019/790) and the EU AI Act's transparency obligations turn into a compliance question: a reservation that is machine-readable has to be honoured by anyone mining for AI training.

Per domain you get:

- `verdict`: `open`, `restricted` (opt-outs, AI blocks, crawl delays, a bot manager in front, or a robots.txt that could not be read), `blocked` (generic crawlers disallowed or a declared bot refused on the homepage) or `unreachable`
- `crawlers[]`: for `*`, your own tokens and 29 known crawlers: `allowed`, `partial`, `blocked`, `no-rules` (no group applies), `no-robots` (the site has no robots.txt) or `unknown` (robots.txt could not be read; `aiAccess.genericCrawlersAllowed` is then `false` because access is not confirmed, not because crawlers are blocked), which group decided (`own` or `wildcard`), the disallowed and allowed paths, `Crawl-delay`
- `tdm.level`: `explicit` (TDMRep `tdm-reservation` header, meta tag or `/.well-known/tdmrep.json`; a Cloudflare `Content-Signal: ai-train=no`; an IETF `Content-Usage: train-ai=n`), `robots-only` (training crawlers disallowed, nothing else), `terms-only` (a clause in the terms) or `none`, with every source and the passages that were found
- `antiBot`: the service recognised from headers, cookies and challenge pages, and whether any fetch in the audit was refused (401/403/429/503) or challenged: the audit runs as a declared bot from a data centre, which is exactly how your crawler would arrive
- `sitemaps`: declared and discovered files, URL counts and `lastmod` range, so you know the size and freshness of what you are about to fetch
- `summary.forScrapers`: the findings in plain sentences

#### If you own the site: can AI read it, and do you want it to?

The **GEO / AI visibility** question has two halves that get mixed up. Answer engines (ChatGPT search, Claude, Perplexity, DuckDuckGo AI, Amazon) need to *index* you to cite you; training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, CCBot) take a *copy* for models. Most robots.txt files were written before that distinction existed. Per domain you get:

- `aiAccess`: training crawlers allowed and blocked, AI search crawlers allowed and blocked, user-initiated fetchers blocked (ChatGPT-User, Claude-User, Perplexity-User: the ones that open your page when someone asks about you), the `Content-Signal` and `Content-Usage` preferences if you set any
- `llmsTxt` and `llmsFullTxt`: present or not, and whether the file is real (title, sections, links) or a soft 404
- `pages[]`: a sample of pages with `noindex`, `nofollow`, `noai`, `X-Robots-Tag`, canonical and TDM headers, so a stray `noindex` on a template shows up
- `tdm`: whether your opt-out exists in the form the AI Act expects (machine-readable) or only in the terms nobody's crawler reads
- `summary.forSiteOwners`: what an AI answer engine can and cannot do with your site today, in plain sentences

Run it in `monitor` mode on a schedule and you are told when a robots.txt change blocks or unblocks an AI crawler, when a TDM reservation appears, when a bot manager shows up in front of the site, or when an llms.txt is added.

#### What is checked, exactly

| Check | How |
| --- | --- |
| robots.txt | Fetched from the origin, parsed as RFC 9309 and Google read it: groups of `User-agent` lines, longest-match rules, `Allow` wins a tie, `*` and `$` in paths, `Crawl-delay`, `Sitemap`. A crawler follows its own group, else `*`. Like Google, only the first 500 KiB is read. Soft 404s (an HTML page at `/robots.txt`), 5xx and 429 are reported for what they do to crawlers |
| AI crawlers | 29 tokens with operator and purpose (training, AI search, user-initiated fetch, search, dataset, archive): OpenAI, Anthropic, Google, Microsoft, Common Crawl, Perplexity, Apple, ByteDance, Meta, Amazon, Cohere, Mistral, DuckDuckGo, You.com, Diffbot, Webz.io, Hive, Huawei, Internet Archive. Add your own with **Extra user-agent tokens** |
| Content signals | Cloudflare's `Content-Signal: search=…, ai-input=…, ai-train=…` (September 2025) and the IETF AIPREF `Content-Usage` rule and header (`train-ai=n`, `search=y`, optionally per path) |
| Sitemaps | Files declared in robots.txt, else `/sitemap.xml` and the usual alternatives; indexes followed breadth-first up to **Maximum sitemap files**; gzip handled; per file the URL count and `lastmod` range. Only files on the domain itself, its `www` twin or the host its homepage redirects to are fetched; sitemaps on other hosts are listed, not read |
| llms.txt | `/llms.txt` parsed (title, summary, sections, links), `/llms-full.txt` and `/ai.txt` presence |
| Pages | The homepage plus URLs spread over the sitemaps (or homepage links): meta robots and per-bot metas, `X-Robots-Tag`, `noindex`/`nofollow`/`noai`, canonical, `tdm-reservation` and `tdm-policy` in headers or meta, `Content-Usage` header |
| TDM reservation | `/.well-known/tdmrep.json`, the headers and meta tags above, robots.txt signals, and up to three terms, legal or AI-policy pages linked from the homepage, scanned in English, Dutch, German and French for passages on text and data mining, AI training, automated access and reservations, each marked prohibitive or not |
| Anti-bot | Cloudflare (incl. Turnstile), Akamai Bot Manager, Imperva, DataDome, HUMAN/PerimeterX, Kasada, F5/Shape, AWS WAF, Fastly, Vercel, Sucuri, Anubis, reCAPTCHA, hCaptcha, from server headers, cookies, script references and challenge pages; plus any refusal of the declared bot |

The audit identifies itself (`crawlability-ai-access-auditor/0.1; +https://apify.com/lwsdjfls/crawlability-ai-access-auditor`), makes at most three requests per second per domain, and never tries to get past a challenge: a refusal is the finding. It fetches robots.txt, sitemaps, a handful of pages and the well-known files; it does not crawl the site.

### Features

- **robots.txt checker** per crawler: RFC 9309 parsing, longest-match rules, `Allow` wins ties, wildcards, crawl delays, soft 404s
- **AI crawler access audit** for 29 known tokens (training, AI search, user-initiated fetchers) plus your own
- **TDM opt-out detection**: TDMRep headers, meta tags and `/.well-known/tdmrep.json`, Cloudflare Content-Signal, IETF Content-Usage, robots.txt signals and terms passages in four languages
- **llms.txt checker**: presence and validity of `llms.txt`, `llms-full.txt` and `ai.txt`
- **Sitemap inventory** with URL counts and `lastmod` ranges
- **Meta robots and X-Robots-Tag** on a page sample, canonical and TDM headers included
- **Anti-bot service detection** from public signatures, plus whether the declared bot was refused
- **Two summaries per domain**: one for scrapers, one for site owners
- **Monitor mode** for policy changes, with a webhook
- Runs through the Apify API, on a schedule, or from Make, Zapier and n8n

### Use cases

#### Scraper builders: feasibility check before building

Run your target list once. `blocked` and `antiBot.challenged` tell you where a declared bot is refused; `crawlers[].disallowed` shows which paths are off limits for `*` and for your own token; `sitemaps.totalUrls` and `lastmodMax` size the job; `tdm.level` tells you whether mining for AI purposes has been reserved.

```js
// npm install apify-client
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('lwsdjfls/crawlability-ai-access-auditor').call({ domains: targets, samplePages: 5 });
const { items } = await client.dataset(run.defaultDatasetId).listItems({ clean: true });
for (const { payload } of items) {
    const optOut = payload.tdm.level !== 'none';
    const refused = payload.verdict === 'blocked';
    console.log(payload.domain, payload.verdict, optOut ? `opt-out: ${payload.tdm.level}` : 'no opt-out', refused ? '' : `${payload.sitemaps.totalUrls} URLs in sitemaps`);
}
```

#### SEO and GEO agencies: AI-visibility audit for clients

Audit a client's domains and their competitors' in one run, hand over `summary.forSiteOwners` per domain, and schedule the `monitor` task so the next robots.txt edit that blocks Perplexity is caught the same week.

#### Data and legal teams: a compliance register of TDM reservations

Keep a dataset of the TDM status of every source in your data pipeline, with the passages and URLs as evidence, refreshed monthly. The Actor reports what the site publishes; whether a clause is a valid reservation under Article 4(3) of Directive 2019/790, or what the AI Act requires of a specific provider, depends on jurisdiction and facts. Take `tdm` to your counsel, with the sources and passages the record gives you. This is not legal advice.

### How to use it

1. Put the domains into **Domains**, bare like `example.com`, and press Start. One run covers robots.txt for every crawler it knows, the sitemaps and their URL counts, `llms.txt`, the well known files, meta robots on a sample of pages, and whichever anti-bot service sits in front of the site.
2. Read the `audit` view for the verdict per domain and the `crawlers` view for the part most people came for: the rule that applies to each individual crawler, so you can see at a glance whether GPTBot, ClaudeBot, CCBot and Google-Extended are allowed or blocked, and whether that was deliberate.
3. Give it a **State key** and schedule it monthly, or weekly if you also audit domains you do not control. The `changes` view then reports the day a robots.txt rule, a Content-Signal or a TDM reservation moves, which on a competitor's site is a strategy change and on your own site is usually an accident.

Turn on **Scan terms and legal pages for TDM clauses** when the legal position matters. A machine readable reservation under TDMRep is easy to check, but plenty of sites only reserve their rights in prose on a terms page, and that scan finds those too. Use **Pages to sample** to control how deep the per page check goes.

A domain that cannot be reached still produces a record with the reason, so a list never comes back shorter than you sent it. What this does not do is judge: it reports what a site declares and what a crawler would encounter. Whether a reservation is enforceable where you operate is a legal question, not a technical one.

```json
{
  "domains": ["nos.nl", "rijksoverheid.nl", "apify.com"],
  "checkTerms": true,
  "stateKey": "monthly-ai-access-audit"
}
```

### Input

| Field | What it does | Example |
| --- | --- | --- |
| `domains` | One per line: `example.com` or `https://www.example.com/`, up to 1,000. Subdomains are separate domains with their own robots.txt. | `["nos.nl", "rijksoverheid.nl", "anthropic.com"]` |
| `mode` | `audit` delivers the full audit of every domain; `monitor` delivers a baseline and afterwards only domains whose policy changed. | `"audit"` |
| `samplePages` | Pages fetched per domain for meta robots, X-Robots-Tag and TDM headers: the homepage plus URLs spread over the sitemaps. | `8` |
| `maxSitemaps` | Sitemap index and child files fetched per domain. | `25` |
| `checkTerms` | Follows up to three terms, legal or policy links from the homepage and reports passages about text and data mining, AI training and automated access. | `true` |
| `extraUserAgents` | Your own crawler tokens to evaluate against robots.txt. | `["MyCompanyBot"]` |
| `includeRobotsText` | Adds the robots.txt content (up to 20 kB) to each record. | `true` |
| `maxConcurrency` | Domains audited in parallel; each domain gets at most three requests per second regardless. Memory: sitemap files over 4 MB (up to 32 MB each) are downloaded at most two at a time across the whole run, and gzip sitemaps are unpacked one at a time, so a higher value does not multiply sitemap memory, and does not speed up those large downloads; smaller sitemap files (gzip ones included) and every other response are read up to 4 MB each, one per domain at a time, in parallel, and never wait for another domain's large download. | `5` |
| `webhookUrl` | Receives a POST with a summary after each run that delivered records; stored encrypted. | `https://hooks.example.com/crawlability` |
| `stateKey` | Separates independent monitors. Defaults to the task ID. | `client-portfolio` |
| `proxyConfiguration` | Off by default on purpose: the audit measures how a site answers a declared bot from a data centre. | `{ "useApifyProxy": false }` |

```json
{
  "domains": ["nos.nl", "www.rijksoverheid.nl", "www.cloudflare.com", "www.bol.com"],
  "samplePages": 8,
  "maxSitemaps": 25,
  "checkTerms": true,
  "extraUserAgents": ["MyCompanyBot"]
}
```

For a scheduled watch, save a task with `"mode": "monitor"`: the first run delivers a baseline, later runs only the domains whose policy changed, with a summary such as `nos.nl: no longer blocks ClaudeBot, Claude-SearchBot; TDM robots-only → explicit`. A run in which a domain does not answer is reported by its verdict (`restricted → unreachable`), not as blocks lifted or files removed, and a robots.txt that could not be read is not reported as blocks lifted or added.

### Output

One record per domain, with `verdict`, `crawlers[]`, `aiAccess`, `tdm`, `antiBot`, `sitemaps`, `llmsTxt`, `pages[]` and the two summaries described above. Below: nos.nl, the Dutch public broadcaster: every AI training crawler blocked in robots.txt, AI search crawlers blocked too, no machine-readable TDM reservation, no bot manager, sitemaps with dated URLs.

Three arrays are shortened in the example so the rest of this page stays readable; a real record carries them in full: `crawlers` holds a row per crawler checked (30 in this run, 4 shown), `robots.groups` a row per user-agent block in robots.txt (26 in this run, 2 shown) and `pages` a row per sampled page (3 in this run, 1 shown). `robots.text` is `null` here because this run had `includeRobotsText` off; with the default on it holds the file verbatim, up to 20 kB.

```json
{
  "envelopeVersion": 1,
  "id": "nos.nl",
  "scope": "domain:nos.nl",
  "changeType": "added",
  "sourceUrl": "https://nos.nl/robots.txt",
  "scrapedAt": "2026-09-15T22:46:47.370Z",
  "firstSeenAt": "2026-09-15T22:46:47.370Z",
  "hash": "a287ddffae101439829d3ca2012d49418bcf93aff45e2f85cf39b48eb1400406",
  "previousHash": null,
  "changes": [],
  "summary": "Added: nos.nl: restricted, blocks 8 AI training crawlers, TDM robots-only",
  "diffText": null,
  "payload": {
    "domain": "nos.nl",
    "origin": "https://nos.nl",
    "reachable": true,
    "verdict": "restricted",
    "summary": {
      "forScrapers": [
        "robots.txt allows generic crawlers on /; 7 path(s) disallowed for *: /hybrid/, /widget-embed/, /special/, /humans.txt, /api and 2 more.",
        "No bot-management service recognised; all fetched pages answered normally to a declared bot.",
        "No TDMRep, Content-Signal or Content-Usage reservation, but robots.txt blocks AI training crawlers (GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended and 3 more): treat as an opt-out for training.",
        "Sitemaps: 2 of 3 files read (cap), at least 174 URLs, last change 2026-09-15."
      ],
      "forSiteOwners": [
        "All listed AI training crawlers are blocked (GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended and 3 more).",
        "AI search crawlers blocked: PerplexityBot, DuckAssistBot, YouBot. The site will not appear in those AI answers.",
        "User-initiated fetches blocked: ChatGPT-User, Meta-ExternalFetcher. Assistants cannot open the site when a user asks about it.",
        "No llms.txt: no curated entry point for language models.",
        "Training crawlers are blocked in robots.txt only; a TDMRep header or Content-Signal would make the reservation explicit for every crawler."
      ]
    },
    "homepage": {
      "url": "https://nos.nl/",
      "finalUrl": "https://nos.nl/",
      "status": 200,
      "title": "NOS.nl - Nieuws, Sport en Evenementen | Nederlandse Omroep Stichting",
      "server": "nginx",
      "redirectedToOtherHost": false
    },
    "robots": {
      "url": "https://nos.nl/robots.txt",
      "status": 200,
      "present": true,
      "bytes": 1304,
      "groups": [
        {
          "agents": [
            "*"
          ],
          "rules": [
            {
              "type": "disallow",
              "path": "/hybrid/"
            },
            {
              "type": "disallow",
              "path": "/widget-embed/"
            },
            {
              "type": "disallow",
              "path": "/special/"
            },
            {
              "type": "disallow",
              "path": "/humans.txt"
            },
            {
              "type": "disallow",
              "path": "/api"
            },
            {
              "type": "disallow",
              "path": "/zoeken"
            },
            {
              "type": "disallow",
              "path": "/regio"
            }
          ],
          "crawlDelay": null,
          "contentSignal": null,
          "contentUsage": []
        },
        {
          "agents": [
            "CCBot"
          ],
          "rules": [
            {
              "type": "disallow",
              "path": "/"
            }
          ],
          "crawlDelay": null,
          "contentSignal": null,
          "contentUsage": []
        }
      ],
      "sitemaps": [
        "https://nos.nl/sitemap/index.xml",
        "https://nos.nl/sitemap/news.xml",
        "https://nos.nl/sitemap/livestreams.xml"
      ],
      "unknownLines": 0,
      "note": null,
      "text": null
    },
    "crawlers": [
      {
        "token": "*",
        "operator": "any crawler",
        "purpose": "generic",
        "status": "partial",
        "matchedBy": "own",
        "rootAllowed": true,
        "disallowed": [
          "/hybrid/",
          "/widget-embed/",
          "/special/",
          "/humans.txt",
          "/api",
          "/zoeken",
          "/regio"
        ],
        "allowed": [],
        "crawlDelay": null,
        "contentSignal": null,
        "contentUsage": []
      },
      {
        "token": "GPTBot",
        "operator": "OpenAI",
        "purpose": "ai-training",
        "status": "blocked",
        "matchedBy": "own",
        "rootAllowed": false,
        "disallowed": [
          "/"
        ],
        "allowed": [],
        "crawlDelay": null,
        "contentSignal": null,
        "contentUsage": []
      },
      {
        "token": "OAI-SearchBot",
        "operator": "OpenAI",
        "purpose": "ai-search",
        "status": "partial",
        "matchedBy": "wildcard",
        "rootAllowed": true,
        "disallowed": [
          "/hybrid/",
          "/widget-embed/",
          "/special/",
          "/humans.txt",
          "/api",
          "/zoeken",
          "/regio"
        ],
        "allowed": [],
        "crawlDelay": null,
        "contentSignal": null,
        "contentUsage": []
      },
      {
        "token": "ChatGPT-User",
        "operator": "OpenAI",
        "purpose": "ai-user",
        "status": "blocked",
        "matchedBy": "own",
        "rootAllowed": false,
        "disallowed": [
          "/"
        ],
        "allowed": [],
        "crawlDelay": null,
        "contentSignal": null,
        "contentUsage": []
      }
    ],
    "aiAccess": {
      "genericCrawlersAllowed": true,
      "trainingAllowed": [],
      "trainingBlocked": [
        "GPTBot",
        "ClaudeBot",
        "anthropic-ai",
        "Google-Extended",
        "Applebot-Extended",
        "Bytespider",
        "meta-externalagent",
        "cohere-ai"
      ],
      "searchAllowed": [
        "OAI-SearchBot",
        "Claude-SearchBot",
        "Amazonbot"
      ],
      "searchBlocked": [
        "PerplexityBot",
        "DuckAssistBot",
        "YouBot"
      ],
      "userFetchBlocked": [
        "ChatGPT-User",
        "Meta-ExternalFetcher"
      ],
      "contentSignal": null,
      "contentUsage": []
    },
    "sitemaps": {
      "declared": [
        "https://nos.nl/sitemap/index.xml",
        "https://nos.nl/sitemap/news.xml",
        "https://nos.nl/sitemap/livestreams.xml"
      ],
      "discovered": [],
      "fetched": [
        {
          "url": "https://nos.nl/sitemap/index.xml",
          "kind": "urlset",
          "status": 200,
          "urlCount": 54,
          "childCount": 0,
          "lastmodMin": "2026-09-15",
          "lastmodMax": "2026-09-15",
          "error": null
        },
        {
          "url": "https://nos.nl/sitemap/news.xml",
          "kind": "urlset",
          "status": 200,
          "urlCount": 120,
          "childCount": 0,
          "lastmodMin": null,
          "lastmodMax": null,
          "error": null
        }
      ],
      "listed": 3,
      "totalSitemaps": 2,
      "totalUrls": 174,
      "lastmodMin": "2026-09-15",
      "lastmodMax": "2026-09-15",
      "truncated": true
    },
    "llmsTxt": {
      "url": "https://nos.nl/llms.txt",
      "present": false,
      "status": 404,
      "bytes": 29630,
      "title": null,
      "summary": null,
      "sections": [],
      "linkCount": 0,
      "note": null
    },
    "llmsFullTxt": {
      "url": "https://nos.nl/llms-full.txt",
      "present": false,
      "status": 404,
      "bytes": 29631
    },
    "aiTxt": {
      "url": "https://nos.nl/ai.txt",
      "present": false,
      "status": 404,
      "bytes": 29631
    },
    "pages": [
      {
        "url": "https://nos.nl/",
        "status": 200,
        "title": "NOS.nl - Nieuws, Sport en Evenementen | Nederlandse Omroep Stichting",
        "xRobotsTag": null,
        "metaRobots": [
          {
            "name": "robots",
            "content": "max-image-preview:large"
          }
        ],
        "noindex": false,
        "nofollow": false,
        "noai": false,
        "tdmReservation": null,
        "tdmPolicy": null,
        "contentUsage": null,
        "canonical": "https://nos.nl/",
        "challenged": false,
        "error": null
      }
    ],
    "pageDirectives": {
      "sampled": 3,
      "fetched": 3,
      "noindex": 0,
      "noai": 0,
      "tdmReserved": 0,
      "challenged": 0
    },
    "tdm": {
      "level": "robots-only",
      "machineReadable": [
        {
          "kind": "robots",
          "url": "https://nos.nl/robots.txt",
          "detail": "robots.txt disallows AI training crawlers: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, cohere-ai"
        }
      ],
      "tdmrep": {
        "url": "https://nos.nl/.well-known/tdmrep.json",
        "present": false,
        "status": 404,
        "entries": [],
        "note": null
      },
      "termsPages": [],
      "termsReservation": false
    },
    "antiBot": {
      "services": [],
      "challenged": false,
      "challengedUrls": []
    },
    "requests": 11,
    "warnings": [],
    "checkedAt": "2026-09-15T22:46:47.370Z"
  }
}
```

### Pricing

| Event | Name | What it means | Price |
| --- | --- | --- | --- |
| `apify-actor-start` | Actor start | Charged by Apify when a run starts: once for a run of up to 1 GB of memory, and once more for every extra GB. It covers the first five seconds of compute. | $0.00005 per run up to 1 GB of memory ($0.05 per 1,000) |
| `domain-audited` **(primary)** | Domain audited | One domain audited: robots.txt per crawler (AI crawlers included), Content-Signal and Content-Usage rules, sitemaps with URL counts and dates, llms.txt, meta robots and X-Robots-Tag on a sample of pages, TDM reservation signals in headers, metadata, tdmrep.json and terms pages, and the bot-management service in front of the site. A domain that does not answer at all is free. | $0.02 per domain ($20.00 per 1,000) |

A list of 100 domains costs $2.00. Unreachable domains are free. A domain audit makes 10–40 requests depending on **Pages to sample**, **Maximum sitemap files** and how many terms pages the homepage links to.

### FAQ

#### The site allows GPTBot in robots.txt but its bot management blocks everything. Which is it?

Both are reported. `crawlers` shows the policy the owner wrote; `antiBot.challenged` shows what the edge actually did to a declared bot. For AI crawlers, the edge wins.

#### Why does Googlebot appear if Google-Extended is blocked?

Google-Extended only controls Gemini training and grounding; blocking it does not affect Google Search or AI Overviews, which follow Googlebot. The crawler table keeps both so you see the difference.

#### Why is TDM "robots-only" when the site blocks GPTBot?

A `Disallow` for training crawlers is machine-readable and is read as an opt-out in practice, but it is not a TDM reservation under the TDMRep protocol or a Content-Signal. `explicit` means such a reservation is present; `robots-only` means only robots.txt says so. Both appear with their source in `tdm.machineReadable`.

#### Is an AI crawler access audit legal, and does it touch personal data?

The audit reads what a site publishes for crawlers: robots.txt, sitemaps, well-known files, headers and a handful of public pages, at most three requests per second, identifying itself and never getting past a challenge. It stores no page content beyond what it reports: the site's robots.txt verbatim (up to 20 kB) while `includeRobotsText` is on, and passages of about 320 characters around a crawling, training or rights-reservation term on the terms pages it reads. Those lines are published by the site itself, and they can name a person (a contact address in robots.txt, a signature under the terms), so the output is not free of personal data by construction and the GDPR can apply to it, as it does to any page you fetch yourself. Nothing is read about your own visitors, customers or employees, and no crawler is asked to identify anyone. Switch off "Include robots.txt text" if you want the record without it, and delete the run's dataset when your retention rules require it. What you do with the finding (crawl or not, train or not) remains your decision and your responsibility.

#### What are the limitations?

- Terms pages rendered entirely by JavaScript cannot be read; `tdm.termsPages[].matches` is then empty and the page is still listed.
- Anti-bot detection names the vendor from public signatures; a custom WAF shows up only as a refusal.
- The page sample is a sample: a `noindex` on one template out of a hundred may not be in it. Raise **Pages to sample** for a fuller picture.
- llms.txt validation checks the format, not whether its links resolve.
- Very large files are not read in full. Sitemaps: up to 32 MB per file, as downloaded and after unpacking (the protocol allows 50 MB); a larger file is listed with an error, and at most 512 MB of sitemap text is read per domain, a refused file counting as 32 MB (`sitemaps.truncated`; `totalUrls` is then a minimum). A sitemap that answers with an error (such as a 403 or 429) or cannot be fetched is listed with that error, and the summary counts it as not read instead of as a sitemap without URLs. A usual location that answers 200 with something that is not a sitemap (a plain-text or JSON error page) is not counted as a sitemap, and the next location is tried. Sitemaps declared in robots.txt or found at a usual location but not fetched (on another host, not an absolute http(s) URL, or **Maximum sitemap files** set to 0) are named in the summary as not fetched, not as missing.
- Every other response is read up to 4 MB. robots.txt: only its first 500 KiB counts (as Google does), and a file over 4 MB is not read at all; `robots.note` says so. Such a file, a robots.txt that answers 5xx or 429 (Google stops crawling on both) and one that cannot be fetched are not treated as a site without robots.txt: every crawler row has status `unknown` (with `rootAllowed: false`), `aiAccess` lists no crawler as allowed or blocked, the verdict is at least `restricted`, and the summaries say crawler access is unknown. `aiAccess.genericCrawlersAllowed` is then `false` because access is not confirmed, not because crawlers are blocked: tell the two apart by the `*` row in `crawlers` (status `unknown` or `blocked`). A homepage over 4 MB is reported as `unreachable` (free) with the reason in `warnings`. An llms-full.txt or ai.txt over 4 MB that answers 200 counts as present with `bytes: 0`; an llms.txt, tdmrep.json or sampled page over 4 MB is reported with a note or error. A tdmrep.json that could not be read (over 4 MB, 5xx, 429 or a failed request) is not named among the places searched for a TDM reservation; the summaries say it could not be read. An error page over 4 MB (a 404 of megabytes) counts as a missing file, like a small one.
- A record keeps at most 2,000 robots.txt rules (`robots.truncated` marks a cut file).

### Related Actors

- [Regulatory Page Change Monitor: be told when a robots.txt, terms page or policy changes](https://apify.com/lwsdjfls/regulatory-page-change-monitor), with line-level diffs.
- [Government PDF Table Extractor: the tables inside the documents you are allowed to fetch](https://apify.com/lwsdjfls/govt-pdf-table-extractor), as JSON with checks.

### Support

A crawler token or bot-management service that is missing? Open an issue on the Actor page with the domain.

# Changelog

This Actor's version history is a separate document: https://apify.com/lwsdjfls/crawlability-ai-access-auditor/changelog.md

# Actor input Schema

## `domains` (type: `array`):

One per line: example.com or https://www.example.com/. Up to 1,000.

## `mode` (type: `string`):

audit: deliver the full audit of every domain. monitor: for a saved task on a schedule, deliver the first audit as a baseline and afterwards only domains whose policy changed (new blocks on AI crawlers, a TDM reservation, an anti-bot service, llms.txt appearing).

## `samplePages` (type: `integer`):

Pages fetched per domain for meta robots, X-Robots-Tag and TDM headers: the homepage plus URLs spread over the sitemaps (or homepage links). 0 checks only the homepage.

## `maxSitemaps` (type: `integer`):

Sitemap index and child files fetched per domain to count URLs and dates. 0 skips sitemaps.

## `checkTerms` (type: `boolean`):

Follows up to three terms/legal/policy links from the homepage and reports passages about text and data mining, AI training, automated access and reservations, with whether they prohibit.

## `extraUserAgents` (type: `array`):

Your own crawler tokens to evaluate against robots.txt, next to the built-in AI and search crawlers.

## `includeRobotsText` (type: `boolean`):

Adds the robots.txt content (up to 20 kB) to each record.

## `maxConcurrency` (type: `integer`):

Each domain gets at most three requests per second regardless. Sitemap files over 4 MB (up to 32 MB each) are downloaded at most two at a time across all domains, and gzip sitemaps are unpacked one at a time: a higher value does not multiply sitemap memory, and does not speed up those large downloads. Smaller sitemap files, gzip ones included, are read in parallel like every other response and never wait for another domain's large download.

## `webhookUrl` (type: `string`):

Receives a POST with a summary after each run that delivered records (in monitor mode: the changes). Must be a public http(s) address; redirects are not followed. Stored encrypted, because webhook URLs often carry a token.

## `stateKey` (type: `string`):

Separates independent monitors. Defaults to the task ID, so each saved task keeps its own baseline.

## `proxyConfiguration` (type: `object`):

Off by default on purpose: the audit measures how a site answers a declared bot from a data centre, which is how AI crawlers arrive. Switch on residential proxies only to audit a site that blocks Apify's addresses outright.

## Actor input object example

```json
{
  "domains": [
    "nos.nl",
    "rijksoverheid.nl",
    "apify.com",
    "anthropic.com"
  ],
  "mode": "audit",
  "samplePages": 8,
  "maxSitemaps": 25,
  "checkTerms": true,
  "extraUserAgents": [],
  "includeRobotsText": true,
  "maxConcurrency": 5,
  "webhookUrl": "https://hooks.example.com/crawlability",
  "stateKey": "client-portfolio",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `audit` (type: `string`):

Verdict, AI crawlers blocked, TDM opt-out level, anti-bot service, llms.txt and sitemap size per domain.

## `crawlers` (type: `string`):

What every listed crawler may do on each domain.

## `changes` (type: `string`):

In monitor mode: domains whose crawler policy changed since the previous run.

## `runSummary` (type: `string`):

Counts, failures and the cost of the run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "domains": [
        "nos.nl",
        "rijksoverheid.nl",
        "apify.com",
        "anthropic.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("lwsdjfls/crawlability-ai-access-auditor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "domains": [
        "nos.nl",
        "rijksoverheid.nl",
        "apify.com",
        "anthropic.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("lwsdjfls/crawlability-ai-access-auditor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "domains": [
    "nos.nl",
    "rijksoverheid.nl",
    "apify.com",
    "anthropic.com"
  ]
}' |
apify call lwsdjfls/crawlability-ai-access-auditor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,lwsdjfls/crawlability-ai-access-auditor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/d5hILdRBrIMUPO109/builds/ok30XfOKBF8MjFXfV/openapi.json