Crawlability & AI Crawler Access Auditor

What does Crawlability & AI Crawler Access Auditor do?

An AI crawler access audit and robots.txt checker for any list of domains. One run tells you, per domain, what generic crawlers and every AI crawler may do (GPTBot, ClaudeBot, CCBot, PerplexityBot, Google-Extended and 25 more), whether the site has opted out of text and data mining in a machine-readable way, what the sitemaps cover, whether an llms.txt exists, what the pages say in meta robots and X-Robots-Tag, and which bot-management service (Cloudflare, Akamai, DataDome, …) answers the door. It is written for two readers at once.

If you crawl: is this target feasible, and is there an opt-out?

Before you build a scraper, or an agent that fetches pages, you want three answers: will the site let a crawler in, will the edge block a declared bot, and has the owner reserved text and data mining rights. The third one is the TDM opt-out detection that Article 4(3) of the DSM Directive (2019/790) and the EU AI Act's transparency obligations turn into a compliance question: a reservation that is machine-readable has to be honoured by anyone mining for AI training.

Per domain you get:

• verdict: open, restricted (opt-outs, AI blocks, crawl delays, a bot manager in front, or a robots.txt that could not be read), blocked (generic crawlers disallowed or a declared bot refused on the homepage) or unreachable
• crawlers[]: for *, your own tokens and 29 known crawlers: allowed, partial, blocked, no-rules (no group applies), no-robots (the site has no robots.txt) or unknown (robots.txt could not be read; aiAccess.genericCrawlersAllowed is then false because access is not confirmed, not because crawlers are blocked), which group decided (own or wildcard), the disallowed and allowed paths, Crawl-delay
• tdm.level: explicit (TDMRep tdm-reservation header, meta tag or /.well-known/tdmrep.json; a Cloudflare Content-Signal: ai-train=no; an IETF Content-Usage: train-ai=n), robots-only (training crawlers disallowed, nothing else), terms-only (a clause in the terms) or none, with every source and the passages that were found
• antiBot: the service recognised from headers, cookies and challenge pages, and whether any fetch in the audit was refused (401/403/429/503) or challenged: the audit runs as a declared bot from a data centre, which is exactly how your crawler would arrive
• sitemaps: declared and discovered files, URL counts and lastmod range, so you know the size and freshness of what you are about to fetch
• summary.forScrapers: the findings in plain sentences

If you own the site: can AI read it, and do you want it to?

The GEO / AI visibility question has two halves that get mixed up. Answer engines (ChatGPT search, Claude, Perplexity, DuckDuckGo AI, Amazon) need to index you to cite you; training crawlers (GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, CCBot) take a copy for models. Most robots.txt files were written before that distinction existed. Per domain you get:

• aiAccess: training crawlers allowed and blocked, AI search crawlers allowed and blocked, user-initiated fetchers blocked (ChatGPT-User, Claude-User, Perplexity-User: the ones that open your page when someone asks about you), the Content-Signal and Content-Usage preferences if you set any
• llmsTxt and llmsFullTxt: present or not, and whether the file is real (title, sections, links) or a soft 404
• pages[]: a sample of pages with noindex, nofollow, noai, X-Robots-Tag, canonical and TDM headers, so a stray noindex on a template shows up
• tdm: whether your opt-out exists in the form the AI Act expects (machine-readable) or only in the terms nobody's crawler reads
• summary.forSiteOwners: what an AI answer engine can and cannot do with your site today, in plain sentences

Run it in monitor mode on a schedule and you are told when a robots.txt change blocks or unblocks an AI crawler, when a TDM reservation appears, when a bot manager shows up in front of the site, or when an llms.txt is added.

What is checked, exactly

Check	How
robots.txt	Fetched from the origin, parsed as RFC 9309 and Google read it: groups of User-agent lines, longest-match rules, Allow wins a tie, * and $ in paths, Crawl-delay, Sitemap. A crawler follows its own group, else *. Like Google, only the first 500 KiB is read. Soft 404s (an HTML page at /robots.txt), 5xx and 429 are reported for what they do to crawlers
AI crawlers	29 tokens with operator and purpose (training, AI search, user-initiated fetch, search, dataset, archive): OpenAI, Anthropic, Google, Microsoft, Common Crawl, Perplexity, Apple, ByteDance, Meta, Amazon, Cohere, Mistral, DuckDuckGo, You.com, Diffbot, Webz.io, Hive, Huawei, Internet Archive. Add your own with Extra user-agent tokens
Content signals	Cloudflare's Content-Signal: search=…, ai-input=…, ai-train=… (September 2025) and the IETF AIPREF Content-Usage rule and header (train-ai=n, search=y, optionally per path)
Sitemaps	Files declared in robots.txt, else /sitemap.xml and the usual alternatives; indexes followed breadth-first up to Maximum sitemap files; gzip handled; per file the URL count and lastmod range. Only files on the domain itself, its www twin or the host its homepage redirects to are fetched; sitemaps on other hosts are listed, not read
llms.txt	/llms.txt parsed (title, summary, sections, links), /llms-full.txt and /ai.txt presence
Pages	The homepage plus URLs spread over the sitemaps (or homepage links): meta robots and per-bot metas, X-Robots-Tag, noindex/nofollow/noai, canonical, tdm-reservation and tdm-policy in headers or meta, Content-Usage header
TDM reservation	/.well-known/tdmrep.json, the headers and meta tags above, robots.txt signals, and up to three terms, legal or AI-policy pages linked from the homepage, scanned in English, Dutch, German and French for passages on text and data mining, AI training, automated access and reservations, each marked prohibitive or not
Anti-bot	Cloudflare (incl. Turnstile), Akamai Bot Manager, Imperva, DataDome, HUMAN/PerimeterX, Kasada, F5/Shape, AWS WAF, Fastly, Vercel, Sucuri, Anubis, reCAPTCHA, hCaptcha, from server headers, cookies, script references and challenge pages; plus any refusal of the declared bot

The audit identifies itself (crawlability-ai-access-auditor/0.1; +https://apify.com/lwsdjfls/crawlability-ai-access-auditor), makes at most three requests per second per domain, and never tries to get past a challenge: a refusal is the finding. It fetches robots.txt, sitemaps, a handful of pages and the well-known files; it does not crawl the site.

Features

• robots.txt checker per crawler: RFC 9309 parsing, longest-match rules, Allow wins ties, wildcards, crawl delays, soft 404s
• AI crawler access audit for 29 known tokens (training, AI search, user-initiated fetchers) plus your own
• TDM opt-out detection: TDMRep headers, meta tags and /.well-known/tdmrep.json, Cloudflare Content-Signal, IETF Content-Usage, robots.txt signals and terms passages in four languages
• llms.txt checker: presence and validity of llms.txt, llms-full.txt and ai.txt
• Sitemap inventory with URL counts and lastmod ranges
• Meta robots and X-Robots-Tag on a page sample, canonical and TDM headers included
• Anti-bot service detection from public signatures, plus whether the declared bot was refused
• Two summaries per domain: one for scrapers, one for site owners
• Monitor mode for policy changes, with a webhook
• Runs through the Apify API, on a schedule, or from Make, Zapier and n8n

Use cases

Scraper builders: feasibility check before building

Run your target list once. blocked and antiBot.challenged tell you where a declared bot is refused; crawlers[].disallowed shows which paths are off limits for * and for your own token; sitemaps.totalUrls and lastmodMax size the job; tdm.level tells you whether mining for AI purposes has been reserved.

// npm install apify-client
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('lwsdjfls/crawlability-ai-access-auditor').call({ domains: targets, samplePages: 5 });
const { items } = await client.dataset(run.defaultDatasetId).listItems({ clean: true });
for (const { payload } of items) {
    const optOut = payload.tdm.level !== 'none';
    const refused = payload.verdict === 'blocked';
    console.log(payload.domain, payload.verdict, optOut ? `opt-out: ${payload.tdm.level}` : 'no opt-out', refused ? '' : `${payload.sitemaps.totalUrls} URLs in sitemaps`);
}

SEO and GEO agencies: AI-visibility audit for clients

Audit a client's domains and their competitors' in one run, hand over summary.forSiteOwners per domain, and schedule the monitor task so the next robots.txt edit that blocks Perplexity is caught the same week.

Data and legal teams: a compliance register of TDM reservations

Keep a dataset of the TDM status of every source in your data pipeline, with the passages and URLs as evidence, refreshed monthly. The Actor reports what the site publishes; whether a clause is a valid reservation under Article 4(3) of Directive 2019/790, or what the AI Act requires of a specific provider, depends on jurisdiction and facts. Take tdm to your counsel, with the sources and passages the record gives you. This is not legal advice.

How to use it

1. Put the domains into Domains, bare like example.com, and press Start. One run covers robots.txt for every crawler it knows, the sitemaps and their URL counts, llms.txt, the well known files, meta robots on a sample of pages, and whichever anti-bot service sits in front of the site.
2. Read the audit view for the verdict per domain and the crawlers view for the part most people came for: the rule that applies to each individual crawler, so you can see at a glance whether GPTBot, ClaudeBot, CCBot and Google-Extended are allowed or blocked, and whether that was deliberate.
3. Give it a State key and schedule it monthly, or weekly if you also audit domains you do not control. The changes view then reports the day a robots.txt rule, a Content-Signal or a TDM reservation moves, which on a competitor's site is a strategy change and on your own site is usually an accident.

Turn on Scan terms and legal pages for TDM clauses when the legal position matters. A machine readable reservation under TDMRep is easy to check, but plenty of sites only reserve their rights in prose on a terms page, and that scan finds those too. Use Pages to sample to control how deep the per page check goes.

A domain that cannot be reached still produces a record with the reason, so a list never comes back shorter than you sent it. What this does not do is judge: it reports what a site declares and what a crawler would encounter. Whether a reservation is enforceable where you operate is a legal question, not a technical one.

{
  "domains": ["nos.nl", "rijksoverheid.nl", "apify.com"],
  "checkTerms": true,
  "stateKey": "monthly-ai-access-audit"
}

Input

Field	What it does	Example
domains	One per line: example.com or https://www.example.com/, up to 1,000. Subdomains are separate domains with their own robots.txt.	["nos.nl", "rijksoverheid.nl", "anthropic.com"]
mode	audit delivers the full audit of every domain; monitor delivers a baseline and afterwards only domains whose policy changed.	"audit"
samplePages	Pages fetched per domain for meta robots, X-Robots-Tag and TDM headers: the homepage plus URLs spread over the sitemaps.	8
maxSitemaps	Sitemap index and child files fetched per domain.	25
checkTerms	Follows up to three terms, legal or policy links from the homepage and reports passages about text and data mining, AI training and automated access.	true
extraUserAgents	Your own crawler tokens to evaluate against robots.txt.	["MyCompanyBot"]
includeRobotsText	Adds the robots.txt content (up to 20 kB) to each record.	true
maxConcurrency	Domains audited in parallel; each domain gets at most three requests per second regardless. Memory: sitemap files over 4 MB (up to 32 MB each) are downloaded at most two at a time across the whole run, and gzip sitemaps are unpacked one at a time, so a higher value does not multiply sitemap memory, and does not speed up those large downloads; smaller sitemap files (gzip ones included) and every other response are read up to 4 MB each, one per domain at a time, in parallel, and never wait for another domain's large download.	5
webhookUrl	Receives a POST with a summary after each run that delivered records; stored encrypted.	https://hooks.example.com/crawlability
stateKey	Separates independent monitors. Defaults to the task ID.	client-portfolio
proxyConfiguration	Off by default on purpose: the audit measures how a site answers a declared bot from a data centre.	{ "useApifyProxy": false }

{
  "domains": ["nos.nl", "www.rijksoverheid.nl", "www.cloudflare.com", "www.bol.com"],
  "samplePages": 8,
  "maxSitemaps": 25,
  "checkTerms": true,
  "extraUserAgents": ["MyCompanyBot"]
}

For a scheduled watch, save a task with "mode": "monitor": the first run delivers a baseline, later runs only the domains whose policy changed, with a summary such as nos.nl: no longer blocks ClaudeBot, Claude-SearchBot; TDM robots-only → explicit. A run in which a domain does not answer is reported by its verdict (restricted → unreachable), not as blocks lifted or files removed, and a robots.txt that could not be read is not reported as blocks lifted or added.

Output

One record per domain, with verdict, crawlers[], aiAccess, tdm, antiBot, sitemaps, llmsTxt, pages[] and the two summaries described above. Below: nos.nl, the Dutch public broadcaster: every AI training crawler blocked in robots.txt, AI search crawlers blocked too, no machine-readable TDM reservation, no bot manager, sitemaps with dated URLs.

Three arrays are shortened in the example so the rest of this page stays readable; a real record carries them in full: crawlers holds a row per crawler checked (30 in this run, 4 shown), robots.groups a row per user-agent block in robots.txt (26 in this run, 2 shown) and pages a row per sampled page (3 in this run, 1 shown). robots.text is null here because this run had includeRobotsText off; with the default on it holds the file verbatim, up to 20 kB.

{
  "envelopeVersion": 1,
  "id": "nos.nl",
  "scope": "domain:nos.nl",
  "changeType": "added",
  "sourceUrl": "https://nos.nl/robots.txt",
  "scrapedAt": "2026-09-15T22:46:47.370Z",
  "firstSeenAt": "2026-09-15T22:46:47.370Z",
  "hash": "a287ddffae101439829d3ca2012d49418bcf93aff45e2f85cf39b48eb1400406",
  "previousHash": null,
  "changes": [],
  "summary": "Added: nos.nl: restricted, blocks 8 AI training crawlers, TDM robots-only",
  "diffText": null,
  "payload": {
    "domain": "nos.nl",
    "origin": "https://nos.nl",
    "reachable": true,
    "verdict": "restricted",
    "summary": {
      "forScrapers": [
        "robots.txt allows generic crawlers on /; 7 path(s) disallowed for *: /hybrid/, /widget-embed/, /special/, /humans.txt, /api and 2 more.",
        "No bot-management service recognised; all fetched pages answered normally to a declared bot.",
        "No TDMRep, Content-Signal or Content-Usage reservation, but robots.txt blocks AI training crawlers (GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended and 3 more): treat as an opt-out for training.",
        "Sitemaps: 2 of 3 files read (cap), at least 174 URLs, last change 2026-09-15."
      ],
      "forSiteOwners": [
        "All listed AI training crawlers are blocked (GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended and 3 more).",
        "AI search crawlers blocked: PerplexityBot, DuckAssistBot, YouBot. The site will not appear in those AI answers.",
        "User-initiated fetches blocked: ChatGPT-User, Meta-ExternalFetcher. Assistants cannot open the site when a user asks about it.",
        "No llms.txt: no curated entry point for language models.",
        "Training crawlers are blocked in robots.txt only; a TDMRep header or Content-Signal would make the reservation explicit for every crawler."
      ]
    },
    "homepage": {
      "url": "https://nos.nl/",
      "finalUrl": "https://nos.nl/",
      "status": 200,
      "title": "NOS.nl - Nieuws, Sport en Evenementen | Nederlandse Omroep Stichting",
      "server": "nginx",
      "redirectedToOtherHost": false
    },
    "robots": {
      "url": "https://nos.nl/robots.txt",
      "status": 200,
      "present": true,
      "bytes": 1304,
      "groups": [
        {
          "agents": [
            "*"
          ],
          "rules": [
            {
              "type": "disallow",
              "path": "/hybrid/"
            },
            {
              "type": "disallow",
              "path": "/widget-embed/"
            },
            {
              "type": "disallow",
              "path": "/special/"
            },
            {
              "type": "disallow",
              "path": "/humans.txt"
            },
            {
              "type": "disallow",
              "path": "/api"
            },
            {
              "type": "disallow",
              "path": "/zoeken"
            },
            {
              "type": "disallow",
              "path": "/regio"
            }
          ],
          "crawlDelay": null,
          "contentSignal": null,
          "contentUsage": []
        },
        {
          "agents": [
            "CCBot"
          ],
          "rules": [
            {
              "type": "disallow",
              "path": "/"
            }
          ],
          "crawlDelay": null,
          "contentSignal": null,
          "contentUsage": []
        }
      ],
      "sitemaps": [
        "https://nos.nl/sitemap/index.xml",
        "https://nos.nl/sitemap/news.xml",
        "https://nos.nl/sitemap/livestreams.xml"
      ],
      "unknownLines": 0,
      "note": null,
      "text": null
    },
    "crawlers": [
      {
        "token": "*",
        "operator": "any crawler",
        "purpose": "generic",
        "status": "partial",
        "matchedBy": "own",
        "rootAllowed": true,
        "disallowed": [
          "/hybrid/",
          "/widget-embed/",
          "/special/",
          "/humans.txt",
          "/api",
          "/zoeken",
          "/regio"
        ],
        "allowed": [],
        "crawlDelay": null,
        "contentSignal": null,
        "contentUsage": []
      },
      {
        "token": "GPTBot",
        "operator": "OpenAI",
        "purpose": "ai-training",
        "status": "blocked",
        "matchedBy": "own",
        "rootAllowed": false,
        "disallowed": [
          "/"
        ],
        "allowed": [],
        "crawlDelay": null,
        "contentSignal": null,
        "contentUsage": []
      },
      {
        "token": "OAI-SearchBot",
        "operator": "OpenAI",
        "purpose": "ai-search",
        "status": "partial",
        "matchedBy": "wildcard",
        "rootAllowed": true,
        "disallowed": [
          "/hybrid/",
          "/widget-embed/",
          "/special/",
          "/humans.txt",
          "/api",
          "/zoeken",
          "/regio"
        ],
        "allowed": [],
        "crawlDelay": null,
        "contentSignal": null,
        "contentUsage": []
      },
      {
        "token": "ChatGPT-User",
        "operator": "OpenAI",
        "purpose": "ai-user",
        "status": "blocked",
        "matchedBy": "own",
        "rootAllowed": false,
        "disallowed": [
          "/"
        ],
        "allowed": [],
        "crawlDelay": null,
        "contentSignal": null,
        "contentUsage": []
      }
    ],
    "aiAccess": {
      "genericCrawlersAllowed": true,
      "trainingAllowed": [],
      "trainingBlocked": [
        "GPTBot",
        "ClaudeBot",
        "anthropic-ai",
        "Google-Extended",
        "Applebot-Extended",
        "Bytespider",
        "meta-externalagent",
        "cohere-ai"
      ],
      "searchAllowed": [
        "OAI-SearchBot",
        "Claude-SearchBot",
        "Amazonbot"
      ],
      "searchBlocked": [
        "PerplexityBot",
        "DuckAssistBot",
        "YouBot"
      ],
      "userFetchBlocked": [
        "ChatGPT-User",
        "Meta-ExternalFetcher"
      ],
      "contentSignal": null,
      "contentUsage": []
    },
    "sitemaps": {
      "declared": [
        "https://nos.nl/sitemap/index.xml",
        "https://nos.nl/sitemap/news.xml",
        "https://nos.nl/sitemap/livestreams.xml"
      ],
      "discovered": [],
      "fetched": [
        {
          "url": "https://nos.nl/sitemap/index.xml",
          "kind": "urlset",
          "status": 200,
          "urlCount": 54,
          "childCount": 0,
          "lastmodMin": "2026-09-15",
          "lastmodMax": "2026-09-15",
          "error": null
        },
        {
          "url": "https://nos.nl/sitemap/news.xml",
          "kind": "urlset",
          "status": 200,
          "urlCount": 120,
          "childCount": 0,
          "lastmodMin": null,
          "lastmodMax": null,
          "error": null
        }
      ],
      "listed": 3,
      "totalSitemaps": 2,
      "totalUrls": 174,
      "lastmodMin": "2026-09-15",
      "lastmodMax": "2026-09-15",
      "truncated": true
    },
    "llmsTxt": {
      "url": "https://nos.nl/llms.txt",
      "present": false,
      "status": 404,
      "bytes": 29630,
      "title": null,
      "summary": null,
      "sections": [],
      "linkCount": 0,
      "note": null
    },
    "llmsFullTxt": {
      "url": "https://nos.nl/llms-full.txt",
      "present": false,
      "status": 404,
      "bytes": 29631
    },
    "aiTxt": {
      "url": "https://nos.nl/ai.txt",
      "present": false,
      "status": 404,
      "bytes": 29631
    },
    "pages": [
      {
        "url": "https://nos.nl/",
        "status": 200,
        "title": "NOS.nl - Nieuws, Sport en Evenementen | Nederlandse Omroep Stichting",
        "xRobotsTag": null,
        "metaRobots": [
          {
            "name": "robots",
            "content": "max-image-preview:large"
          }
        ],
        "noindex": false,
        "nofollow": false,
        "noai": false,
        "tdmReservation": null,
        "tdmPolicy": null,
        "contentUsage": null,
        "canonical": "https://nos.nl/",
        "challenged": false,
        "error": null
      }
    ],
    "pageDirectives": {
      "sampled": 3,
      "fetched": 3,
      "noindex": 0,
      "noai": 0,
      "tdmReserved": 0,
      "challenged": 0
    },
    "tdm": {
      "level": "robots-only",
      "machineReadable": [
        {
          "kind": "robots",
          "url": "https://nos.nl/robots.txt",
          "detail": "robots.txt disallows AI training crawlers: GPTBot, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Bytespider, meta-externalagent, cohere-ai"
        }
      ],
      "tdmrep": {
        "url": "https://nos.nl/.well-known/tdmrep.json",
        "present": false,
        "status": 404,
        "entries": [],
        "note": null
      },
      "termsPages": [],
      "termsReservation": false
    },
    "antiBot": {
      "services": [],
      "challenged": false,
      "challengedUrls": []
    },
    "requests": 11,
    "warnings": [],
    "checkedAt": "2026-09-15T22:46:47.370Z"
  }
}

Pricing

Event	Name	What it means	Price
apify-actor-start	Actor start	Charged by Apify when a run starts: once for a run of up to 1 GB of memory, and once more for every extra GB. It covers the first five seconds of compute.	$0.00005 per run up to 1 GB of memory ($0.05 per 1,000)
domain-audited (primary)	Domain audited	One domain audited: robots.txt per crawler (AI crawlers included), Content-Signal and Content-Usage rules, sitemaps with URL counts and dates, llms.txt, meta robots and X-Robots-Tag on a sample of pages, TDM reservation signals in headers, metadata, tdmrep.json and terms pages, and the bot-management service in front of the site. A domain that does not answer at all is free.	$0.02 per domain ($20.00 per 1,000)

A list of 100 domains costs $2.00. Unreachable domains are free. A domain audit makes 10–40 requests depending on Pages to sample, Maximum sitemap files and how many terms pages the homepage links to.

FAQ

The site allows GPTBot in robots.txt but its bot management blocks everything. Which is it?

Both are reported. crawlers shows the policy the owner wrote; antiBot.challenged shows what the edge actually did to a declared bot. For AI crawlers, the edge wins.

Why does Googlebot appear if Google-Extended is blocked?

Google-Extended only controls Gemini training and grounding; blocking it does not affect Google Search or AI Overviews, which follow Googlebot. The crawler table keeps both so you see the difference.

Why is TDM "robots-only" when the site blocks GPTBot?

A Disallow for training crawlers is machine-readable and is read as an opt-out in practice, but it is not a TDM reservation under the TDMRep protocol or a Content-Signal. explicit means such a reservation is present; robots-only means only robots.txt says so. Both appear with their source in tdm.machineReadable.

Is an AI crawler access audit legal, and does it touch personal data?

The audit reads what a site publishes for crawlers: robots.txt, sitemaps, well-known files, headers and a handful of public pages, at most three requests per second, identifying itself and never getting past a challenge. It stores no page content beyond what it reports: the site's robots.txt verbatim (up to 20 kB) while includeRobotsText is on, and passages of about 320 characters around a crawling, training or rights-reservation term on the terms pages it reads. Those lines are published by the site itself, and they can name a person (a contact address in robots.txt, a signature under the terms), so the output is not free of personal data by construction and the GDPR can apply to it, as it does to any page you fetch yourself. Nothing is read about your own visitors, customers or employees, and no crawler is asked to identify anyone. Switch off "Include robots.txt text" if you want the record without it, and delete the run's dataset when your retention rules require it. What you do with the finding (crawl or not, train or not) remains your decision and your responsibility.

What are the limitations?

• Terms pages rendered entirely by JavaScript cannot be read; tdm.termsPages[].matches is then empty and the page is still listed.
• Anti-bot detection names the vendor from public signatures; a custom WAF shows up only as a refusal.
• The page sample is a sample: a noindex on one template out of a hundred may not be in it. Raise Pages to sample for a fuller picture.
• llms.txt validation checks the format, not whether its links resolve.
• Very large files are not read in full. Sitemaps: up to 32 MB per file, as downloaded and after unpacking (the protocol allows 50 MB); a larger file is listed with an error, and at most 512 MB of sitemap text is read per domain, a refused file counting as 32 MB (sitemaps.truncated; totalUrls is then a minimum). A sitemap that answers with an error (such as a 403 or 429) or cannot be fetched is listed with that error, and the summary counts it as not read instead of as a sitemap without URLs. A usual location that answers 200 with something that is not a sitemap (a plain-text or JSON error page) is not counted as a sitemap, and the next location is tried. Sitemaps declared in robots.txt or found at a usual location but not fetched (on another host, not an absolute http(s) URL, or Maximum sitemap files set to 0) are named in the summary as not fetched, not as missing.
• Every other response is read up to 4 MB. robots.txt: only its first 500 KiB counts (as Google does), and a file over 4 MB is not read at all; robots.note says so. Such a file, a robots.txt that answers 5xx or 429 (Google stops crawling on both) and one that cannot be fetched are not treated as a site without robots.txt: every crawler row has status unknown (with rootAllowed: false), aiAccess lists no crawler as allowed or blocked, the verdict is at least restricted, and the summaries say crawler access is unknown. aiAccess.genericCrawlersAllowed is then false because access is not confirmed, not because crawlers are blocked: tell the two apart by the * row in crawlers (status unknown or blocked). A homepage over 4 MB is reported as unreachable (free) with the reason in warnings. An llms-full.txt or ai.txt over 4 MB that answers 200 counts as present with bytes: 0; an llms.txt, tdmrep.json or sampled page over 4 MB is reported with a note or error. A tdmrep.json that could not be read (over 4 MB, 5xx, 429 or a failed request) is not named among the places searched for a TDM reservation; the summaries say it could not be read. An error page over 4 MB (a 404 of megabytes) counts as a missing file, like a small one.
• A record keeps at most 2,000 robots.txt rules (robots.truncated marks a cut file).

Related Actors

• Regulatory Page Change Monitor: be told when a robots.txt, terms page or policy changes, with line-level diffs.
• Government PDF Table Extractor: the tables inside the documents you are allowed to fetch, as JSON with checks.

Support

A crawler token or bot-management service that is missing? Open an issue on the Actor page with the domain.