AI Crawler Access Audit: which AI crawlers your robots.txt allows or blocks, and whether llms.txt exists

For every URL on your list the Actor reads the site's robots.txt, evaluates it for more than 150 known AI and search crawler tokens (GPTBot, ClaudeBot, Google-Extended, CCBot and others) and tells you which of them may access that URL and which rule made the decision. It also checks whether the site publishes llms.txt. At most two plain GET requests are sent per origin (/robots.txt and /llms.txt), evaluation follows RFC 9309 (a dedicated User-agent group beats *, the longest matching rule wins), and the bot dictionary ships inside the Actor, so nothing is downloaded at run time. The Actor reports what your robots.txt says to each bot; it does not observe what the bots actually do.

What it is good for

• AI visibility report for an agency: run dozens of client sites at once and get, per URL, which AI bots are allowed or disallowed and by which rule; the dataset is ready to be turned into an "AI visibility" report.
• Verify that your AI-training block really works: check that GPTBot, ClaudeBot, Google-Extended, CCBot or Bytespider are disallowed on every path that matters, and that the search-type bots that bring you traffic (OAI-SearchBot, Claude-SearchBot, PerplexityBot) were not blocked by accident. botsNotRespecting tells you how many of the checked bots declare they ignore robots.txt anyway.
• A building block for automations: one normalized JSON record per URL with a bundled dictionary of ~160 AI bot tokens (operator, category, stated respect for robots.txt), so you do not maintain your own list of user agents.
• llms.txt check: whether the site publishes /llms.txt with a valid H1 title, and how many sections and links it has (structure only, never the content).

How to use it

1. Open the Actor and paste your list of URLs into the urls field, one per line (or pass a JSON array through the API).
2. Leave bots empty to check the whole dictionary, or list the bot tokens you care about (for example GPTBot, ClaudeBot); turn checkLlmsTxt off if you only need the robots.txt decisions. The other defaults work for most runs.
3. Click Start. When the run finishes, the status message reads Processed N of M URLs (X with errors, Y not charged).
4. Download the results from the Dataset tab as JSON, CSV or Excel, or read them through the API.

Input

Field	Type	Default	Description
urls	list of strings	small self-test list	Absolute http(s) URLs, one per line; duplicates skipped; max 1,000 per run
bots	list of strings	(all known bots)	AI bot tokens to check, e.g. GPTBot, Bytespider. Case-insensitive matching; duplicates use first spelling. Invalid tokens are dropped. Empty = check all ~160 bots from the built-in dictionary.
checkLlmsTxt	boolean	true	Whether to also fetch and inspect llms.txt from each origin. Set to false to skip llms.txt checks entirely (saves requests).
timeoutSecs	integer 1–60	15	Maximum time per file (applied separately to robots.txt and llms.txt).
maxConcurrency	integer 1–50	10	URLs processed in parallel.

Output

One dataset record per input URL. Common fields:

Field	Type	Meaning
inputValue	string	URL as provided (credentials masked)
error	string or null	INVALID_URL, TIMEOUT, NETWORK_ERROR, TOO_MANY_REDIRECTS or null when the server responded
billable	boolean	true when the record was charged; false for records written free of charge (INVALID_URL, TIMEOUT, NETWORK_ERROR, TOO_MANY_REDIRECTS)
errorMessage	string or null	Human-readable reason
checkedAt	string	ISO 8601 timestamp

Robot-specific fields (one robots.txt fetch per origin, cached for the run):

Field	Type	Meaning
robotsUrl	string or null	<origin>/robots.txt; null for INVALID_URL
robotsStatus	int or null	HTTP status of the robots.txt response
robotsState	ok / missing / unreachable / invalid / null	State of robots.txt; null only for INVALID_URL
robotsTruncated	boolean	Whether the file exceeded 512,000 bytes
redirectCount	int 0–5	Number of redirects followed for robots.txt
robotsFromCache	boolean	true when robots.txt was reused from the same origin earlier in the run
sitemaps	string[]	Sitemap: URLs from robots.txt; [] when file is not ok

Audit fields, one BotDecision object per bot checked:

Field	Type	Meaning
bots	BotDecision[]	One entry per bot token, in input / dictionary order
botsChecked	int	Number of bots checked
botsAllowed	int	Bots with allowed: true
botsDisallowed	int	Bots with allowed: false
botsExplicit	int	Bots with a dedicated User-agent line in robots.txt
botsNotRespecting	int	Bots flagged as respect: "no" in the dictionary (regardless of the decision)
defaultGroupAllowed	boolean or null	Decision for the * wildcard group
defaultGroupRule	{ type, pattern } or null	Rule that decided for the default group

Each BotDecision object:

Field	Type	Meaning
bot	string	Bot token (spelling from the dictionary when known)
operator	string or null	Organization behind the bot
category	string or null	Category from the dictionary (AI Assistants, AI Data Scrapers, etc.)
respectsRobotsTxt	yes / no / unclear / null	Stated respect from the ai.robots.txt community list
allowed	boolean or null	RFC 9309 decision for this bot and URL; null at INVALID_URL
explicit	boolean	Whether the file has a group exactly matching this bot token
matchedGroup	string or null	Bot token (spelling from file) or * (wildcard) of the group that decided
matchedRule	{ type, pattern } or null	Longest-matching Allow/Disallow rule that decided
crawlDelay	number or null	Crawl-delay value from the deciding group, in seconds

llms.txt fields (fetched in parallel with robots.txt, cached per origin):

Field	Type	Meaning
llmsTxtUrl	string or null	<origin>/llms.txt; null for INVALID_URL; set even when checkLlmsTxt is false
llmsTxtStatus	int or null	HTTP status of the llms.txt response
llmsTxtState	present / absent / invalid / unreachable / skipped / null	skipped when checkLlmsTxt: false; null only for INVALID_URL
llmsTxtBytes	int or null	UTF-8 size in bytes of the text read (after truncation to 512,000); null when no text
llmsTxtTitle	string or null	H1 title from llms.txt (≤ 200 chars); null when no H1 on the first non-empty line
llmsTxtSections	int	Number of ## section headings
llmsTxtLinks	int	Number of list lines (- / * ) containing an http(s) Markdown link
llmsTxtTruncated	boolean	Whether llms.txt exceeded 512,000 bytes
llmsTxtFromCache	boolean	true when llms.txt was reused from the same origin

Dictionary metadata:

Field	Type	Meaning
dictionaryVersion	string	Snapshot date YYYY-MM-DD from the bot dictionary
dictionarySize	int	Total number of bot tokens in the dictionary snapshot

A SUMMARY record in the default key-value store holds the run counters (inputEntries, inputItems, skippedDuplicates, droppedEmptyOrNonString, truncatedOverLimit, resumedSkipped, processed, checked = charged, failed, notBilled, notSavedAfterLimit, unexpectedErrors, stoppedByChargeLimit).

Pricing

This Actor uses Apify pay per event pricing. Each run costs a small fixed start fee plus a price per URL audited, that is per URL whose server answered the robots.txt request; the exact amounts are shown in the Pricing tab. INVALID_URL, TIMEOUT, NETWORK_ERROR and TOO_MANY_REDIRECTS records are written free of charge. There are no additional platform usage fees.

A URL is charged only when its record is written, and every charged record is written. When the run's "Maximum cost per run" is reached, the Actor stops and reports how many URLs were left unprocessed.

Limits and behavior

• Up to 1,000 URLs per run; exact duplicates are skipped.
• Private, loopback, link-local and localhost addresses, URLs with embedded credentials and blocked ports are reported as INVALID_URL without any request and without charge. Credentials are masked in inputValue (user:***@host). A user name without a password is masked as well (https://***@host/); a password containing a raw /, \, ? or # is not recognized as credentials, so percent-encode such characters.
• A run interrupted by a platform migration resumes without processing or charging the same URL twice.
• At most two files are fetched per origin: robots.txt and llms.txt (one file when checkLlmsTxt is false), each following up to 5 redirects; the actual page is never requested.
• robots.txt and llms.txt are cached per origin within a run: subsequent URLs from the same origin reuse the result without additional requests.
• Maximum 5 redirects per file; redirects to private addresses are rejected (unreachable).
• Maximum 512,000 bytes read per file; larger files are truncated at the last complete line.
• No retries: each file is fetched once.
• timeoutSecs is applied independently to robots.txt and llms.txt.
• When robots.txt is unreachable (5xx, timeout, network error), all bots are considered disallowed per RFC 9309 §2.3.1.4.
• When robots.txt is missing (4xx) or invalid (NUL byte), all bots are considered allowed.
• The record is billed (billable: true) whenever the server responded to the robots.txt request, including 5xx errors and redirects that end without a valid location. Only INVALID_URL, TIMEOUT, NETWORK_ERROR, and TOO_MANY_REDIRECTS are not billed.
• Errors fetching llms.txt do not affect billing or the error field; they are reported only in llmsTxtState.
• llms.txt is inspected only for structure (H1 title, section count, link count); no file content enters the record beyond the H1 title.

FAQ

Why is Bytespider shown as not respecting robots.txt even when the decision for it is allowed?

The respectsRobotsTxt field reflects the community-maintained ai.robots.txt database, not the actual decision for this particular URL. Bytespider is listed with respect: "no" in that database, so every record for Bytespider shows respectsRobotsTxt: "no" regardless of whether the site's robots.txt allows or disallows it.

Why does a 500 error on robots.txt block all bots?

Per RFC 9309 §2.3.1.4, when a robots.txt file cannot be fetched due to a server error, crawlers must assume full disallow. The Actor follows the same rule: robotsState: "unreachable" → every bot gets allowed: false. Note that this record is still charged since the server did respond (with a 5xx code).

What happens when a site doesn't have an llms.txt?

The Actor reports llmsTxtState: "absent" (4xx status) or "unreachable" (5xx/timeout/network error). The error field and billing are unaffected; they depend only on the robots.txt outcome.

Can I check just a few bots instead of all 160+?

Yes, pass a list of bot tokens in the bots input field (e.g. ["GPTBot", "ClaudeBot"]). Case-insensitive matching; the spelling from the built-in dictionary is used in the output. Tokens not found in the dictionary produce records with operator: null, category: null, respectsRobotsTxt: null.

Is the bot dictionary up to date?

The dictionary is a snapshot of the ai-robots-txt/ai.robots.txt community repository, bundled with the Actor at build time. The snapshot date is visible in every record as dictionaryVersion. The dictionary is not fetched from the internet at runtime.

Other Actors by Gridline Tools

Other checks from the same publisher, each with one row per address and the same pricing rule (start fee plus price per checked item):

• HTTP Status and Headers Checker: Audit a list of URLs in one run: status code, every hop of the redirect chain and the final URL, response time, server and content type, and six key security headers.
• Robots.txt Checker: Test a list of URLs against each site's robots.txt for a chosen user agent (RFC 9309).
• RDAP Domain Lookup and Expiry Checker: Query RDAP registries for a list of domains and get registrar, creation and expiry dates, EPP statuses, name servers and DNSSEC in one table.

Disclaimer

The Actor sends plain HTTP GET requests to <origin>/robots.txt and <origin>/llms.txt only; it never requests the actual page content. Only the H1 title and section/link counts from llms.txt are stored in the dataset; no file content beyond the title enters any record. The bot dictionary is a snapshot of the ai-robots-txt/ai.robots.txt repository (MIT license, snapshot date in dictionaryVersion). You are responsible for having the right to query those URLs and for complying with the terms of service of the sites you check. Gridline Tools does not store the checked URLs or results beyond your own Apify account.