AI Crawler Checker – robots.txt Rules for GPTBot & ClaudeBot

This AI crawler checker reads each website's robots.txt and reports, for 18 AI crawlers including GPTBot, ClaudeBot, Google-Extended, PerplexityBot and CCBot, whether that crawler may fetch the site, quoting the line that decided every verdict. It also reads the site's Content-Signal, llms.txt and ai.txt.

What it does

For each domain the actor fetches four files in parallel over a direct connection: /robots.txt, /llms.txt, /ai.txt and the homepage. Nothing else on the site is requested. Then it answers two questions: may each agent fetch the path you care about, and what does the site say about AI training as a whole.

One verdict per agent, in agents:

Matching follows robots.txt conventions: a group that names the agent beats the * group, the longest matching rule wins, an Allow wins a tie, and * and $ work as wildcards. allowed_by_agent repeats the verdicts as a flat {agent: true/false/null} map, which is the easiest field to use in a spreadsheet.

Site-level answers

Unknown is not allowed

The rule this checker is built around: a robots.txt that could not be read gives null verdicts, never "allowed". "We do not know the rules" and "there are no rules" are different answers, and treating the first as the second is how a checker tells you that you may crawl a site that forbids it.

• A 200 answer is read as written. An empty file means no rules.
• Any answer whose body is plainly a ruleset is parsed whatever its status code.
• A 4xx other than 429 means no rules, per RFC 9309: every agent is allowed, matched_by says no robots.txt.
• A 5xx, a timeout, a refused connection, a 403, a 429 or a challenge page means unknown: status is unreadable, every allowed is null, ai_training_allowed and blocks_any_ai_agent are null, and warnings says why. The site is not retried from another address.

Who it is for

• SEO and content teams checking that their own robots.txt says what they meant. Blocking GPTBot for training while keeping OAI-SearchBot for search is a one-line mistake in either direction, and each verdict names the line responsible.
• Publishers auditing every site and brand they run in one pass, with the evidence kept.
• AI, RAG and data teams who must honour robots.txt before crawling. Run the source list first, keep the rows where your agent is true, drop false and null, and store rule, matched_by and checked_at as the record of what the site said on the day.
• Agencies reporting on clients' AI crawler access, including llms.txt and pages that depend on JavaScript.
• Researchers measuring how many sites in a list block AI training.

How to use it

In the Apify Console:

1. Paste domains or URLs into Websites, one per line, up to 5,000. Only the host is used, so https://www.nytimes.com/section/world and nytimes.com are the same site.
2. Leave AI crawlers to check empty for all 18, or list the ones you care about. Any other user-agent token works too, including your own crawler's.
3. Set Path to check if you intend to fetch something other than the homepage, for example /blog/.
4. Turn Include sites whose robots.txt could not be read off if you only want answers you can act on; those sites are then left out and not charged.
5. Start the run and export the dataset as CSV, Excel or JSON.

From the API, this call starts a run, waits for it and returns the rows:

curl -X POST \
  "https://api.apify.com/v2/acts/locomint~ai-crawler-checker/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"domains": ["nytimes.com", "vercel.com"], "agents": ["GPTBot", "ClaudeBot"]}'

The synchronous endpoint waits up to 300 seconds. For a long list, start the run with POST https://api.apify.com/v2/acts/locomint~ai-crawler-checker/runs?token=YOUR_APIFY_TOKEN and read its dataset when the run finishes. With the Python client (

pip install apify-client

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("locomint/ai-crawler-checker").call(
    run_input={"domains": ["nytimes.com", "vercel.com", "python.org"], "includeUnreadable": False}
)
for row in client.dataset(run.default_dataset_id).iterate_items():
    print(row["domain"], row["ai_training_allowed"], row["blocked_agents"])

Input example

{
  "domains": ["nytimes.com", "vercel.com", "python.org"],
  "agents": ["GPTBot", "ClaudeBot", "Google-Extended", "PerplexityBot"],
  "path": "/",
  "checkLlmsTxt": true,
  "checkHomepage": true,
  "includeUnreadable": true,
  "concurrency": 5,
  "siteTimeout": 30
}

Output example

A real row, from a run on 11 September 2026 with all 18 agents. allowed_by_agent and agents held all 18 entries; four and two are shown. Vercel's robots.txt blocks none of the AI crawlers, but it publishes Content-Signal: ai-train=no, so ai_training_allowed is false.

{
  "domain": "vercel.com",
  "input": "vercel.com",
  "path": "/",
  "status": "ok",
  "ai_training_allowed": false,
  "blocks_any_ai_agent": false,
  "blocked_agents": [],
  "content_signal_ai_train": false,
  "content_signals": "search=yes, ai-input=yes, ai-train=no",
  "allowed_by_agent": {"GPTBot": true, "OAI-SearchBot": true, "ChatGPT-User": true, "ClaudeBot": true},
  "agents": [
    {"agent": "GPTBot", "allowed": true, "rule": null, "matched_by": "User-agent: *", "explicit": false, "crawl_delay": null, "vendor": "OpenAI", "purpose": "training"},
    {"agent": "ClaudeBot", "allowed": true, "rule": null, "matched_by": "User-agent: *", "explicit": false, "crawl_delay": null, "vendor": "Anthropic", "purpose": "training"}
  ],
  "robots_url": "https://vercel.com/robots.txt",
  "robots_status": 200,
  "robots_present": true,
  "robots_bytes": 629,
  "robots_truncated": false,
  "sitemaps": ["https://vercel.com/sitemap.xml"],
  "llms_txt": true,
  "ai_txt": false,
  "homepage_status": 200,
  "pay_per_crawl": false,
  "text_without_javascript": 138,
  "javascript_dependent": false,
  "warnings": [],
  "checked_at": "2026-09-11T18:33:01+00:00"
}

In the same run, nytimes.com blocked 16 of the 18 agents, each through a group that names it:

{"agent": "GPTBot", "allowed": false, "rule": "Disallow: /", "matched_by": "User-agent: GPTBot", "explicit": true, "crawl_delay": null, "vendor": "OpenAI", "purpose": "training"}

Googlebot and Amazonbot were allowed by groups of their own. python.org blocked none and had no Content-Signal, so its ai_training_allowed was true.

Pricing

Worked example: 10,000 domains is two runs of 5,000, so 10,000 x $0.0005 = $5.00 plus two start events of $0.00005 each for 1 GB runs. From 26 September 2026 the price per domain becomes $0.003.

You pay only these event prices; Apify compute is not billed to you separately, and the four files per domain are one charge. Unreadable sites are charged by default because the row is still an answer ("unknown"); set includeUnreadable to false and they are left out and cost nothing. Entries that are not a domain or URL produce no row and no charge. If you set a maximum cost per run, the actor stops before the row that would pass it.

FAQ

Does "allowed" mean I may use the site's content?

It means the site's robots.txt does not disallow that crawler from that path. robots.txt governs crawling; it is not a licence to the content, and a site's terms of service can say more. ai_training_allowed reflects robots.txt and Content-Signal only.

Why are all the verdicts null for one site?

Its robots.txt could not be read: the site answered 5xx, 403 or 429, served a challenge page, refused the connection or did not answer in time. warnings gives the reason. Unknown is kept as unknown rather than guessed; turn includeUnreadable off to leave such sites out.

Why does a site with no robots.txt show every agent allowed?

Under RFC 9309 a missing file (a 404 or another 4xx apart from 429) means there are no rules, so everything is allowed. matched_by says no robots.txt, which keeps that case apart from a site that wrote rules and allowed the agent. A 402 on robots.txt also counts as no rules, so read pay_per_crawl next to the verdicts.

Why is ai_training_allowed false when no agent is blocked?

The site publishes a Content-Signal directive with ai-train=no, as vercel.com did in the example above. The site said so in words, and that settles the training question whatever the per-agent rules allow; content_signals quotes the directive.

Can I check my own crawler or a path other than the homepage?

Yes. Any user-agent token in agents is checked as written, and path makes every verdict about that path, so /blog/ can be blocked on a site whose homepage is open. The path is evaluated against the rules; it is not fetched.

What counts as having llms.txt?

A 200 answer at /llms.txt with content that does not start as an HTML page. Many sites answer every unknown path with their HTML error page and a 200 status, and those do not count. null means the file was not checked or the site gave no answer.

How big a robots.txt can it read?

Up to 3 MB, which covers the largest files in normal use; some sites publish several hundred kilobytes and put their AI rules at the end. If a file is ever cut, robots_truncated is true and warnings says a rule near the end may be missing.

Limits

• Four files per domain: robots.txt, llms.txt, ai.txt and the homepage. No other page is fetched, and path is evaluated against the rules, not requested.
• The host is checked without www.. A site that answers only on its www. name comes back unreadable; subdomains such as blog.example.com are checked as given.
• A 403 or 429 on robots.txt is treated as a block, so the verdicts are null, not "no file".
• When a file names the same agent in two separate groups, only the first group is read.
• Meta robots tags, X-Robots-Tag headers, TDM reservation files and terms of service are not read. Content-Signal is read from robots.txt only.
• Verdicts describe the file at checked_at. Sites change their rules; re-run before a crawl.
• 5,000 domains and 50 agents per run, 30 seconds per site by default (5 to 90). A challenge is never retried from another address.

Other Locomint actors

• Google Maps Scraper & Email Extractor – Business Leads: Search terms and a city in, business records with website contacts out.
• Google Maps Scraper – Multi-City Lead Lists with Emails: Many categories across many cities in one deduplicated run.
• Google Maps Place Details Scraper – Bulk Place ID Lookup: Place IDs or place-page links in, full records out.
• Website Email Scraper – Contact Details, Socials & WhatsApp: Contact points from website URLs you supply.
• Website Content Crawler – Markdown for AI, Emails & Contacts: A whole site as Markdown, with its contact points.
• Bulk Email Verifier & Validator: Checks whether addresses can receive mail.
• Company Enrichment API – Domain to Emails, Socials & Tech: A domain in, its contacts and technologies out.
• Schema Markup Validator & Generator – JSON-LD Checker: Checks and generates schema.org markup.