AI Crawler Access Auditor (robots.txt + llms.txt)

Paste in a list of domains. For each one you get a row that shows which AI crawlers its robots.txt allows, partially allows or blocks, whether /llms.txt and /llms-full.txt exist, and which sitemaps it declares. You don't need a login, proxies or setup.

Bots checked (14): GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-User, Google-Extended, PerplexityBot, CCBot, Bytespider, Applebot-Extended, Meta-ExternalAgent, Amazonbot, cohere-ai.

Use cases

• SEO / GEO audits: check whether a client's site can show up in AI search and answer engines, or whether it blocks them by accident.
• Competitor research: see which competitors opt out of AI training and which ones publish an llms.txt.
• Publisher policy tracking: schedule a weekly run over a list of sites and spot when a robots.txt policy changes.
• Agency onboarding: run one quick, repeatable check on every new client domain.
• Dataset and compliance prep: before you collect data, record which sites have opted out of AI crawlers.

Input

{ "domains": ["apify.com", "nytimes.com", "github.com"], "timeoutSecs": 10, "maxDomains": 500 }

Sample output (one row per domain, trimmed)

{
  "input": "nytimes.com",
  "origin": "https://nytimes.com",
  "ok": true,
  "robotsTxtStatus": 200,
  "llmsTxt": false,
  "llmsFullTxt": false,
  "aiBotsChecked": 14,
  "aiBotsBlocked": 13,
  "bots": { "GPTBot": "blocked", "ClaudeBot": "blocked", "Google-Extended": "blocked", "PerplexityBot": "blocked", "CCBot": "blocked", "Amazonbot": "partial", "...": "..." },
  "sitemaps": ["https://www.nytimes.com/sitemaps/new/sitemap.xml.gz", "..."],
  "checkedAt": "2026-10-10T09:18:38.000Z",
  "error": null
}

The bots values mean:

• allowed: / isn't disallowed.
• partial: some paths are disallowed.
• blocked: Disallow: / applies to that bot.

3 test audits (live run, 2026-10-10)

Domain	AI bots blocked (of 14)	Pattern	llms.txt	llms-full.txt	Sitemaps declared
apify.com	0	All 14 allowed	yes	no	1
nytimes.com	13	All blocked except Amazonbot (partial)	no	no	25
github.com	1	Bytespider blocked; the other 13 partial (some paths disallowed)	yes	yes	0

Pricing

You pay per audited domain (domain-audited). If a domain is unreachable or invalid, you get a free error row. Set a max cost per run and the actor stops at your budget.

Limits

• It checks the rules for the root path / only.
• It reads robots.txt as written. It doesn't test whether a firewall or CDN actually blocks a bot.
• Results show the site at the moment it was checked (checkedAt).