AI Crawler Checker: GPTBot, ClaudeBot robots.txt + llms.txt
Pricing
$6.00 / 1,000 results
AI Crawler Checker: GPTBot, ClaudeBot robots.txt + llms.txt
Check whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers are allowed by a site's robots.txt, and whether llms.txt and sitemaps exist.
Pricing
$6.00 / 1,000 results
Rating
0.0
(0)
Developer
Zachary Vorsteg
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
12 days ago
Last modified
Share
AI Crawler Readiness Audit (robots.txt, llms.txt, sitemaps)
Check which AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent, Amazonbot) a site's robots.txt allows. Also checks whether /llms.txt and /llms-full.txt exist and which sitemaps robots.txt lists.
Output fields
One row per site: allowed/blocked counts, blockedBots[], the verdict per bot with the robots rule that matched, llms.txt status and size, sitemapsInRobots[].
Data source & compliance
Reads 3 public files per site. The robots.txt matching follows RFC 9309: longest match wins, and Allow wins a tie. It is a minimal implementation, so edge cases may differ from each bot's real behavior.
Pricing (proposal)
Pay per result: $0.006 per result ($6.00 per 1,000), plus Apify's default start event. See .actor/pricing.proposal.json. Competitor prices are in ../../evidence/apify_competitor_pricing.json (fetched 2026-10-09).
Example
See samples/sample-output-first5.json (the full test run is in samples/sample-output.json).
Local test
Tested 2026-10-09 on 4 sites: nytimes.com allowed 1 of 13 bots. docs.apify.com and cloudflare.com allowed 13 of 13 and serve llms.txt. python.org allowed 13 of 13 and has no llms.txt.
Run again with: ../_test.sh ai-crawler-readiness-audit '<input json>'
