AI Crawler Access Checker

Bulk AI Crawler Policy Checker

bmiller1009/bulk-ai-crawler-policy-checker

Check many domains' robots.txt AI crawler policy in one run.

Robots AI Crawler Policy Checker

seeb/robots-ai-crawler-policy-checker

Analyze robots.txt and crawler policy text for AI bot access rules, disallow paths, sitemap references, and blocked model-training crawlers.

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.

See exactly which AI crawlers a site allows. Per domain it reads robots.txt for 25 named AI agents (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot…), plus ai.txt, llms.txt, noai meta, X-Robots-Tag and tdmrep.json, and quotes the rule behind every answer. Alerts on policy changes.

AI Crawler Policy Auditor

junipr/ai-crawler-policy-auditor

Audit robots and AI crawler policy signals for AI crawler access, blocking, and site-readiness.

AI Crawler Policy Checker: robots.txt & llms.txt

accountable_eel/ai-crawler-policy-lookup

Check what a website's robots.txt says about AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more) and whether it publishes an llms.txt. Reads the site's own public files directly: no API key, no third-party service. Charged only when an explicit AI policy or llms.txt is found.

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

Parses a site's public robots.txt into structured crawler groups, allow/disallow counts, crawl delays, sitemap declarations, and explicit or inherited policies for major AI crawlers.

AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers

aiqlabs/seo-ai-visibility-auditor

Crawls a site from one starting URL and reports how visible it is to AI: which AI crawlers robots.txt admits, whether llms.txt exists, and what structured data and metadata each page offers. Give it a domain to crawl - for a fixed list of pages, use SEO Audit Tool instead.

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.