Chinese AI Crawler Access Checker for robots.txt

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

Audit domains for llms.txt, robots.txt AI bot rules, sitemap hints, and public AI crawler readiness signals.

LLMs.txt AI Discovery Auditor

glowing_glove/llms-txt-ai-discovery-auditor

Check websites for llms.txt, llms-full.txt, robots AI crawler rules, sitemap hints, and AI search readiness gaps.

Audit robots.txt policies for major AI crawlers and inspect llms.txt discovery files for public websites with structured machine-readable results.

AI Crawler Policy Lookup: robots.txt & llms.txt Checker

accountable_eel/ai-crawler-policy-lookup

Check what a website's robots.txt says about AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more) and whether it publishes an llms.txt. Reads the site's own public files directly: no API key, no third-party service. Charged only when an explicit AI policy or llms.txt is found.

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Audit websites for AI crawler/readiness basics: llms.txt, sitemap, robots rules, titles, metadata, and AI crawler directives.

Check robots.txt, llms.txt, sitemap hints, and AI crawler blocking rules for websites and domains.

AI Bot & LLM Crawler Exposure Auditor

glowing_glove/ai-bot-llm-crawler-exposure-auditor

Audit robots.txt, llms.txt, AI crawler directives, public content exposure, and policy gaps for websites in the AI search era.

How does a site present itself to AI? Per-crawler robots.txt verdicts for 13 major AI user agents (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and more), llms.txt presence and shape, sitemap signals, and a concrete fix list. Charged only when the audit completes.