AI Crawler Policy Lookup — robots.txt & llms.txt Checker

AI Crawler Access Checker

zinin/ai-crawler-access-checker

Audit up to 100 sites for 16 AI crawler policies. Get search vs training verdicts, robots.txt and llms.txt evidence, confidence, unknown states, priorities, and next actions for GEO/AEO reviews.

See exactly which AI crawlers a site allows. Per domain it reads robots.txt for 25 named AI agents (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot…), plus ai.txt, llms.txt, noai meta, X-Robots-Tag and tdmrep.json, and quotes the rule behind every answer. Alerts on policy changes.

llms.txt Checker: AI Search Readiness Audit

apifmcpfactory/llms-txt-checker

Audits whether a website is readable by AI assistants (llms.txt, robots.txt AI-crawler access, sitemap) and returns a 0-100 score with concrete fixes - use when asked whether a site is visible to ChatGPT/Claude or how to improve that; $0.005 per site audited.

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

Audit one page or discover up to 100 same-origin URLs from XML sitemaps. Check AI crawler access, robots.txt, llms.txt, JSON-LD, indexability, citations, and answer-ready content. Export scored GEO reports with transparent evidence and prioritized fixes for AI search visibility.

AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers

aiqlabs/seo-ai-visibility-auditor

Crawls a site from one starting URL and reports how visible it is to AI: which AI crawlers robots.txt admits, whether llms.txt exists, and what structured data and metadata each page offers. Give it a domain to crawl - for a fixed list of pages, use SEO Audit Tool instead.

AI Tooling Detector Company AI Adoption Signals for Clay

mambalabs/ai-tooling-detector

Detects whether a company declares, deploys, or charges for AI. Returns a flat Clay-ready row with an AI maturity tier, detected AI vendors, llms.txt status, and pricing-page evidence.

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

Agent Accessibility Auditor llms txt Robots AI Schema Check

mambalabs/agent-accessibility-auditor

Can an AI agent read this site. Checks llms.txt, robots AI crawler rules and the Content Signal policy, structured data, render mode, and machine readable endpoints like sitemap, OpenAPI and well known files. Every field is a fact read off a fetch, never a model opinion.

GEO / AEO Website Audit — AI Search Readiness

rtworule/ai-search-readiness-auditor

Audit public websites for GEO and AEO readiness: AI crawler access, robots.txt, llms.txt, sitemaps, Schema.org, social metadata, scores, and prioritized fixes.