AI Crawler Robots.txt Audit & llms.txt Checker

AI SEO Audit: robots.txt, llms.txt & 26 AI Crawlers

aiqlabs/seo-ai-visibility-auditor

Audits any URL for classic SEO health and for AI/LLM discoverability - AI crawler access in robots.txt, llms.txt, structured data, metadata - in a single pass.

AI Visibility Site Audit — robots.txt & llms.txt checks

43sunsets/ai-visibility-site-audit

Audit how visible your own website is to AI crawlers and assistants: robots.txt rules for 13 AI bots, llms.txt, structured data, meta hygiene and sitemap — with a scored report and a ready-to-edit llms.txt draft.

llms.txt Compliance Checker

zinin/llms-txt-auditor

Give it a domain. It checks whether the site publishes a valid llms.txt / llms-full.txt (the emerging AI-crawler-friendly-docs convention) and whether robots.txt blocks GPTBot, ClaudeBot, Google-Extended, PerplexityBot and 5 other AI crawlers. llms.txt-first, no browser, no LLM, no API key.

Deterministically audit which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and more) a site's robots.txt allows or blocks, plus llms.txt and sitemap declarations.

AI Crawler Access Checker

zinin/ai-crawler-access-checker

Check which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended & more) can access your website. Bulk audit of robots.txt rules, llms.txt presence and sitemap for AI search visibility (GEO/AEO).

AI Crawler Policy Audit API | 🤖 $1/1K

webdata_labs/ai-crawler-policy-audit-api

[💵 $1.00 / 1K] Audit robots.txt and llms.txt to see whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended and other AI crawlers are allowed.

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.

Fetch a public website's robots.txt and report which AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, CCBot, Bytespider, and more) are blocked, allowed, or defaulted. Returns an AI-visibility readiness score with recommendations for GEO and AI-SEO strategy.

📄 llms.txt Generator & Validator — AI Search Visibility

upstanding_biobot/llms-txt-generator

Generate llms.txt files for AI search visibility. Fetches sitemap, validates structure, checks AI bot access (OAI-SearchBot, PerplexityBot, ClaudeBot, Google-Extended). 2 blue ocean competitors only.

Ai Visibility Suite - Dark Visitors Alternative

alizarin_refrigerator-owner/ai-visibility-suite---dark-visitors-alternative

Comprehensive AI bot monitoring, robots.txt analysis, LLMs.txt generation & AI shopping optimization. Monitor AI crawlers visits, check AI compliance, generate AI-friendly configurations, and optimize for AI shopping agents. AI Bot Directory Robots.txt LLMs.txt AI Shopping Competitor AI Audit