robots.txt & llms.txt Monitor - Crawler & AI Access Rules

LLMs.txt AI Discovery Auditor

glowing_glove/llms-txt-ai-discovery-auditor

Check websites for llms.txt, llms-full.txt, robots AI crawler rules, sitemap hints, and AI search readiness gaps.

Check whether AI agents can use a website: robots.txt AI-agent blocks, llms.txt, JSON-LD structured data, sitemap and summarisation signals.

AI Crawler Policy Checker: robots.txt & llms.txt

accountable_eel/ai-crawler-policy-lookup

Check what a website's robots.txt says about AI crawlers (GPTBot, ClaudeBot, Google-Extended, PerplexityBot, and more) and whether it publishes an llms.txt. Reads the site's own public files directly: no API key, no third-party service. Charged only when an explicit AI policy or llms.txt is found.

Find public llms.txt, llms-full.txt, and AI-readable documentation links for company domains.

llms.txt Auditor & AI Crawler Policy Checker

zinin/llms-txt-auditor

Audit public llms.txt, llms-full.txt, and root robots.txt rules for nine named AI crawlers. Get evidence-backed presence, policy, confidence, gaps, and review actions for each complete domain audit.

AI Crawler Robots.txt Audit & llms.txt Checker

davidbenittah/ai-crawler-robots-audit

Check robots.txt and llms.txt on up to 50 sites and see which AI crawlers are blocked: OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-SearchBot, GPTBot and ClaudeBot. Returns a 0-100 AI search access score, the rule matched per bot, and a paste-ready robots.txt fix.

Keyless, plain-HTTP compliance auditor. Given a list of sites, it fetches and parses robots.txt, security.txt (RFC 9116), ads.txt, and humans.txt into one structured audit row per site — AI-crawler posture, security contacts/expiry, and ad-tech supply-chain record counts. No API key needed.

Sitemap & llms.txt URL Extractor

noclat/sitemap-llms-txt-extractor

Get every URL a website publishes. Reads robots.txt, follows all sitemaps including sitemap indexes and gzipped files, probes the usual sitemap paths when robots.txt names none, and reads llms.txt and llms-full.txt. Filter by pattern or last-modified date.

Track which AI crawlers each website blocks or allows (GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, Bytespider and more) plus whether it adopted the llms.txt standard. One clean row per domain from public robots.txt and llms.txt. Zero-config default run, no login, no PII.

Check websites for llms.txt and llms-full.txt, extract sections and links for AI readiness and GEO audits.