Website AI Content & Data Leakage Scanner

Find public llms.txt, llms-full.txt, and AI-readable documentation links for company domains.

Crawl a website and generate llms.txt + llms-full.txt for AI/LLM discoverability and RAG pipelines. Batch, API-callable, pay-per-page.

Crawl any website and generate llms.txt and llms-full.txt files following the llms.txt standard. BFS crawler with URL filtering and markdown/plaintext output. For AI readiness, SEO for AI, documentation portals, and archiving.

AI Bot & LLM Crawler Exposure Auditor

glowing_glove/ai-bot-llm-crawler-exposure-auditor

Audit robots.txt, llms.txt, AI crawler directives, public content exposure, and policy gaps for websites in the AI search era.

Generate an llms.txt draft and priority AI-readable link inventory from public sitemaps.

LLMs.txt Scraper - AI Documentation Index Data

benthepythondev/llms-txt-scraper

Scrape llms.txt files and extract AI-readable documentation links, headings, titles and source metadata.

Check robots.txt, llms.txt, sitemap hints, and AI crawler blocking rules for websites and domains.

Audit domains for llms.txt, robots.txt AI bot rules, sitemap hints, and public AI crawler readiness signals.

llms.txt Generator: Make Any Website AI-Readable

apifmcpfactory/llms-txt-generator

Generates a spec-compliant llms.txt file for any website from the site's own pages and metadata (no AI invention) - the fix when an AI-readiness audit finds llms.txt missing; $0.02 per site.

Crawl a website and generate an llms.txt file (the emerging standard that helps LLMs and AI agents understand your site). Optionally build llms-full.txt with full page content. Output saved to the key-value store.