LLMs.txt Discovery

LLMs.txt Scraper - AI Documentation Index Data

benthepythondev/llms-txt-scraper

Scrape llms.txt files and extract AI-readable documentation links, headings, titles and source metadata.

Crawl any website and generate llms.txt and llms-full.txt files following the llms.txt standard. BFS crawler with URL filtering and markdown/plaintext output. For AI readiness, SEO for AI, documentation portals, and archiving.

llms-txt-converter is an Apify Actor that crawls any documentation site and outputs the two GEO-standard files (llms.txt and llms-full.txt) used by ChatGPT, Claude, Perplexity, Gemini, and Bing Copilot to find and cite your docs.

Check websites for llms.txt and llms-full.txt, extract sections and links for AI readiness and GEO audits.

Crawl a website and generate llms.txt + llms-full.txt for AI/LLM discoverability and RAG pipelines. Batch, API-callable, pay-per-page.

Audit a website for /llms.txt and /llms-full.txt AI discovery readiness. Get a readiness score, markdown structure summary, and actionable recommendations. Built for GEO/AI-SEO consultants, documentation teams, and devrel.

LLMs.txt Generator

onescales/the-llms-txt-generator

The most powerful LLMs.txt Generator tool online. Generates LLMs.txt , llms-full.txt and markdown .md files within seconds! Get your website discovered, and recommended by ChatGPT, Claude, Google Gemini, Perplexity, Grok, and every AI. (Great for AEO, AIO, GEO, AI SEO) Made by Hi LLMs

Crawl a website and generate an llms.txt file (the emerging standard that helps LLMs and AI agents understand your site). Optionally build llms-full.txt with full page content. Output saved to the key-value store.

AI Visibility Site Audit — robots.txt & llms.txt checks

43sunsets/ai-visibility-site-audit

Audit how visible your own website is to AI crawlers and assistants: robots.txt rules for 13 AI bots, llms.txt, structured data, meta hygiene and sitemap — with a scored report and a ready-to-edit llms.txt draft.