LLMs.txt & llms-full.txt Generator for Any Website

Crawl a website and generate llms.txt + llms-full.txt for AI/LLM discoverability and RAG pipelines. Batch, API-callable, pay-per-page.

Crawl any website and generate llms.txt and llms-full.txt files following the llms.txt standard. BFS crawler with URL filtering and markdown/plaintext output. For AI readiness, SEO for AI, documentation portals, and archiving.

Crawl a website and generate an llms.txt file (the emerging standard that helps LLMs and AI agents understand your site). Optionally build llms-full.txt with full page content. Output saved to the key-value store.

LLMs.txt Generator

onescales/the-llms-txt-generator

The most powerful LLMs.txt Generator tool online. Generates LLMs.txt , llms-full.txt and markdown .md files within seconds! Get your website discovered, and recommended by ChatGPT, Claude, Google Gemini, Perplexity, Grok, and every AI. (Great for AEO, AIO, GEO, AI SEO) Made by Hi LLMs

llms.txt Checker, Generator & Monitor: AI Search Readiness.

apifmcpfactory/llms-txt-suite

Generate, audit and monitor your llms.txt so ChatGPT and AI agents can read your site. Get an AI-readiness score and a ready-to-publish file. Works on public pages only. We store nothing personal.

Documentation Crawler for RAG

liquid_bark/docs-crawler-for-rag

Specialized crawler for developer documentation sites. Detects frameworks (Docusaurus, GitBook, ReadTheDocs, MkDocs, Sphinx), extracts clean content, and outputs semantically chunked Markdown optimized for RAG pipelines.

SEO, GEO & AEO Audit — AI Search Readiness Checker

fayoussef/seo-geo-aeo-audit

Audit any website for Google SEO and AI search visibility in one run. Get 0–100 SEO, GEO & AEO scores per page, AI-crawler & llms.txt checks, schema validation, and a prioritized fix list with a shareable HTML report. No API keys needed — just enter a URL.

AI Visibility Site Audit — robots.txt & llms.txt checks

43sunsets/ai-visibility-site-audit

Audit how visible your own website is to AI crawlers and assistants: robots.txt rules for 13 AI bots, llms.txt, structured data, meta hygiene and sitemap — with a scored report and a ready-to-edit llms.txt draft.

Check websites for llms.txt and llms-full.txt, extract sections and links for AI readiness and GEO audits.

LLMs.txt Checker - AI Readiness Scanner

alizarin_refrigerator-owner/llms-txt-checker

Batch check websites for llms.txt files & identify AI optimization opportunities. A emerging standard that websites use to communicate with AI systems. It helps LLMs understand, What the company does, How to present information about the business, Contact information & key pages Brand guidelines