llms.txt Generator & Auditor

Make any site readable by AI assistants — generate a spec-shaped llms.txt in one run, and audit an existing one for coverage and correctness.

llms.txt is the emerging convention that tells AI crawlers what a site is and which pages matter, the way robots.txt tells search crawlers what to skip. Anthropic, Cloudflare and Perplexity already publish one. Most sites do not, and hand-writing it is tedious and goes stale the moment you publish.

What it does

Generate — discovers your pages (sitemap first, including sitemap indexes; falls back to following same-origin links), pulls each page's title and description, groups them into sections by site structure, and writes a valid llms.txt to the key-value store.

Audit — fetches your existing llms.txt and reports:

• coverage % — how many published pages are actually listed
• missing URLs — pages you publish but never told the AI about
• stale URLs — entries pointing at pages that no longer exist
• spec issues — missing # title line, missing > summary, no links at all

llms-full.txt (optional) — the long-form companion with each page's full content as clean markdown: the file an assistant ingests directly. Headings, code blocks with language tags, tables, lists and links are preserved; nav, footers, cookie banners and ad slots are stripped.

Input

{
  "siteUrl": "https://docs.example.com",
  "mode": "both",
  "maxPages": 200,
  "includePatterns": ["/docs/", "/guides/"],
  "excludePatterns": ["/tag/", "/author/"],
  "generateFullText": true
}

Output

A result row (site name, discovery method, page counts, the generated file, the full audit), one page row per page (url, title, description, language, word count, markdown), and an error row for anything that failed. The files themselves are saved to the key-value store as llms.txt and llms-full.txt, ready to upload to your web root.

Why this is reliable

It reads your own site. There is no anti-bot, no login, no rate-limit war and no third party that can cut you off — which is not true of most scrapers. If every page fails to load the run fails loudly rather than handing you an empty file that looks like a result.

Typical uses

Publishing llms.txt for a docs site or product site · keeping it current as a scheduled job so it never drifts from what you actually publish · auditing client sites as an agency deliverable · producing clean markdown corpora for your own RAG pipeline.