llms.txt Generator - Make Your Site LLM-Readable
Pricing
Pay per event
Generate a ready-to-upload llms.txt for any website. Crawl from a start URL or read an XML sitemap, group pages into sections and build the file from each page's own title and meta description. One dataset row per page so you can review before publishing.
Page where the crawl starts, usually the home page or the docs root. Only links on the same site (same origin) are followed. Provide this or a sitemap URL.
An XML sitemap (or sitemap index, followed one level deep). When set, pages come from its entries instead of following links, and the start URL is not crawled.
Maximum number of pages processed in one run (fetched pages plus pages skipped because robots.txt disallows them). Extra URLs are dropped with a warning.
How many clicks away from the start URL the crawl may go. 0 fetches the start page only. Ignored in sitemap mode.
Optional. When set, only URLs whose path (including the query string) contains at least one of these substrings are kept, for example /docs/ or /blog/.
Optional. URLs whose path (including the query string) contains any of these substrings are skipped, for example /tag/ or ?page=. Applied after the include patterns.
Optional. Title used for the llms.txt H1 heading. Default: og:site_name of the start page, otherwise its hostname.
Optional. Short summary placed in the llms.txt blockquote. Default: the meta description of the start page; the blockquote is omitted when there is none.
When enabled, robots.txt is read once per site and disallowed pages are not fetched (they get a skipped row). Disable only for sites you own.
