llms.txt Generator: crawl a site, write llms.txt and full text

Give it one or more sites. It crawls each one up to the page limit you set and writes an llms.txt file in the format of the llms.txt proposal [1], ready to place at the root of the site. On request it also writes llms-full.txt with the main text of every listed page as Markdown.

The file it writes

The layout follows the proposal [1]:

• An H1 with the site name. The only required part.
• A blockquote with a short summary of the site.
• H2 sections, each a Markdown list of links in the form [title](url): description.
• An Optional section last, for links an agent can skip when it needs a shorter context.

The site name comes from the home page's og:site_name, or the first part of its title. The summary is the home page's meta description. You can set both yourself. Page titles drop the site suffix repeated on every page (Download Python | Python.org becomes Download Python). Each description is the page's meta description, or its first paragraph when the meta description is missing or the same on many pages. Pages are grouped by the first part of their path: two or more pages under /docs/ make a Docs section, and single pages go under Pages. Pages matching your Optional patterns go in the Optional section.

llms-full.txt is an extra file, not part of the proposal. It starts with the same H1 and summary, then gives each page as an H2 with its source URL and its main text, with menus, headers and footers left out. Headings inside a page move down two levels so the page titles stay the top level.

How it crawls

• Reads robots.txt first and skips every page it disallows. It waits at least a quarter second between requests to the same site.
• Takes pages from the site's XML sitemaps first (those named in robots.txt, or /sitemap.xml), then follows links on the pages it reads.
• Stays on the start URL's host, skips links to files such as PDF, images and archives, and skips pages marked noindex.
• Include, exclude and Optional patterns are regular expressions matched against the page URL.

Input

• Sites (required): one start URL per site
• Maximum pages per site (default 50)
• Use the sitemap (default on)
• Also write llms-full.txt (default off)
• Only include URLs matching, Exclude URLs matching, Put in the Optional section (regular expressions)
• Site name and summary (optional overrides)
• Timeout per request (default 20 seconds)

Output

Key value store: llms-<host>.txt and, when asked, llms-full-<host>.txt, as UTF-8 text.

Dataset: one item per site with startUrl, ok, pageCount, sections (pages per section), llmsTxtUrl, llmsFullTxtUrl, llmsFullTxtCharacters, llmsTxt (the file's text), pages (each with url, title, description and llmsSection), skipped (pages left out by robots.txt, noindex, your filters, or because they were not HTML or failed) and fetchErrors.

A site whose start page cannot be loaded gets ok: false and an error, and the rest of the run carries on. Export as JSON, CSV or Excel, or read it over the API.

Pricing

Pay per event: one event per page listed in an llms.txt file. Pages skipped or failed are not charged. Set the maximum pages per site to control cost. When the run reaches the maximum charge you set, the file for that site lists only the pages within it.

Scope and conduct

Only public pages are read, without a browser, so content that a site builds with JavaScript after loading is not seen. The actor does not log in anywhere and collects no personal data. Generate files for sites you run or have permission to describe.

Support

Questions and problems: the Issues tab on this page. Replies are written by an AI assistant on behalf of the developer and reviewed by the developer.

References

[1] llms-txt, The /llms.txt file, v2. https://llmstxt.org/