llms.txt Generator (llmstxt.org)

Give it a website; get a ready-to-publish llms.txt and llms-full.txt that follow the llmstxt.org proposal, so AI assistants and agents can find and read your best content. Polite HTTP-only crawl (no browser), robots.txt and sitemap aware.

What you get

• llms.txt: # Site name, > summary (from the home page meta description), then ## Section lists of - [Page title](url): description, grouped by first URL path segment; legal/utility pages go under ## Optional.
• llms-full.txt: the same header plus the full main content of each page converted to Markdown (nav/footer/scripts stripped, size capped).
• Validation: the generated llms.txt is checked against the format (H1, blockquote, H2 lists, link syntax); problems land in specProblems.
• Extras: tells you whether the site already publishes /llms.txt.
• Files are saved in the key-value store (<host>-llms.txt, <host>-llms-full.txt) and also returned as strings in the dataset.

Input

Field	Default	Description
urls	required	Up to 50 sites or domains.
maxPagesPerSite	30	Pages fetched per site (max 200). Billed per page.
useSitemap	true	Use robots.txt Sitemap: / /sitemap.xml (index and .gz supported); falls back to link crawling.
maxDepth	2	Link-crawl depth when there is no sitemap.
includeFullText	true	Build llms-full.txt.
maxCharsPerPage	20000	Per-page cap in llms-full.txt.
delayMs / concurrency	500 / 3	Politeness controls.
respectRobots	true	Obey robots.txt.

{ "urls": ["https://example.com"], "maxPagesPerSite": 50 }

Output (dataset, one item per site)

{ "type": "llms", "siteOrigin": "https://htmx.org", "siteName": "</> htmx", "pagesIncluded": 10, "usedSitemap": true,
  "urlsDiscovered": 188, "existingLlmsTxt": false, "specProblems": [], "llmsTxt": "# </> htmx\n\n> ...", "llmsFullTxt": "...",
  "llmsTxtFile": "htmx.org-llms.txt", "skipped": { "robots": 0, "error": 0, "noindex": 0, "nonHtml": 0, "duplicateCanonical": 1 } }

Pages with noindex, HTTP errors, non-HTML content, robots-disallowed paths and duplicate content are left out.

Pricing (pay per event; set in Apify Console)

Event	When	Planned price
apify-actor-start	once per run	~$0.03
page-processed	each page fetched and included	~$0.005

Charging only happens on the Apify platform. Compare: hand-writing llms.txt, or monthly SaaS generators.

Use via API / MCP

Call it from the Apify API, the Apify MCP server, or any agent framework; the output is plain text in llmsTxt, ready to save at https://yoursite/llms.txt. Review the result before publishing: the generator picks pages heuristically.

Politeness and safety

Identifies as LlmsTxtGen; obeys robots.txt; at most 2 requests in flight per host with delayMs spacing; pages fetched sequentially per site; only http/https; localhost, internal hosts and private/reserved IPs are refused (at connect time and on every redirect). No JavaScript execution: client-rendered sites yield little content. Public pages only.

Run locally

npm install && npm test
mkdir -p storage/key_value_stores/default
echo '{"urls":["https://example.com"],"maxPagesPerSite":5}' > storage/key_value_stores/default/INPUT.json
npm start     # dataset in storage/datasets/default/, files in storage/key_value_stores/default/

Tested locally on example.com, sqlite.org (link crawl), htmx.org (sitemap, 188 URLs). Docker: docker build -t llms-txt-generator . && docker run --user 996 -v <dir>:/usr/src/app/storage llms-txt-generator.