llms.txt Generator: Turn Any Website into an LLM & RAG Pack (llms.txt, Markdown, Chunks)

Turn any website or docs site into a ready-to-use LLM pack in one run:

• llms.txt in the llmstxt.org format (site name, summary, and sections of linked pages with descriptions)
• llms-full.txt with every page as clean Markdown
• chunks.jsonl: RAG-ready chunks with URL, title and chunk metadata
• pages.jsonl: one clean Markdown page per line

One pack per site. It crawls sitemaps and links, extracts the main content (no menus, footers or cookie banners) and gives you the files.

• ✅ llms.txt generator: publish it at yoursite.com/llms.txt for AI search and AI assistants
• ✅ Website to Markdown for ChatGPT, Claude, custom GPTs, Gemini Gems and NotebookLM
• ✅ Vector DB ready: chunks split on headings and paragraphs, with overlap, for Pinecone, Qdrant, Chroma, pgvector and similar
• ✅ Whole site or one section: give a docs URL like /docs/ and it packs only that section
• ✅ Free error rows: offline, blocked or empty sites are never charged

Who it's for

• Site owners and SEOs: add an llms.txt for AI search (GEO/AEO) in minutes
• AI builders: feed docs into a RAG chatbot, a custom GPT or an agent
• Agencies: "make my client's site AI-ready" as a deliverable
• Researchers: a clean offline copy of a site's text

What you get (real output, shortened)

{ "host": "docs.python.org", "crawlScope": "samePath", "siteName": "Python documentation", "pagesIncluded": 18,
  "tokensEstimate": 81596, "chunkCount": 206, "sections": [ { "name": "Pages", "pages": 18 } ],
  "llmsTxtUrl": "https://api.apify.com/v2/key-value-stores/…/records/docs.python.org-3-tutorial-llms-txt",
  "llmsFullTxtUrl": "…", "chunksJsonlUrl": "…", "pagesJsonlUrl": "…", "chargedPacks": 1, "charged": true }

llms.txt preview:

# Python documentation
> Python is an easy to learn, powerful programming language. …
## Pages
- [The Python Tutorial](https://docs.python.org/3/tutorial/): Python is an easy to learn, powerful programming language…
- [1. Whetting Your Appetite](https://docs.python.org/3/tutorial/appetite.html): If you do much work on computers, eventually…

Try it in 1 minute

1. Add a homepage (whole site) or a section URL (https://docs.example.com/guide/).

2. Optional: max pages (default 50), chunk size, include/exclude URL patterns.

3. Run, then open the llms.txt, llms-full.txt and chunks.jsonl links in the output.

   { "websites": ["https://docs.python.org/3/tutorial/"], "maxPagesPerSite": 50, "chunkSize": 2000 }

Price

One llm-pack event per site per started 50 pages (1-50 pages = 1 pack, 51-100 = 2).

Apify plan	Per pack (up to 50 pages)
Free	$0.10
Bronze	$0.09
Silver	$0.08
Gold	$0.07

• No start fee from us. Only Apify's standard minimum start event.
• If no pack can be built (every site offline, blocked or empty), the run fails with a clear message and costs nothing.

All settings

Field	Default	What it does
websites	–	Homepages or section URLs, one pack each
maxPagesPerSite	50	Pages per pack (up to 5,000)
crawlScope	auto	Section URL = that section; homepage = whole site
useSitemaps	true	Also find pages via sitemap.xml
maxCrawlDepth	5	Clicks from the start URL
contentMode	article	Main content only, or the whole page minus nav
chunkSize / chunkOverlap	2000 / 200	RAG chunking (characters)
include/excludeUrlPatterns	–	Filter pages
maxChargedPacks	0	Hard stop (0 = none)

FAQ

What is llms.txt? It's a proposed standard: a Markdown file at /llms.txt that tells AI tools what a site is about and where its key pages are. JavaScript-only sites? Pages that render only in the browser may come back empty. Those are free error rows, never charged. Need just Markdown per page at scale? Use our Website to Markdown for LLM actor (charged per page).

Use it from code, no-code tools or AI agents

• API: POST https://api.apify.com/v2/acts/blond_terrarium~site-to-llm-pack/run-sync-get-dataset-items?token=YOUR_TOKEN
• Make, Zapier, n8n, webhooks through Apify integrations
• AI agents (MCP): https://mcp.apify.com?tools=blond_terrarium/site-to-llm-pack