llms.txt Generator + Docs to Markdown & RAG Chunks

Turn a public documentation site into three things an LLM pipeline can use straight away:

1. Clean Markdown per page (dataset): navigation, sidebars, footers and tables of contents removed; fenced code blocks with their language; tables; absolute links; title, breadcrumb path, heading outline, word and token counts, last-modified date and a content hash per page.
2. RAG chunks (chunks.json): heading-aware chunks with a token estimate, the heading path and a stable id, never split inside a code block.
3. llms.txt and llms-full.txt: an index of the site in the llmstxt.org format, plus the full text of every page in one file.

It crawls politely by construction: same origin only, optional path prefix, robots.txt obeyed (Disallow and Crawl-delay), rate-limited per host, no logins, no paywalls, no personal data.

Built and maintained by Lathesmith with AI assistance.

Who it is for

• Teams building RAG or agent memory over a product's docs (their own or a vendor's) who want chunks with provenance, not raw HTML.
• Maintainers who want to publish llms.txt for their own documentation without writing a crawler.
• Developers who want a Markdown mirror of a docs site for offline search, diffing between releases (use contentHash and lastModified), or feeding a coding assistant.
• AI agents that need "give me this library's docs as Markdown" as a tool call (see "Use from an AI agent (MCP)").

How to use it

1. Enter one or more Start URLs, for example https://crawlee.dev/js/docs/quick-start.
2. Optionally set a Path prefix such as /js/docs so the crawl stays inside the docs and does not wander into the blog or the marketing pages.
3. Leave Max pages at 200 for a first run (the hard cap is 2,000 per run; split larger sites by path prefix).
4. Run it. Results appear in the run's Dataset (one item per page) and Key-value store (llms.txt, llms-full.txt, chunks.json, OUTPUT).

The defaults suit static documentation generators: Docusaurus, MkDocs and Material, Sphinx and Read the Docs, VitePress, GitBook, Mintlify, Nextra and plain HTML docs. Pages are fetched over plain HTTP, without a browser (see "How pages are fetched").

Input

Field	Default	What it does
startUrls	required	Public documentation pages to start from (up to 20). The crawl never leaves their origin.
pathPrefix	empty	Only crawl URLs whose path starts here, e.g. /js/docs.
maxPages	200	Stop after this many pages have been converted (cap 2,000). Only converted pages are billed.
maxDepth	20	How many links away from a start URL the crawler may go.
useSitemap	true	Seed the queue from sitemap.xml (after the same filters) so coverage does not depend on navigation links.
excludeUrlPatterns	[]	Regular expressions; matching URLs are skipped.
useDefaultExcludes	true	Skip login, sign-up, account, profile, user, cart, checkout, admin and search URLs.
ignoreQueryParams	true	Treat /page?x=1 as /page. Tracking parameters are always dropped.
requestsPerMinute	120	Rate limit (2 requests/second). Cannot exceed 240. A robots.txt Crawl-delay lowers it further.
proxyConfiguration	none	Optional Apify Proxy. Not needed for public docs; proxy usage is billed by Apify separately.
chunkTokens	512	Upper bound per RAG chunk (128 to 2,048).
includeChunksInDataset	false	Also embed each page's chunks in its dataset item.
includeLlmsFull	true	Write llms-full.txt as well as llms.txt.
siteName, siteDescription	detected	Override the H1 and summary of llms.txt.

Minimal input:

{
    "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }],
    "pathPrefix": "/js/docs",
    "maxPages": 200
}

Output

Dataset: one item per page

{
    "url": "https://crawlee.dev/js/docs/quick-start",
    "canonicalUrl": "https://crawlee.dev/js/docs/quick-start",
    "title": "Quick Start",
    "description": "With this short tutorial you can start scraping with Crawlee in a minute or two.",
    "lang": "en",
    "siteName": "Crawlee",
    "headingPath": ["Introduction", "Quick Start"],
    "headings": [
        { "level": 1, "text": "Quick Start" },
        { "level": 2, "text": "Choose your crawler", "id": "choose-your-crawler" }
    ],
    "markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two...\n\n```bash\nnpx crawlee create my-crawler\n```",
    "wordCount": 612,
    "tokenEstimate": 910,
    "lastModified": "2026-09-20T10:00:00.000Z",
    "contentHash": "3f1c6c3f6c0d8c1e...",
    "chunkCount": 4,
    "depth": 0,
    "statusCode": 200,
    "crawledAt": "2026-10-06T19:02:11.431Z"
}

lastModified comes from page metadata (article:modified_time, JSON-LD dateModified, a "last updated" element) or the HTTP Last-Modified header, and is null when the site exposes none.

Key-value store

Record	Content
llms.txt	# Site name, a > summary, then ## Section lists of - [Title](url): description entries. Sections follow the URL structure under the path prefix; changelogs, blogs and legacy versions go to ## Optional.
llms-full.txt	Every page's Markdown in crawl order, each introduced by its title and a Source: line.
chunks.json	Array of { id, url, title, headingPath, text, tokens, index, total }. If the array would exceed 8 MB it is split into chunks-part-NN.json records and chunks.json lists the parts.
OUTPUT	Run summary: pages processed, failed and skipped, chunk and token totals, duration, pages per minute, why the crawl stopped (stopReason), robots.txt status per origin, the effective rate limit, which events were charged, memory use (peak resident memory and heap), and the page converter's limits and timings (converter).

Example llms.txt:

# Crawlee for JavaScript

> Crawlee is a web scraping and browser automation library for Node.js.

This file lists the 20 documentation pages crawled from this site, each converted to Markdown. The full text of every page is in llms-full.txt.

## Overview

- [Quick Start](https://crawlee.dev/js/docs/quick-start): With this short tutorial you can start scraping with Crawlee in a minute or two.
- [Introduction](https://crawlee.dev/js/docs/introduction): Crawlee covers your crawling and scraping end-to-end.

## Guides

- [Request Storage](https://crawlee.dev/js/docs/guides/request-storage): Crawlee has several request storage types that are useful for specific tasks.

Example chunk:

{
    "id": "3f1c6c3f6c0d-2",
    "url": "https://crawlee.dev/js/docs/quick-start",
    "title": "Quick Start",
    "headingPath": ["Quick Start", "Choose your crawler"],
    "text": "## Choose your crawler\n\nCrawlee comes with three main crawler classes: `CheerioCrawler`, `PuppeteerCrawler` and `PlaywrightCrawler`...",
    "tokens": 287,
    "index": 2,
    "total": 4
}

How pages are fetched

The Actor fetches the HTML the server sends and parses it once with Cheerio (on Crawlee's HttpCrawler); it does not run a browser. That covers Docusaurus, MkDocs, Sphinx, VitePress, GitBook, Mintlify, Nextra, Hugo, Jekyll and any other server-rendered documentation, and it keeps runs fast and cheap. Measured on the Apify platform at the default 1,024 MB and the default limit of 2 requests per second: 200 pages of the Crawlee docs in about 2 minutes (~90 pages per minute), and 2,000 pages of the Python documentation (large pages) in about 34 minutes (~60 pages per minute). Short runs pay a fixed cost for robots.txt and sitemap fetches. Memory stays flat as the crawl grows: only a small index of the pages is kept in memory, and the output files are assembled from local disk at the end. The 2,000-page Python run peaked at about 600 MB. At 512 MB, typical documentation still fits, but very heavy sites stop early with all output files written for the pages converted so far (see "Limits and behaviour").

Documentation whose body is injected only by client-side JavaScript (some single-page-app portals) comes back empty; such pages are reported under skipped.empty in OUTPUT and are not billed. A browser-rendering option is on the roadmap; if you need it for a specific site, say so in the Issues tab.

Pricing

This Actor uses pay-per-event pricing. You pay a small fee when a run starts and a per-page fee for every page that is delivered to the dataset. Pages that fail, are blocked by robots.txt, are outside the scope, or contain no content are not charged. The current prices are shown in the Pricing tab of this page; a 200-page run costs 200 times the page price plus one run-start fee, and nothing else: the Actor's own compute is included.

Set a maximum total charge on the run if you want a hard ceiling; the Actor lowers maxPages to fit it and stops cleanly when it is reached.

Limits and behaviour

• Scope: same origin as the start URLs; optional path prefix; up to 2,000 pages and 20 start URLs per run. For larger sites, run once per section.
• Crawl order: the start URLs first, then the pages around them (under the start URL's directory, e.g. /js/docs/ for /js/docs/quick-start, in the same docs version), found through links and the sitemap. The rest of the site (blog, other sections, old docs versions) is crawled only if maxPages is not used up by then. The llms.txt summary line comes from the start page, or the site's home page, never from another page.
• Very large pages: a page whose HTML is larger than about 7.9 MB at the default 1,024 MB of memory (about 3.8 MB at 512 MB; the limit grows with the run's memory) is skipped, logged and not billed; it is counted under skipped.tooLarge in OUTPUT.
• Very deeply nested pages: a page whose HTML nests elements more than 600 levels deep (real documentation nests far less) is skipped, logged and not billed; it is counted under skipped.tooDeep in OUTPUT.
• Per-page time and memory limits: each page is converted in a separate worker with a time limit (30 seconds at 1,024 MB of memory or more, longer on smaller runs, up to 2 minutes at 256 MB) and its own memory budget. Real documentation pages take well under a second. A page that hits either limit is skipped, logged and not billed, and the crawl carries on; it is counted under skipped.timeout or skipped.outOfMemory in OUTPUT (a page the converter fails on for another reason is counted under skipped.conversionError).
• robots.txt: always honoured. Disallow rules skip pages; Crawl-delay lowers the rate limit. If robots.txt returns a server error the run stops before crawling, as RFC 9309 requires. The crawler identifies itself as LathesmithDocsCrawler (User-Agent Mozilla/5.0 (compatible; LathesmithDocsCrawler/0.1; Apify Actor; +https://apify.com/store)), so site owners can address it in robots.txt.
• Rate limit: 120 requests per minute by default, at most 240, with low concurrency. It cannot be raised above that.
• Early stop: if memory use nears the run's memory limit or the run's timeout approaches, the crawl stops, still writes every file from the pages converted so far, and records why in OUTPUT.stopReason. Pages that were not converted are not billed.
• No access control bypass: no logins, cookies, CAPTCHA solving or paywall circumvention. Pages that need a session come back as errors or empty and are not billed.
• No personal data: the Actor extracts documentation text only. Login, account, profile, user, cart and admin URLs are excluded by default.
• Versioned docs: sitemaps often list every old version (/docs/3.10/..., /v2/...). Current, unversioned pages are queued first and old versions go to the ## Optional section of llms.txt; to leave them out entirely, add an exclusion such as /docs/\d or /v\d+/.
• Content extraction is heuristic. It recognises the main content area of the common documentation generators and falls back to the largest text block. Unusual layouts can leave a little chrome in or drop a sidebar note; use excludeUrlPatterns and report layouts that go wrong in the Issues tab.
• Token estimates are estimates (about cl100k, within roughly 15%), not a tokenizer.
• Legal: use it on documentation you own or are allowed to copy. You are responsible for complying with the target site's terms and copyright law; the Actor only helps you stay within robots.txt and rate limits.

Use from an AI agent (MCP)

Apify Actors can be called as tools from MCP-compatible agents and assistants through the Apify MCP server, and from code through the Apify API and SDKs. Ask your agent to run this Actor with an input like:

{ "startUrls": [{ "url": "https://docs.example.com/docs/" }], "pathPrefix": "/docs", "maxPages": 100 }

and read llms.txt or chunks.json from the run's key-value store, or the dataset items for per-page Markdown. See Apify's documentation on the MCP server and the API for setup details.

Integrations

Like every Actor, this one works with Apify's integrations: schedules (re-crawl weekly and diff contentHash), webhooks, and exports to JSON, CSV, Excel or XML. Dataset items are also readable directly by LangChain and LlamaIndex Apify loaders.

Support

Open an issue in the Issues tab with the start URL and the run id; issues are answered in writing. Replies may be drafted with AI assistance and are reviewed before posting.