llms.txt Generator + Docs Site to Markdown (RAG-ready)

Turn any documentation site into clean Markdown for LLM context, RAG pipelines and AI agents, and get a ready-made llms.txt and llms-full.txt for the site. Point it at a docs URL, get one dataset row per page with the main content converted to Markdown, plus optional token-sized chunks for vector databases.

Typical uses: feed a framework's docs to Cursor, Claude, ChatGPT or a custom agent; build a RAG index of a product's documentation; keep a local Markdown mirror of docs; generate an llms.txt for your own docs site.

How it works

• HTTP only (no browser): fetches pages and converts them with cheerio + Turndown. Fast and cheap. Sites that render content only with JavaScript in the browser are not supported (see Limitations).
• Discovery: reads robots.txt and sitemap.xml (including sitemap indexes) first; if the sitemap has nothing in scope it falls back to following links within the start URL's path prefix.
• Polite: identifies itself as WintersWorksDocsToLLMs/1.0 (+wintersworks.shop@gmail.com), respects robots.txt (disallowed pages are skipped), limits request rate per host.
• Clean content: strips navigation, sidebars, footers, breadcrumbs, table of contents and "edit this page" links, keeping the main article (main, article, .markdown, .theme-doc-markdown, etc., or your own CSS selector).

Input

Field	Description
startUrls	Docs URLs, e.g. https://docs.python.org/3/tutorial/. Pages under the same path prefix are converted.
maxPages	Cap on pages converted (and charged). Default 50.
includeGlobs / excludeGlobs	URL globs (* within a segment, ** anywhere).
sameDomainOnly, pathPrefixOnly	Scope controls (both default on).
useSitemap	Try sitemap.xml first. Default on.
contentSelector	Optional CSS selector for the main content.
includeMarkdown	Put the Markdown in each dataset row. Default on.
chunkSize, chunkOverlap	If chunkSize > 0, also output RAG chunks (approx tokens, chars/4), split on paragraphs, each with its nearest heading.
maxConcurrency	Parallel requests (default 4, max 10).
respectRobotsTxt	Default on.

Example:

{ "startUrls": ["https://docs.apify.com/sdk/js/docs"], "maxPages": 15, "chunkSize": 500, "chunkOverlap": 50 }

Output

One dataset item per page: url, title, description, headings (outline of h1-h4), markdown, wordCount, approxTokens (characters / 4, an estimate, not a real tokenizer count), crawledAt, and chunks (index, heading, tokens, text) when enabled.

{ "url": "https://docs.apify.com/sdk/js/docs/overview", "title": "Apify SDK for JavaScript",
  "headings": [{"level":1,"text":"Apify SDK for JavaScript"},{"level":2,"text":"Quick start"}],
  "wordCount": 242, "approxTokens": 410, "markdown": "# Apify SDK for JavaScript\n\n..." }

The run also saves llms.txt (site title, summary, list of page links with one-line descriptions) and llms-full.txt (all pages' Markdown concatenated, each prefixed with its source URL) to the key-value store. Their URLs are logged and listed in the OUTPUT record. With several start URLs you get one pair of files per start URL.

Pricing

Pay per event: $0.001 per page converted ($1 per 1,000 pages). Pages that fail, are blocked by robots.txt, or have no content are not charged. llms.txt and llms-full.txt are free. The run stops cleanly when your spending limit is reached.

FAQ

Does it work for GitBook, Docusaurus, MkDocs, Sphinx, ReadTheDocs? These render server-side HTML and are the main target; tested on a Docusaurus site and the Sphinx-built Python docs. Results vary by theme; use contentSelector if the default misses. Can I get only part of a site? Use a deeper start URL, or include/exclude globs. Are the token counts exact? No: characters / 4, an estimate. Is the llms.txt official? It follows the llms.txt proposal format (title, summary, link list) but descriptions come from each page's meta description.

Limitations

• No JavaScript rendering: pages whose content is built client-side (some SPA docs) will come out empty and are skipped (not charged).
• No login or paywalled docs. Respect each site's terms of use and copyright; you are responsible for how you use the converted content.
• Images are kept as absolute links, not downloaded. Very large tables and unusual widgets may convert imperfectly.
• Sitemaps listing multiple doc versions can include old versions within your path prefix; narrow the start URL or use excludeGlobs.

Is it legal? (public data only)

Yes, in the sense that it only reads public or official data: public documentation pages you point it at (respecting robots.txt). It does not log in, bypass paywalls or access private data. You remain responsible for how you use the results, including privacy, copyright and marketing laws. Not affiliated with or endorsed by the data sources.

Quick start and cost

Run it with the prefilled input to see sample output in seconds; the default run costs only a few cents at most. Then adjust the input and schedule it.

Use from code and AI agents

Call it from any language with the Apify API, or let an AI agent find and run it through the Apify MCP server.

from apify_client import ApifyClient
run = ApifyClient("<YOUR_APIFY_TOKEN>").actor("wintersworks/docs-to-markdown-llms-txt").call(run_input={"startUrls": ["https://docs.apify.com/sdk/js/docs"], "maxPages": 10})
items = ApifyClient("<YOUR_APIFY_TOKEN>").dataset(run["defaultDatasetId"]).list_items().items

Results land in the run's dataset (JSON, CSV, Excel). Set a maximum charge per run in the run options to cap spend.

About

Built and maintained by Winters Works. Questions or feature requests: open an issue on the Actor page or email wintersworks.shop@gmail.com.