# llms.txt Generator + Docs to Markdown & RAG Chunks (`lathesmith/docs-to-llms-markdown`) Actor Generate llms.txt and llms-full.txt for a public documentation website, plus clean LLM-ready Markdown per page and heading-aware RAG chunks with token estimates, in one run. Polite docs crawler/scraper: same-origin, robots.txt, rate limits, no logins. Pay per page. Callable by AI agents via MCP. - **URL**: https://apify.com/lathesmith/docs-to-llms-markdown.md - **Developed by:** [Lathesmith](https://apify.com/lathesmith) (community) - **Categories:** AI, Developer tools, SEO tools - **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.50 / 1,000 pages This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## llms.txt Generator + Docs to Markdown & RAG Chunks Turn a public documentation site into three things an LLM pipeline can use straight away: 1. **Clean Markdown per page** (dataset): navigation, sidebars, footers and tables of contents removed; fenced code blocks with their language; tables; absolute links; title, breadcrumb path, heading outline, word and token counts, last-modified date and a content hash per page. 2. **RAG chunks** (`chunks.json`): heading-aware chunks with a token estimate, the heading path and a stable id, never split inside a code block. 3. **`llms.txt` and `llms-full.txt`**: an index of the site in the [llmstxt.org](https://llmstxt.org/) format, plus the full text of every page in one file. It crawls politely by construction: same origin only, optional path prefix, robots.txt obeyed (Disallow and Crawl-delay), rate-limited per host, no logins, no paywalls, no personal data. Built and maintained by Lathesmith with AI assistance. ### Who it is for - Teams building **RAG or agent memory** over a product's docs (their own or a vendor's) who want chunks with provenance, not raw HTML. - Maintainers who want to **publish `llms.txt`** for their own documentation without writing a crawler. - Developers who want a **Markdown mirror** of a docs site for offline search, diffing between releases (use `contentHash` and `lastModified`), or feeding a coding assistant. - **AI agents** that need "give me this library's docs as Markdown" as a tool call (see "Use from an AI agent (MCP)"). ### How to use it 1. Enter one or more **Start URLs**, for example `https://crawlee.dev/js/docs/quick-start`. 2. Optionally set a **Path prefix** such as `/js/docs` so the crawl stays inside the docs and does not wander into the blog or the marketing pages. 3. Leave **Max pages** at 200 for a first run (the hard cap is 2,000 per run; split larger sites by path prefix). 4. Run it. Results appear in the run's **Dataset** (one item per page) and **Key-value store** (`llms.txt`, `llms-full.txt`, `chunks.json`, `OUTPUT`). The defaults suit static documentation generators: Docusaurus, MkDocs and Material, Sphinx and Read the Docs, VitePress, GitBook, Mintlify, Nextra and plain HTML docs. Pages are fetched over plain HTTP, without a browser (see "How pages are fetched"). ### Input | Field | Default | What it does | |---|---|---| | `startUrls` | required | Public documentation pages to start from (up to 20). The crawl never leaves their origin. | | `pathPrefix` | empty | Only crawl URLs whose path starts here, e.g. `/js/docs`. | | `maxPages` | 200 | Stop after this many pages have been converted (cap 2,000). Only converted pages are billed. | | `maxDepth` | 20 | How many links away from a start URL the crawler may go. | | `useSitemap` | true | Seed the queue from `sitemap.xml` (after the same filters) so coverage does not depend on navigation links. | | `excludeUrlPatterns` | `[]` | Regular expressions; matching URLs are skipped. | | `useDefaultExcludes` | true | Skip login, sign-up, account, profile, user, cart, checkout, admin and search URLs. | | `ignoreQueryParams` | true | Treat `/page?x=1` as `/page`. Tracking parameters are always dropped. | | `requestsPerMinute` | 120 | Rate limit (2 requests/second). Cannot exceed 240. A robots.txt `Crawl-delay` lowers it further. | | `proxyConfiguration` | none | Optional Apify Proxy. Not needed for public docs; proxy usage is billed by Apify separately. | | `chunkTokens` | 512 | Upper bound per RAG chunk (128 to 2,048). | | `includeChunksInDataset` | false | Also embed each page's chunks in its dataset item. | | `includeLlmsFull` | true | Write `llms-full.txt` as well as `llms.txt`. | | `siteName`, `siteDescription` | detected | Override the H1 and summary of `llms.txt`. | Minimal input: ```json { "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }], "pathPrefix": "/js/docs", "maxPages": 200 } ``` ### Output #### Dataset: one item per page ````json { "url": "https://crawlee.dev/js/docs/quick-start", "canonicalUrl": "https://crawlee.dev/js/docs/quick-start", "title": "Quick Start", "description": "With this short tutorial you can start scraping with Crawlee in a minute or two.", "lang": "en", "siteName": "Crawlee", "headingPath": ["Introduction", "Quick Start"], "headings": [ { "level": 1, "text": "Quick Start" }, { "level": 2, "text": "Choose your crawler", "id": "choose-your-crawler" } ], "markdown": "# Quick Start\n\nWith this short tutorial you can start scraping with Crawlee in a minute or two...\n\n```bash\nnpx crawlee create my-crawler\n```", "wordCount": 612, "tokenEstimate": 910, "lastModified": "2026-09-20T10:00:00.000Z", "contentHash": "3f1c6c3f6c0d8c1e...", "chunkCount": 4, "depth": 0, "statusCode": 200, "crawledAt": "2026-10-06T19:02:11.431Z" } ```` `lastModified` comes from page metadata (`article:modified_time`, JSON-LD `dateModified`, a "last updated" element) or the HTTP `Last-Modified` header, and is `null` when the site exposes none. #### Key-value store | Record | Content | |---|---| | `llms.txt` | `# Site name`, a `>` summary, then `## Section` lists of `- [Title](url): description` entries. Sections follow the URL structure under the path prefix; changelogs, blogs and legacy versions go to `## Optional`. | | `llms-full.txt` | Every page's Markdown in crawl order, each introduced by its title and a `Source:` line. | | `chunks.json` | Array of `{ id, url, title, headingPath, text, tokens, index, total }`. If the array would exceed 8 MB it is split into `chunks-part-NN.json` records and `chunks.json` lists the parts. | | `OUTPUT` | Run summary: pages processed, failed and skipped, chunk and token totals, duration, pages per minute, why the crawl stopped (`stopReason`), robots.txt status per origin, the effective rate limit, which events were charged, memory use (peak resident memory and heap), and the page converter's limits and timings (`converter`). | Example `llms.txt`: ``` ## Crawlee for JavaScript > Crawlee is a web scraping and browser automation library for Node.js. This file lists the 20 documentation pages crawled from this site, each converted to Markdown. The full text of every page is in llms-full.txt. ### Overview - [Quick Start](https://crawlee.dev/js/docs/quick-start): With this short tutorial you can start scraping with Crawlee in a minute or two. - [Introduction](https://crawlee.dev/js/docs/introduction): Crawlee covers your crawling and scraping end-to-end. ### Guides - [Request Storage](https://crawlee.dev/js/docs/guides/request-storage): Crawlee has several request storage types that are useful for specific tasks. ``` Example chunk: ```json { "id": "3f1c6c3f6c0d-2", "url": "https://crawlee.dev/js/docs/quick-start", "title": "Quick Start", "headingPath": ["Quick Start", "Choose your crawler"], "text": "## Choose your crawler\n\nCrawlee comes with three main crawler classes: `CheerioCrawler`, `PuppeteerCrawler` and `PlaywrightCrawler`...", "tokens": 287, "index": 2, "total": 4 } ``` ### How pages are fetched The Actor fetches the HTML the server sends and parses it once with Cheerio (on Crawlee's `HttpCrawler`); it does not run a browser. That covers Docusaurus, MkDocs, Sphinx, VitePress, GitBook, Mintlify, Nextra, Hugo, Jekyll and any other server-rendered documentation, and it keeps runs fast and cheap. Measured on the Apify platform at the default 1,024 MB and the default limit of 2 requests per second: 200 pages of the Crawlee docs in about 2 minutes (~90 pages per minute), and 2,000 pages of the Python documentation (large pages) in about 34 minutes (~60 pages per minute). Short runs pay a fixed cost for robots.txt and sitemap fetches. Memory stays flat as the crawl grows: only a small index of the pages is kept in memory, and the output files are assembled from local disk at the end. The 2,000-page Python run peaked at about 600 MB. At 512 MB, typical documentation still fits, but very heavy sites stop early with all output files written for the pages converted so far (see "Limits and behaviour"). Documentation whose body is injected only by client-side JavaScript (some single-page-app portals) comes back empty; such pages are reported under `skipped.empty` in `OUTPUT` and are not billed. A browser-rendering option is on the roadmap; if you need it for a specific site, say so in the Issues tab. ### Pricing This Actor uses **pay-per-event** pricing. You pay a small fee when a run starts and a per-page fee for every page that is delivered to the dataset. Pages that fail, are blocked by robots.txt, are outside the scope, or contain no content are not charged. The current prices are shown in the **Pricing** tab of this page; a 200-page run costs 200 times the page price plus one run-start fee, and nothing else: the Actor's own compute is included. Set a **maximum total charge** on the run if you want a hard ceiling; the Actor lowers `maxPages` to fit it and stops cleanly when it is reached. ### Limits and behaviour - **Scope:** same origin as the start URLs; optional path prefix; up to 2,000 pages and 20 start URLs per run. For larger sites, run once per section. - **Crawl order:** the start URLs first, then the pages around them (under the start URL's directory, e.g. `/js/docs/` for `/js/docs/quick-start`, in the same docs version), found through links and the sitemap. The rest of the site (blog, other sections, old docs versions) is crawled only if `maxPages` is not used up by then. The `llms.txt` summary line comes from the start page, or the site's home page, never from another page. - **Very large pages:** a page whose HTML is larger than about 7.9 MB at the default 1,024 MB of memory (about 3.8 MB at 512 MB; the limit grows with the run's memory) is skipped, logged and not billed; it is counted under `skipped.tooLarge` in `OUTPUT`. - **Very deeply nested pages:** a page whose HTML nests elements more than 600 levels deep (real documentation nests far less) is skipped, logged and not billed; it is counted under `skipped.tooDeep` in `OUTPUT`. - **Per-page time and memory limits:** each page is converted in a separate worker with a time limit (30 seconds at 1,024 MB of memory or more, longer on smaller runs, up to 2 minutes at 256 MB) and its own memory budget. Real documentation pages take well under a second. A page that hits either limit is skipped, logged and not billed, and the crawl carries on; it is counted under `skipped.timeout` or `skipped.outOfMemory` in `OUTPUT` (a page the converter fails on for another reason is counted under `skipped.conversionError`). - **robots.txt:** always honoured. `Disallow` rules skip pages; `Crawl-delay` lowers the rate limit. If `robots.txt` returns a server error the run stops before crawling, as RFC 9309 requires. The crawler identifies itself as `LathesmithDocsCrawler` (User-Agent `Mozilla/5.0 (compatible; LathesmithDocsCrawler/0.1; Apify Actor; +https://apify.com/store)`), so site owners can address it in robots.txt. - **Rate limit:** 120 requests per minute by default, at most 240, with low concurrency. It cannot be raised above that. - **Early stop:** if memory use nears the run's memory limit or the run's timeout approaches, the crawl stops, still writes every file from the pages converted so far, and records why in `OUTPUT.stopReason`. Pages that were not converted are not billed. - **No access control bypass:** no logins, cookies, CAPTCHA solving or paywall circumvention. Pages that need a session come back as errors or empty and are not billed. - **No personal data:** the Actor extracts documentation text only. Login, account, profile, user, cart and admin URLs are excluded by default. - **Versioned docs:** sitemaps often list every old version (`/docs/3.10/...`, `/v2/...`). Current, unversioned pages are queued first and old versions go to the `## Optional` section of `llms.txt`; to leave them out entirely, add an exclusion such as `/docs/\d` or `/v\d+/`. - **Content extraction** is heuristic. It recognises the main content area of the common documentation generators and falls back to the largest text block. Unusual layouts can leave a little chrome in or drop a sidebar note; use `excludeUrlPatterns` and report layouts that go wrong in the Issues tab. - **Token estimates** are estimates (about cl100k, within roughly 15%), not a tokenizer. - **Legal:** use it on documentation you own or are allowed to copy. You are responsible for complying with the target site's terms and copyright law; the Actor only helps you stay within robots.txt and rate limits. ### Use from an AI agent (MCP) Apify Actors can be called as tools from MCP-compatible agents and assistants through the Apify MCP server, and from code through the Apify API and SDKs. Ask your agent to run this Actor with an input like: ```json { "startUrls": [{ "url": "https://docs.example.com/docs/" }], "pathPrefix": "/docs", "maxPages": 100 } ``` and read `llms.txt` or `chunks.json` from the run's key-value store, or the dataset items for per-page Markdown. See Apify's documentation on the MCP server and the API for setup details. ### Integrations Like every Actor, this one works with Apify's integrations: schedules (re-crawl weekly and diff `contentHash`), webhooks, and exports to JSON, CSV, Excel or XML. Dataset items are also readable directly by LangChain and LlamaIndex Apify loaders. ### Support Open an issue in the **Issues** tab with the start URL and the run id; issues are answered in writing. Replies may be drafted with AI assistance and are reviewed before posting. # Actor input Schema ## `startUrls` (type: `array`): One or more public documentation pages to start from, for example `https://crawlee.dev/js/docs/quick-start`. The crawl stays on the same origin (scheme + host) as each start URL. Up to 20 URLs. ## `pathPrefix` (type: `string`): Only crawl URLs whose path starts with this prefix, e.g. `/js/docs`. Leave empty to crawl the whole origin. Every start URL must be inside the prefix. ## `maxPages` (type: `integer`): Stop after this many pages have been converted. Only converted pages are billed. Hard cap 2,000 per run; split larger sites by path prefix. ## `maxDepth` (type: `integer`): How many links away from a start URL the crawler may go. 0 crawls only the start URLs. ## `useSitemap` (type: `boolean`): Read the site's sitemap (from robots.txt or /sitemap.xml) and queue matching pages up front, so coverage does not depend on navigation links. Sitemap URLs still go through the prefix, exclusion and robots.txt filters. ## `excludeUrlPatterns` (type: `array`): Regular expressions (case-insensitive); any URL matching one is skipped. Example: `/api-reference/`, `\.html$`, `/v[0-9]+/`. ## `useDefaultExcludes` (type: `boolean`): Skip login, sign-up, account, profile, user, cart, checkout, admin and search URLs. Keep this on unless the docs genuinely live under one of those paths. ## `ignoreQueryParams` (type: `boolean`): Treat `/page?x=1` and `/page` as the same page and drop the query string. Turn off only for sites whose pages are selected by query parameters (e.g. `index.php?title=`); tracking parameters are always dropped. ## `requestsPerMinute` (type: `integer`): Rate limit for the crawl (default 120, i.e. 2 requests per second). Cannot exceed 240. If robots.txt declares a `Crawl-delay`, the lower of the two applies. ## `proxyConfiguration` (type: `object`): Optional. Not needed for public documentation sites. Apify Proxy usage is billed to your account by Apify on top of this Actor's price. ## `chunkTokens` (type: `integer`): Upper bound for the RAG chunks in `chunks.json`. Chunks follow headings and never split inside a code block; sections larger than this are split at paragraph boundaries. ## `includeChunksInDataset` (type: `boolean`): Also include each page's chunks inside its dataset item (field `chunks`). Handy for per-page consumers; makes dataset items larger. `chunks.json` is always written. ## `includeLlmsFull` (type: `boolean`): Write the full-text companion file with every page's Markdown. Turn off to keep only the `llms.txt` index. ## `siteName` (type: `string`): Override for the H1 of `llms.txt`. Detected from the pages when empty. ## `siteDescription` (type: `string`): Override for the one-paragraph summary under the H1 of `llms.txt`. Taken from the start page's meta description when empty. ## Actor input object example ```json { "startUrls": [ { "url": "https://crawlee.dev/js/docs/quick-start" } ], "pathPrefix": "/js/docs", "maxPages": 200, "maxDepth": 20, "useSitemap": true, "excludeUrlPatterns": [], "useDefaultExcludes": true, "ignoreQueryParams": true, "requestsPerMinute": 120, "proxyConfiguration": { "useApifyProxy": false }, "chunkTokens": 512, "includeChunksInDataset": false, "includeLlmsFull": true } ``` # Actor output Schema ## `pages` (type: `string`): No description ## `llmsTxt` (type: `string`): No description ## `llmsFullTxt` (type: `string`): No description ## `chunks` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ { "url": "https://crawlee.dev/js/docs/quick-start" } ], "pathPrefix": "/js/docs", "excludeUrlPatterns": [], "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("lathesmith/docs-to-llms-markdown").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [{ "url": "https://crawlee.dev/js/docs/quick-start" }], "pathPrefix": "/js/docs", "excludeUrlPatterns": [], "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("lathesmith/docs-to-llms-markdown").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ { "url": "https://crawlee.dev/js/docs/quick-start" } ], "pathPrefix": "/js/docs", "excludeUrlPatterns": [], "proxyConfiguration": { "useApifyProxy": false } }' | apify call lathesmith/docs-to-llms-markdown --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,lathesmith/docs-to-llms-markdown" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/8Hs8sXuUHxT8TmlJ7/builds/UjcYfzqApcn3iycSg/openapi.json