Docs → RAG Corpus Builder (LLM-ready Markdown + llms.txt)

Turn any documentation site into an embeddings-ready RAG corpus in one run: clean markdown chunks with code blocks preserved, a heading path on every chunk, version awareness, a generated llms.txt, and optional OpenAI embeddings. Built to be called by AI agents as easily as by humans.

Who it's for: teams building RAG chatbots and coding assistants over product docs, agent builders who need fresh grounded context, and platform engineers keeping vector stores in sync with living documentation.

What it does

1. Crawls the docs site (Crawlee CheerioCrawler, plain HTTP — fast and cheap). Sitemap discovery runs first (robots.txt sitemaps, then /sitemap.xml); link crawling is the fallback, constrained by your include/exclude globs. Same-origin only, robots.txt respected, Retry-After honored on 429s.
2. Extracts the actual article content. Platform-specific selector heuristics for Docusaurus, MkDocs Material, GitBook, ReadMe and Sphinx (auto-detected), with a Mozilla Readability fallback for everything else. Nav, sidebars, footers, breadcrumbs, version badges and heading-anchor noise are stripped.
3. Converts to markdown with real fidelity: <pre><code class="language-x"> becomes a ```x fenced block (line structure preserved even for prism/shiki line-span highlighting), tables become GFM tables.
4. Chunks on heading boundaries first, then packs to your token budget (cl100k estimate via js-tiktoken) with sentence-level overlap. Fenced code blocks and tables are never split — an oversized one becomes a single chunk flagged oversized: true.
5. Dedupes near-identical pages (print views, mirrors) via simhash and logs every skip.
6. Ships artifacts: one dataset record per chunk, plus llms.txt, corpus.jsonl and MANIFEST.json in the key-value store — and embeddings.jsonl when you provide an OpenAI key.

Input

Field	Type	Default	Description
startUrls	array	required	Entry points. The crawl stays on these URLs' origins.
includeGlobs	string[]	[]	Only crawl URLs matching at least one minimatch glob (matched against full URL and pathname, e.g. /api/**).
excludeGlobs	string[]	[]	Skip URLs matching any glob.
maxPages	integer	500	Hard page cap (max 5,000).
versionMode	enum	all	all tags every page with its detected version; latest-only drops versioned URLs (unless pinned).
versionPattern	string	/(v\d+(\.\d+)*|\d+\.\d+)/	Regex that detects the version segment in URL paths → detectedVersion.
pinnedVersion	string	—	In latest-only mode, keep URLs of exactly this version (e.g. v2).
maxChunkTokens	integer	512	Token budget per chunk.
chunkOverlapTokens	integer	64	Sentence-level overlap between consecutive chunks of a section.
openaiApiKey	secret string	—	Enables embeddings. Sent only to api.openai.com, never logged or stored.
embeddingModel	string	text-embedding-3-small	OpenAI embedding model.
proxyConfiguration	proxy	—	Apify Proxy or custom proxies.

Output

Dataset — one record per chunk

Real sample from a run against https://docs.apify.com/ (maxPages: 30):

{
  "id": "https://docs.apify.com/academy/advanced-web-scraping/crawling/crawling-sitemaps#7",
  "url": "https://docs.apify.com/academy/advanced-web-scraping/crawling/crawling-sitemaps",
  "title": "Crawling sitemaps",
  "headingPath": ["Crawling sitemaps", "Using Crawlee"],
  "detectedVersion": null,
  "chunkIndex": 7,
  "markdown": "## Using Crawlee\n\nFortunately, you don't have to worry about any of the above steps if you use [Crawlee](https://crawlee.dev), a scraping framework, which has rich traversing and parsing support for sitemap. It can traverse nested sitemaps, download, and parse compressed sitemaps, and extract URLs from them. You can get all the URLs in a few lines of code:\n\n```js\nimport { RobotsFile } from 'crawlee';\n\nconst robots = await RobotsFile.find('https://www.mysite.com');\n\nconst allWebsiteUrls = await robots.parseUrlsFromSitemaps();\n```",
  "tokenEstimate": 131,
  "oversized": false
}

The final dataset record is a run summary (recordType: "summary") with counts and artifact URLs. Records are validated against the actor's dataset schema on insert, and every run's Output tab links the chunks and all four artifacts directly (actor output schema).

Key-value store artifacts

Key	Contents
llms.txt	Site title, summary and a curated per-section link list in the llmstxt.org format.
corpus.jsonl	Every chunk, one JSON object per line — pipe straight into your ingestion job.
MANIFEST.json	Pages crawled, chunk/token totals, versions detected, full settings echo.
embeddings.jsonl	{ "id", "vector" } per chunk (only when openaiApiKey is set); id joins corpus.jsonl.

llms.txt from the same real run:

# Apify Documentation

> Documentation corpus generated from docs.apify.com — clean markdown chunks for RAG and LLM context.

## Overview

- [Apify Documentation](https://docs.apify.com/)

## Academy

- [Apify Academy](https://docs.apify.com/academy): Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.
- [Actor description & SEO description](https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description): Learn about Actor description and meta description. Where to set them and best practices for both content and length.
…

MANIFEST.json (excerpt, same run): 30 pages crawled in sitemap mode, 0 failed, 310 chunks, 1 oversized, 49,546 tokens total.

Pricing (pay-per-event)

Event	Charged when	Suggested price
page-processed	Per page successfully extracted, after its chunks are stored	$1.50 / 1,000 pages
corpus-built	Once per completed run, after MANIFEST.json is written	$0.10 / run
embeddings-1k	Per started 1,000 chunks embedded, after embeddings.jsonl is stored	$0.20 / 1,000 chunks

Failed pages and failed runs are never charged. A default 500-page run costs about $0.85 (+ embeddings if enabled; OpenAI usage is billed to your own key).

Use cases

1. Docs chatbot in an afternoon. Point the actor at docs.yourproduct.com, load corpus.jsonl + embeddings.jsonl into pgvector/Pinecone/Qdrant, and your support bot cites the exact section (headingPath) it answered from.
2. Keep a coding agent current. Schedule weekly runs over a fast-moving framework's docs with versionMode: "latest-only" so your agent stops recommending deprecated v1 APIs. Diff MANIFEST.json between runs to re-embed only what changed.
3. Publish llms.txt for your own product. Generate a spec-compliant llms.txt from your real docs structure and serve it at /llms.txt so ChatGPT, Claude and Perplexity ground themselves on your documentation instead of hallucinating it.

Calling this actor from AI agents

The pipeline is exposed as one exported function (runCorpusBuild(input) in src/pipeline.ts), so the actor is a single tool call for an agent:

• Apify MCP server — connect your agent (Claude Desktop, or any MCP client) to https://mcp.apify.com and add this actor. It appears as a callable tool: the agent passes the input JSON, waits for the run, then reads the dataset and corpus.jsonl/llms.txt from the key-value store. See Apify MCP docs.
• Task runs — create an Apify Task with your site preset (globs, budget, version pin) and let agents trigger it via POST /v2/actor-tasks/:taskId/runs?token=… — no input assembly needed.
• API — POST /v2/acts/<you>~docs-rag-builder/runs with the input as JSON body; poll the run, then GET /v2/key-value-stores/:storeId/records/corpus.jsonl.

Running locally

npm install
npm run build     # tsc — zero errors
npm test          # node --test — extraction fixtures for all 5 doc platforms
apify run         # uses storage/key_value_stores/default/INPUT.json (docs.apify.com, 30 pages)

FAQ

Does it execute JavaScript? No — it's HTTP + Cheerio by design, which makes it ~10× cheaper and faster. JS-only SPAs without server-rendered content are out of scope for v0.1.

How accurate are token counts? Real cl100k_base encoding via js-tiktoken — the same tokenizer family OpenAI embedding models use, not a character heuristic.

What does oversized: true mean? The chunk exceeds maxChunkTokens because it is a single indivisible code block or table. Splitting code mid-fence would poison retrieval, so it ships whole and flagged; for embedding, oversized text is truncated to the model's input limit.

Can it crawl multiple sites in one run? Yes — add several startUrls; each URL's origin is allowed and everything stays within that set.

Why did a page I expected not appear? Check MANIFEST.json: it records duplicate skips, empty-content skips and failures per run, and the actor log names every skipped URL.

Do I need an OpenAI key? No — chunks, llms.txt and corpus.jsonl are produced without it. The key only enables embeddings.jsonl.

Changelog

0.1.0

• Initial release: sitemap-first crawling, 5-platform extraction heuristics + Readability fallback, fence-safe heading-aware chunking, simhash dedupe, llms.txt/corpus.jsonl/MANIFEST artifacts, optional OpenAI embeddings, pay-per-event billing.