Docs-to-RAG Pipeline Builder

Point it at a documentation site. Get back clean, chunked, embedding-ready markdown.

Not a generic scraper with a markdown option — a pipeline built around the one thing that makes a documentation corpus useful for retrieval: every chunk knows where it came from, keeps its code fences intact, and carries the heading path that makes it answerable on its own.

No paid API keys anywhere. Tokenisation, extraction and markdown conversion run locally. Embeddings are optional and free either way: on-device, or through a Cloudflare Worker you deploy on the free Workers AI tier.

---

What it does

┌─────────────┐
   start URLs ──────►│  sitemap.xml│──┐
                     └─────────────┘  │
                                      ▼
           ┌──────────────────────────────────────┐
           │  Pass 1  ·  CheerioCrawler (HTTP)    │
           └──────────────┬───────────────────────┘
                          │  content quality gate
                 ┌────────┴────────┐
        passes   │                 │  too thin / SPA shell
                 ▼                 ▼
                 │       ┌──────────────────────────────┐
                 │       │ Pass 2 · PlaywrightCrawler   │
                 │       └──────────────┬───────────────┘
                 └────────┬─────────────┘
                          ▼
   ┌───────────────────────────────────────────────────────┐
   │  Extract   explicit selector → generator profile       │
   │            → Defuddle → density heuristic → body       │
   │            (all scored; best wins)                     │
   ├───────────────────────────────────────────────────────┤
   │  Markdown  Turndown + GFM, code-fence languages,       │
   │            tables, admonitions → GitHub alerts         │
   ├───────────────────────────────────────────────────────┤
   │  Chunk     heading tree · breadcrumbs · never splits   │
   │            a code block or a table · real BPE tokens   │
   ├───────────────────────────────────────────────────────┤
   │  Dedup     exact hash + heading-scoped SimHash         │
   ├───────────────────────────────────────────────────────┤
   │  Embed     none · on-device · your Cloudflare Worker   │
   └───────────────────────────────────────────────────────┘
                          ▼
      dataset (one item per chunk) · corpus.jsonl · corpus.md
                  · llms.txt · llms-full.txt · run-report.json

Quick start

On Apify

{
  "startUrls": [{ "url": "https://docs.example.com" }],
  "maxCrawlPages": 500
}

startUrls is the entire minimum input; everything else has a documented default. maxCrawlPages is shown here only because it is the one worth raising: it defaults to 50, which keeps a first run quick, and 500 takes most of a documentation site.

A run is also bounded by the clock, not only by the page count. The crawl stops in time to write its output before the run's own timeout, so a site too big or too slow for the time allowed returns a smaller corpus and says so in the summary, rather than being killed mid-crawl with an empty dataset. Set maxCrawlTimeSecs to choose that budget yourself.

Locally, no account needed

npm install
npm run build

npx docs-to-rag https://docs.example.com --out ./corpus

corpus/
├── chunks.jsonl        one JSON object per chunk, ready for a vector store
├── corpus.jsonl        the same, as a key-value-store artifact
├── corpus.md           every page as one readable markdown document
├── llms.txt            an index of the site, llmstxt.org style
├── llms-full.txt       the whole corpus as a single document
└── run-report.json     counts, extractors used, failures

Run npx docs-to-rag --help for the full flag list. Any field in INPUT_SCHEMA.json works as --kebab-case-name.

---

What a chunk looks like

{
  "id": "9f2c1a77b0e34d15",
  "url": "https://docs.example.com/api/authentication",
  "anchorUrl": "https://docs.example.com/api/authentication#oauth-2-0",
  "pageTitle": "Authentication",
  "breadcrumb": ["Authentication", "Authorization flows"],
  "heading": "OAuth 2.0",
  "headingLevel": 3,
  "chunkIndex": 2,
  "chunkCount": 5,

  // `text` is what you embed. It opens with the heading path, which is the
  // difference between a retrievable chunk and an orphaned paragraph.
  "text": "Authentication > Authorization flows > OAuth 2.0\n\n### OAuth 2.0\n\nSet `redirect_uri` to...",
  "rawText": "### OAuth 2.0\n\nSet `redirect_uri` to...",

  "tokenCount": 612,          // real cl100k_base tokens, counted offline
  "charCount": 2481,
  "contentHash": "1b0f...",
  "generator": "mkdocs-material",
  "extractor": "profile:mkdocs-material",
  "crawledAt": "2026-08-24T10:14:22.104Z",

  // present only when embeddings are enabled
  "embedding": [0.0123, -0.0456, ...],
  "embeddingModel": "@cf/baai/bge-m3",
  "embeddingDims": 1024
}

anchorUrl is a real deep link. Cite it and the reader lands on the exact section.

---

The parts that took the work

The crawler escalates instead of guessing

Pass 1 fetches everything over plain HTTP with CheerioCrawler. Each page's extracted content is scored, and pages that fail the gate — a JavaScript shell, an empty article, a generator known to render client-side — are re-fetched in a real browser by PlaywrightCrawler. You pay for Chromium only on the pages that need it.

This is deliberately not AdaptivePlaywrightCrawler: that class is still experimental, hands the request handler a reduced context, and decides by sampling. Here the decision is a pure function of the extracted content, so it is unit-tested and identical on every run.

Extraction degrades instead of failing silently

Four strategies run and are scored against each other:

1. your mainContentSelector, if you set one — it wins outright
2. a documentation generator profile — MkDocs (Material and default), Docusaurus, Sphinx (Read the Docs, alabaster, Furo, PyData), Starlight, VitePress, VuePress, Nextra, Antora, GitBook, Mintlify, Docsify, Docsy, Just the Docs, Redoc, ReadMe.io, plus best-effort profiles for Hugo Book and Slate
3. Defuddle, the extractor behind Obsidian Web Clipper
4. a link-density heuristic, then the bare <body>

Every profile except Hugo Book and Slate was verified against a live page from that generator, saved as a committed fixture and asserted in the test suite; those two are labelled unverified in profiles.ts because no reachable demo site existed when the table was built. The scoring is what makes that safe: if a selector is wrong or goes stale, the profile simply loses to Defuddle instead of quietly emitting an empty page.

Chunking that respects structure

• Splits on the heading tree, not a sliding window.
• Never splits inside a fenced code block or a table. An oversized code block is split at line boundaries and each piece is re-fenced with the original language; an oversized table is split at row boundaries with the header repeated.
• Overlap is never taken from inside a code fence — a half-open fence poisons everything downstream.
• Sections below minChunkSize are merged, because a fifteen-token orphan matches on stray terms and answers nothing.
• Sizes are real BPE token counts from gpt-tokenizer, offline, not characters ÷ 4.

Deduplication that knows what a heading means

Exact hashing catches byte-identical pages. Near-duplicate detection uses a 64-bit SimHash with banded lookup, so it stays linear across tens of thousands of chunks.

Near-duplicate matching is scoped to the chunk's heading. Without that, "Install on Linux" and "Install on Windows" with identical bodies differ by one word in a hundred, land inside any useful similarity threshold, and one of them is silently lost. The default threshold is conservative for the same reason: dropping real content is worse than keeping a duplicate.

Markdown fidelity

Code fences recover their language from class="language-*", lang-*, highlight-*, data-language, and wrapper divs — MkDocs, Sphinx, Docusaurus and Starlight each hide it somewhere different. Line-number tables unwrap to the code cell instead of becoming a markdown table. Admonitions become GitHub alerts. KaTeX keeps its TeX and drops the duplicate visual markup. snake_case is not escaped to snake\_case, because CommonMark does not treat intraword underscores as emphasis.

---

Embeddings, free, two ways

	local	cloudflare-worker
Runs	on this machine / actor	your Cloudflare account
Network	model weights downloaded once, then none	one HTTPS call per batch
Cost	none	free tier, 10,000 Neurons/day
Default model	Xenova/all-MiniLM-L6-v2 (384d)	@cf/baai/bge-m3 (1024d)
Best for	offline, small corpora	better vectors, bigger crawls

Deploy the Worker in about three minutes — see worker/README.md for the dashboard walkthrough and the wrangler one-liner. Then:

{
  "startUrls": [{ "url": "https://docs.example.com" }],
  "embeddingProvider": "cloudflare-worker",
  "workerUrl": "https://docs-to-rag-worker.you.workers.dev",
  "workerApiKey": "<the API_TOKEN you set on the Worker>"
}

At bge-m3's rate the free daily allocation covers roughly 9 million tokens of embeddings — a 500-page site chunked at 800 tokens costs about 0.5% of one day.

If embedding fails, the crawl does not. Failed batches are counted and reported; the markdown corpus is written either way. You lose vectors, never content.

Optional: LLM contextualisation

contextualizeChunks: true prepends one model-written sentence to each chunk situating it in its document, before embedding — the "contextual retrieval" pattern. A chunk reading "Set this to your callback URL" is nearly unretrievable; the same chunk led by "This describes the redirect_uri parameter of the OAuth authorisation endpoint" is not.

It costs one LLM call per chunk, so it is off by default and capped by maxContextualizedChunks (500).

---

Common recipes

Only the docs, not the blog

{ "includeUrlGlobs": ["**/docs/**"], "excludeUrlGlobs": ["**/blog/**", "**/changelog/**"] }

One version of versioned docs

{ "startUrls": [{ "url": "https://docs.example.com/en/stable/" }], "crawlScope": "same-path-prefix" }

A fully client-rendered site (Docsify, Redoc, an old React docs app)

{ "renderingMode": "browser-only", "browserWaitForSelector": ".markdown-section" }

Match your embedding model's window

{ "chunkSize": 512, "chunkOverlap": 64, "tokenizer": "o200k_base" }

The site has an unusual layout

{ "mainContentSelector": "#article-body", "extraExcludeSelectors": [".version-banner", ".related-links"] }

Full input reference

Every field, its type, default and description lives in INPUT_SCHEMA.json, which the Apify Console renders as a form. The test suite asserts that file and the code agree on every default and every enum, so it cannot drift.

Highlights:

Field	Default	Notes
startUrls	—	The only required field.
crawlScope	same-hostname	Also same-domain, same-path-prefix, any.
maxCrawlPages	50	Reported in the summary when it binds. Raise it for a whole site.
maxCrawlTimeSecs	0	Wall-clock ceiling on the crawl. 0 derives one from the run's timeout, keeping a reserve for embedding and output.
renderingMode	auto	HTTP first, browser where needed.
escalateBelowWords	60	Raise it on sites with many short reference pages: a genuinely short page gets a browser re-fetch that returns the same thin content. "Browser re-fetches" in the run summary tells you if that is happening.
mainContentSelector	—	Wins outright when set.
chunkSize / chunkOverlap	800 / 100	In tokenizer tokens.
splitHeadingLevel	3	Deeper headings stay inside their parent chunk.
nearDuplicateThreshold	0.97	Conservative on purpose.
embeddingProvider	none	local or cloudflare-worker.
outputMode	chunks	Or pages, or both.

Output

Dataset — one item per chunk (or per page, or both). Every record carries itemType: "chunk" | "page", so outputMode: "both" stays parseable: a loader filters on that field rather than guessing from which keys are present. The Apify Console gets two prebuilt views: Chunks for reading, Provenance for auditing where content came from.

Key-value store — corpus.jsonl, corpus.md, llms.txt, llms-full.txt, run-report.json, and OUTPUT with the run summary.

The run ends with a readable summary and, when something looks off, specific advice: the page budget bound the crawl, the generic body extractor won too often, most links were rejected as out of scope, too many chunks were deduplicated. A crawl that quietly returns 40 pages instead of 400 is the most common way a tool like this fails, and the report is the defence.

Development

npm install
npm run verify           # typecheck + full test suite
npm test                 # vitest
npm run build            # tsc -> dist/

# See which extractor wins on each committed fixture, and why
npx tsx scripts/extract-report.ts test/fixtures
npx tsx scripts/extract-report.ts test/fixtures --markdown

The content pipeline is pure functions over { url, html }, so extraction, markdown conversion and chunking are all tested offline against real saved documentation pages. Crawler tests run against a synthetic site on localhost. Nothing in npm test touches the network.

Architecture notes and the design rationale are in docs/ARCHITECTURE.md.

Deploying to Apify

npm install -g apify-cli
apify login
apify push

.actor/actor.json, .actor/Dockerfile and INPUT_SCHEMA.json are ready. The Docker base image pins both the Node major and the exact Playwright version so the bundled Chromium always matches the dependency.

Licence

MIT.