RAG Dataset Builder

Crawl a site and get back a finished, embedding-ready RAG dataset — not just clean text. Plenty of tools (including RAG Web Browser, which holds a large majority of users in this space) stop at "here is the page as markdown." The actual gap for anyone building retrieval is the last mile: turning that markdown into properly-bounded chunks with a real token budget, context- preserving overlap, and no repeated boilerplate. That last mile is what this actor does.

What it does

Crawl — discovers pages via sitemap (including sitemap indexes), falling back to same-origin link following, then extracts clean markdown per page (nav/footers/cookie banners stripped), reusing the same battle-tested extraction as llms-txt-generator and llm-dataset-builder.

Semantic chunking — this is the moat. Dumb fixed-size splitting (every N characters) is what most tools ship, and it routinely cuts a chunk mid-sentence, mid-table, or mid-code-block, which quietly wrecks retrieval quality. This actor instead:

1. Splits on heading boundaries first — a section under a heading is the natural unit of meaning, so it stays whole as one chunk whenever it fits the token budget.
2. Never splits mid-code-block and never splits mid-table. Fenced code and markdown tables are treated as atomic units through the whole pipeline.
3. Only falls back to sentence-boundary splitting when a single section is too big to be one chunk — and even then, every resulting piece repeats the section's heading so it never loses context.
4. Applies configurable overlap (in tokens) between adjacent chunks from the same page, built from whole trailing sentences/blocks of the previous chunk — never a mid-word cut.
5. Runs cross-page near-duplicate removal (shingle + Jaccard similarity) so repeated boilerplate — a footer, a repeated disclaimer, a page crawled twice — doesn't show up as redundant rows in your embedding index.

Every chunk carries full provenance: source URL, page title, heading path (H1 > H2 > H3), chunk index/count within its page, and its estimated token count.

Input

{
  "siteUrl": "https://docs.example.com",
  "maxPages": 200,
  "targetTokens": 350,
  "maxTokens": 512,
  "overlapTokens": 40,
  "dedupThreshold": 85
}

Output

One chunk row per chunk (url, pageTitle, headingPath, headingPathString, chunkIndex, chunkCount, text, tokenCount, characters, overlapTokens), each ready to drop straight into an embedding call — the rows ARE your JSONL dataset. Optionally (includeDuplicateRows) one duplicate row per chunk that got merged away, showing what it matched and the similarity score. One final summary row with pages/chunks/tokens/duplicates-removed for the whole crawl, also saved to the key-value store as SUMMARY.

Limits — read before you trust the numbers

• The token counter is an estimator, not a real tokenizer. It's chars/4 adjusted down for whitespace and punctuation — a defensible, deterministic rule of thumb, not a BPE vocabulary. If you need exact counts for a specific embedding model, re-count with that model's real tokenizer before billing against it. maxTokens is still respected against this estimate, so actual token counts from a real tokenizer will usually run close to, and occasionally slightly over, the cap.
• Near-duplicate detection is lexical (shingle + Jaccard), not semantic. Two chunks that say the same thing in different words will not be caught — only chunks that share enough literal n-grams. This is intentional (self-contained, no embedding model needed to run this actor) but it is not a substitute for semantic dedup downstream.
• Sentence splitting is regex-based, not a real NLP sentence boundary model. Uncommon abbreviations or unusual punctuation can occasionally produce a slightly off split — it never breaks a code block or table, but a prose sentence boundary can be imperfect.
• A single code block or table bigger than maxTokens is kept whole rather than corrupted — the "never split mid-block" rule wins over the token cap in that one edge case. It becomes an oversized chunk; check the tokenCount field if this matters for your embedding model's limit.

Typical uses

Building the corpus for a RAG pipeline over your own docs or a client's site · re-chunking a site after a content refresh so the vector index doesn't drift · producing a clean, deduplicated corpus before it ever reaches your embedding budget · an evaluation baseline to compare against a fixed-size chunker you're already running.