# 🧩 RAG Dataset Builder - Crawl to Embedding-Ready Chunks (`that_red_bird/rag-dataset-builder`) Actor

🧩 Crawl a site and get back a FINISHED, embedding-ready RAG dataset — not just clean text. ✅ Semantic chunking on heading boundaries, never mid-code-block or mid-table. ✅ Token estimator, configurable overlap, near-duplicate removal, full provenance per chunk.

- **URL**: https://apify.com/that\_red\_bird/rag-dataset-builder.md
- **Developed by:** [mohamed alaya](https://apify.com/that_red_bird) (community)
- **Categories:** AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## RAG Dataset Builder

Crawl a site and get back a **finished, embedding-ready RAG dataset** — not just clean text.
Plenty of tools (including RAG Web Browser, which holds a large majority of users in this space)
stop at "here is the page as markdown." The actual gap for anyone building retrieval is the last
mile: turning that markdown into properly-bounded chunks with a real token budget, context-
preserving overlap, and no repeated boilerplate. That last mile is what this actor does.

### What it does

**Crawl** — discovers pages via sitemap (including sitemap indexes), falling back to same-origin
link following, then extracts clean markdown per page (nav/footers/cookie banners stripped),
reusing the same battle-tested extraction as `llms-txt-generator` and `llm-dataset-builder`.

**Semantic chunking** — this is the moat. Dumb fixed-size splitting (every N characters) is what
most tools ship, and it routinely cuts a chunk mid-sentence, mid-table, or mid-code-block, which
quietly wrecks retrieval quality. This actor instead:

1. Splits on **heading boundaries first** — a section under a heading is the natural unit of
   meaning, so it stays whole as one chunk whenever it fits the token budget.
2. **Never splits mid-code-block and never splits mid-table.** Fenced code and markdown tables are
   treated as atomic units through the whole pipeline.
3. Only falls back to **sentence-boundary splitting** when a single section is too big to be one
   chunk — and even then, every resulting piece repeats the section's heading so it never loses
   context.
4. Applies **configurable overlap** (in tokens) between adjacent chunks from the same page, built
   from whole trailing sentences/blocks of the previous chunk — never a mid-word cut.
5. Runs **cross-page near-duplicate removal** (shingle + Jaccard similarity) so repeated
   boilerplate — a footer, a repeated disclaimer, a page crawled twice — doesn't show up as
   redundant rows in your embedding index.

Every chunk carries full **provenance**: source URL, page title, heading path (`H1 > H2 > H3`),
chunk index/count within its page, and its estimated token count.

### Input

```json
{
  "siteUrl": "https://docs.example.com",
  "maxPages": 200,
  "targetTokens": 350,
  "maxTokens": 512,
  "overlapTokens": 40,
  "dedupThreshold": 85
}
```

### Output

One `chunk` row per chunk (`url`, `pageTitle`, `headingPath`, `headingPathString`, `chunkIndex`,
`chunkCount`, `text`, `tokenCount`, `characters`, `overlapTokens`), each ready to drop straight
into an embedding call — the rows ARE your JSONL dataset. Optionally (`includeDuplicateRows`) one
`duplicate` row per chunk that got merged away, showing what it matched and the similarity score.
One final `summary` row with pages/chunks/tokens/duplicates-removed for the whole crawl, also saved
to the key-value store as `SUMMARY`.

### Limits — read before you trust the numbers

- **The token counter is an estimator, not a real tokenizer.** It's `chars/4` adjusted down for
  whitespace and punctuation — a defensible, deterministic rule of thumb, not a BPE vocabulary. If
  you need exact counts for a specific embedding model, re-count with that model's real tokenizer
  before billing against it. `maxTokens` is still respected against this estimate, so actual token
  counts from a real tokenizer will usually run close to, and occasionally slightly over, the cap.
- **Near-duplicate detection is lexical (shingle + Jaccard), not semantic.** Two chunks that say
  the same thing in different words will not be caught — only chunks that share enough literal
  n-grams. This is intentional (self-contained, no embedding model needed to run this actor) but
  it is not a substitute for semantic dedup downstream.
- **Sentence splitting is regex-based**, not a real NLP sentence boundary model. Uncommon
  abbreviations or unusual punctuation can occasionally produce a slightly off split — it never
  breaks a code block or table, but a prose sentence boundary can be imperfect.
- A single code block or table **bigger than `maxTokens` is kept whole rather than corrupted** —
  the "never split mid-block" rule wins over the token cap in that one edge case. It becomes an
  oversized chunk; check the `tokenCount` field if this matters for your embedding model's limit.

### Typical uses

Building the corpus for a RAG pipeline over your own docs or a client's site · re-chunking a site
after a content refresh so the vector index doesn't drift · producing a clean, deduplicated corpus
before it ever reaches your embedding budget · an evaluation baseline to compare against a
fixed-size chunker you're already running.

# Actor input Schema

## `siteUrl` (type: `string`):

The site to crawl, e.g. https://docs.example.com. Pages are discovered from its sitemap, or by following same-origin links if there is no sitemap.

## `maxPages` (type: `integer`):

Upper bound on pages to crawl and chunk.

## `useSitemap` (type: `boolean`):

Prefer the site's sitemap (including sitemap indexes) for discovery. Falls back to following same-origin links automatically if none is found.

## `includePatterns` (type: `array`):

Keep only URLs containing one of these substrings, e.g. /docs/, /guides/.

## `excludePatterns` (type: `array`):

Drop URLs containing any of these, e.g. /tag/, /author/, ?page=.

## `targetTokens` (type: `integer`):

Preferred chunk size. Chunk packing tries to land near this when a section has to be split, and a whole section under maxTokens is always kept as one chunk even if it's smaller or larger than this target.

## `maxTokens` (type: `integer`):

Hard cap. No chunk exceeds this except the rare case where a single code block or table is itself bigger than the cap — it is kept whole rather than corrupted (see README limits). Must be >= targetTokens.

## `overlapTokens` (type: `integer`):

How much of the end of one chunk is repeated at the start of the next chunk from the same page, so retrieval doesn't lose context at a chunk boundary. Overlap always lands on a whole sentence or block, never mid-word. Set to 0 to disable.

## `minChunkTokens` (type: `integer`):

Chunks estimated below this many tokens are dropped as too small to be useful on their own (e.g. a lone short heading with no body).

## `dedupThreshold` (type: `integer`):

Two chunks anywhere in the crawl are treated as near-duplicates and merged into one when their shingle-based Jaccard similarity is at or above this. Higher = fewer false merges, more repeated boilerplate left in. Expressed 0-100; 85 means 0.85.

## `shingleSize` (type: `integer`):

Length of the word n-grams used to fingerprint each chunk for near-duplicate detection. Smaller catches shorter overlaps but is noisier; larger is stricter.

## `includeLinks` (type: `boolean`):

Preserve [text](url) links in the extracted markdown. Turn off for pure prose with no link noise.

## `includeDuplicateRows` (type: `boolean`):

Emit one row per removed near-duplicate chunk (type: "duplicate") showing which surviving chunk it matched and the similarity score. Off by default so the dataset stays a clean, ready-to-embed corpus.

## `concurrency` (type: `integer`):

How many pages to fetch in parallel.

## `proxyConfiguration` (type: `object`):

Optional Apify proxy configuration.

## Actor input object example

```json
{
  "siteUrl": "https://docs.apify.com",
  "maxPages": 100,
  "useSitemap": true,
  "targetTokens": 350,
  "maxTokens": 512,
  "overlapTokens": 40,
  "minChunkTokens": 15,
  "dedupThreshold": 85,
  "shingleSize": 5,
  "includeLinks": true,
  "includeDuplicateRows": false,
  "concurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `downloadCsv` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `count` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteUrl": "https://docs.apify.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("that_red_bird/rag-dataset-builder").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "siteUrl": "https://docs.apify.com" }

# Run the Actor and wait for it to finish
run = client.actor("that_red_bird/rag-dataset-builder").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteUrl": "https://docs.apify.com"
}' |
apify call that_red_bird/rag-dataset-builder --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,that_red_bird/rag-dataset-builder"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/tWc76qb46OPyA1Rz2/builds/Svpj8hqEGAMj489jy/openapi.json