# 🧩 LLM Dataset Builder - Site to Embedding-Ready Chunks (`that_red_bird/llm-dataset-builder`) Actor

⚡ Turn any site into a RAG dataset in one run. ✅ Clean markdown (code blocks, tables, headings kept; nav and cookie banners stripped), split into semantic chunks on heading boundaries with token estimates, configurable overlap and a source URL on every chunk. ✅ Ready to embed, no post-processing.

- **URL**: https://apify.com/that\_red\_bird/llm-dataset-builder.md
- **Developed by:** [mohamed alaya](https://apify.com/that_red_bird) (community)
- **Categories:** AI, Agents, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## LLM Dataset Builder

Turn any website into an **embedding-ready dataset** in one run — not just markdown, the finished
artifact you can push straight into a vector store.

### Why this exists

Plenty of tools give you "the page as markdown". Then you still have to strip the navigation, decide
where to split, keep chunks from cutting mid-sentence, count tokens, add overlap, and attach a source
URL to every piece so retrieval can cite it. That post-processing is the actual work. This does it.

### What you get per chunk

`text` · `estimatedTokens` · `characters` · **`url`** (provenance — cite the source) ·
`pageTitle` · **`heading`** (the section the chunk came from) · `chunkIndex` / `chunkCount` ·
`language` · a stable `id`

### How the chunking works

Splitting is **heading-aware first**. Sections are packed up to your target size and only hard-split
when a single section is oversized — because splitting mid-sentence is what quietly ruins retrieval
quality. Overlap is carried between chunks so a sentence spanning a boundary is still findable.
Chunks below `minChunkTokens` are dropped: they are almost always nav scraps and they pollute results.

Tune with `chunkTokens` (512 suits most embedding models; 256 for tighter retrieval, 1024 for more
context per hit), `overlapTokens` and `minChunkTokens`.

### Content extraction

Headings, fenced code blocks **with language tags**, tables, lists, blockquotes and links are
preserved. Navigation, headers, footers, sidebars, cookie banners, ad slots, share widgets and
related-posts blocks are stripped. Set `includeLinks: false` for cleaner embedding text when URLs
add noise.

### Discovery

Sitemap first, including sitemap indexes; falls back to following same-origin links. Narrow the crawl
with `includePatterns` / `excludePatterns`. Note that sitemaps are usually alphabetical, so when an
include filter is set the **whole** sitemap is read before filtering — a section late in the alphabet
would otherwise be silently missed.

### Output modes

- **`chunks`** (default) — one row per chunk, ready to embed
- **`pages`** — one row per page with full markdown, if you want to chunk it yourself

### Typical uses

Building RAG over your own docs · ingesting a vendor's documentation into a support bot · creating a
searchable knowledge base · preparing fine-tuning or evaluation corpora · giving an internal agent
grounded product knowledge.

### Reliability

It reads public pages and your own sites — no login, no anti-bot, no third party that can cut you
off. If no page yields usable content the run fails loudly rather than handing back an empty dataset.

# Actor input Schema

## `siteUrl` (type: `string`):

The site to turn into a dataset, e.g. https://docs.example.com. Pages are discovered from the sitemap, or by following same-origin links if there is none.

## `maxPages` (type: `integer`):

Upper bound on pages to process.

## `outputMode` (type: `string`):

chunks = one row per embedding-ready chunk (what you feed a vector store) · pages = one row per page with full markdown.

## `chunkTokens` (type: `integer`):

Approximate tokens per chunk. 512 suits most embedding models; use 256 for tighter retrieval or 1024 for more context per hit.

## `overlapTokens` (type: `integer`):

Overlap carried between chunks so a sentence split across a boundary is still retrievable.

## `minChunkTokens` (type: `integer`):

Drop chunks smaller than this — they are usually nav scraps and pollute retrieval.

## `useSitemap` (type: `boolean`):

Prefer the sitemap for discovery, including sitemap indexes. Falls back to link-following automatically.

## `includePatterns` (type: `array`):

Keep only URLs containing one of these substrings, e.g. /docs/, /guides/.

## `excludePatterns` (type: `array`):

Drop URLs containing any of these, e.g. /tag/, /changelog/, ?page=.

## `includeLinks` (type: `boolean`):

Keep [text](url) links. Turn off for cleaner embedding text when the URLs add noise.

## `concurrency` (type: `integer`):

How many pages to fetch in parallel.

## `proxyConfiguration` (type: `object`):

Optional Apify proxy configuration.

## Actor input object example

```json
{
  "siteUrl": "https://developers.cloudflare.com",
  "maxPages": 100,
  "outputMode": "chunks",
  "chunkTokens": 512,
  "overlapTokens": 50,
  "minChunkTokens": 20,
  "useSitemap": true,
  "includeLinks": true,
  "concurrency": 5
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `downloadCsv` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `count` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteUrl": "https://developers.cloudflare.com"
};

// Run the Actor and wait for it to finish
const run = await client.actor("that_red_bird/llm-dataset-builder").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "siteUrl": "https://developers.cloudflare.com" }

# Run the Actor and wait for it to finish
run = client.actor("that_red_bird/llm-dataset-builder").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteUrl": "https://developers.cloudflare.com"
}' |
apify call that_red_bird/llm-dataset-builder --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,that_red_bird/llm-dataset-builder"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/WMWgk6JhHPFwq4RGf/builds/yEOhaZtsMXUOR7sHo/openapi.json