# RAG Dataset Builder - Source-Linked Chunks for Retrieval (`leadproof/rag-dataset-builder`) Actor

Turn web page records into deduplicated, source-linked text chunks ready to embed. Structure-aware splitting, exact token counts, stable chunk IDs, offsets back to the source and a JSONL export. Prepares data only: it does not index or search.

- **URL**: https://apify.com/leadproof/rag-dataset-builder.md
- **Developed by:** [Lead Proof](https://apify.com/leadproof) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 document chunkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RAG Dataset Builder

Turn web page records into deduplicated, source-linked text chunks that are ready to embed and load into a
retrieval system. Every chunk carries its source URL, its heading path, an exact token count, a stable ID
and character offsets back into the source text.

**This Actor prepares data. It does not create embeddings, run a vector database or answer questions.**
Chunks are not searchable until you embed them and load them into your own vector store or search engine.

### What you give it

Exactly one source per run:

| Source | Field | Notes |
| --- | --- | --- |
| Inline PageRecords | `records` | Up to 10,000 objects. Minimum per record: a URL (`finalUrl` or `requestedUrl`) and `markdown` or `text`. |
| Apify dataset of PageRecords | `datasetId` | For example the output of a crawler or page reader run. If the run writing it is still running, this Actor waits for it to finish (`sourceWaitSeconds`). Read page by page; only text fields are requested, never HTML. The run gets read access to that dataset only. |
| Website | - | **Not in this release.** Crawling a site and chunking it in one run is implemented but switched off while its pricing is settled; a run that asks for it is refused. Crawl with any PageRecord producer and pass the dataset. |

Records follow **PageRecord v1** (`contracts/page-record.v1.consumer.schema.json`; the canonical schema belongs
to the LeadProof Web Page Reader). Records with status `failed` or `skipped`, HTTP 4xx/5xx or no text are not
chunked; each gets a reason in the document report. An optional `sourceId` on a record sets your own stable
document identity (for example a product SKU or CMS ID).

Main settings (all validated; the run fails with a clear message on invalid values):

| Field | Default | Meaning |
| --- | --- | --- |
| `tokenizer` | `cl100k_base` | `cl100k_base` or `o200k_base` (tiktoken, exact counts), or `characters` (Unicode code points, labeled as not model tokens in every row). |
| `chunkSize` | 512 | Target size. Chunks are packed up to it. |
| `maxChunkSize` | = `chunkSize` | Hard limit, never exceeded. Raise it to keep tables, code blocks and list items whole when they are slightly larger than the target. |
| `chunkOverlap` | 64 (or chunkSize/8) | Tokens repeated from the end of the previous chunk of the same section. Must be smaller than `chunkSize`. |
| `sourceField` | `auto` | `auto` uses markdown when present (keeps structure), else text. |
| `dedupeMode` | `exact` | `none`, `exact`, or `near` (see Deduplication). |
| `includeUrlPatterns`, `excludeUrlPatterns` | none | Glob patterns on the source URL (`*` matches anything, including `/`). |
| `languages` | none | Keep only these primary language codes (`en`, `he`); records without a declared language are kept. |
| `maxDocuments`, `maxDocumentCharacters`, `maxTotalCharacters`, `maxChunks` | 1,000 / 500,000 / 20,000,000 / 20,000 | Input and output caps (see Limits). |
| `sourceWaitSeconds` | 900 | How long to wait for the run that writes `datasetId`. |
| `saveNormalizedSources` | false | Also store each document's normalized text as `source-<documentId>` so offsets can be checked. |

### Website mode (not in this release)

Chunking a site in one run, by crawling it with the LeadProof Website Crawler first, is built and tested: one
crawl per run with page, time and cost limits, the crawl reattached rather than repeated after a restart, and a
crawl stopped by a limit reported as partial coverage. It is switched off here because the waiting time and the
low-yield runs it can produce are not covered by this Actor's price yet. Input that asks for it is refused with a
message, in the Console and through the API alike.

Until it returns: run a crawler (the LeadProof Website Crawler, or anything that writes PageRecord v1), then pass
its dataset as `datasetId`. That is the same pipeline, with the crawl billed to you as its own run.

### Sources are read only when finished

A source is never read while it is still being written, so a momentarily empty page can never be mistaken for the
end of the data:

- `datasetId`: if the dataset belongs to an Apify run that is still running, this Actor waits for that run
  (`sourceWaitSeconds`, default 900). If it is still running after that, the run fails with the retryable
  `SOURCE_STILL_RUNNING` and reads nothing. If the producing run is not visible to this Actor, the dataset is read
  as it is, with the warning `source_run_not_visible`. Continuously written sources are not supported.
- `website` (off in this release): the crawl would be awaited to the end.
- The dataset's `itemCount` is never used as a read limit (Apify updates it asynchronously; measured 1,200 against
  1,500 items right after a push). Pages are read until an empty page; after the producer has finished, a higher
  `itemCount` only triggers a few short retries.

### What you get

1. **Dataset: one row per chunk.** Also available as JSONL: `{{apiDefaultDatasetUrl}}/items?format=jsonl&clean=true`
   (linked in the run's Output tab and in `MANIFEST.outputs.chunksJsonl`).
2. **`MANIFEST`** (key-value store): status, counts, skipped reasons, deduplication measurements, the settings
   and their fingerprints, billing summary.
3. **`DOCUMENTS`** (key-value store, JSONL): one line per input record: chunked, skipped with a reason, or failed;
   duplicate chunks removed and what they duplicated. Records after the first 5,000 continue in `DOCUMENTS-002`.

A real row (from `examples/output/chunks.jsonl`, shortened):

```json
{
  "schemaVersion": "1.0", "inputId": "he-guide", "status": "succeeded", "warnings": [], "error": null,
  "chunkId": "chk_6920b44b7a5e599c08e7fd0d60a0181e",
  "documentId": "doc_1b17d4b5757827b0d20f8295c6818492",
  "sourceUrl": "https://gym.example.co.il/membership",
  "title": "מדריך מנוי",
  "headingPath": ["מדריך מנוי לחדר הכושר", "מחירים"],
  "text": "## מחירים\n\n| מסלול | מחיר חודשי | כניסות |\n|-------|-----------|--------|\n| בסיסי | 150 ₪ | 8 |\n| מלא | 290 ₪ | ללא הגבלה |",
  "tokenCount": 80,
  "tokenizer": {"name": "cl100k_base", "library": "tiktoken", "libraryVersion": "0.14.0", "unit": "token", "exactModelTokens": true},
  "chunkIndex": 1,
  "contentHash": "sha256:357696da...",
  "offsets": {"start": 116, "end": 239, "unit": "unicode_code_point", "utf8Start": 205, "utf8End": 374,
              "basis": "normalized_source_text", "sourceField": "markdown", "sourceHash": "sha256:ba614461...",
              "sourceCharacters": 449, "normalization": "rag-normalize-v1", "overlapWithPrevious": null},
  "sourceReferences": [{"inputId": "he-guide", "url": "https://gym.example.co.il/membership", "...": "..."}],
  "metadata": {"language": "he", "blockTypes": ["heading", "table"], "continuation": [], "documentChunkCount": 4, "...": "..."}
}
```

Row contract: `contracts/chunk-record.v1.schema.json`. Manifest contract: `contracts/manifest.v1.schema.json`.

#### Offsets and source text

`text` is always exactly `normalizedSource[offsets.start:offsets.end]`. `normalizedSource` is the record's
`offsets.sourceField` (markdown or text) after **rag-normalize-v1**: Unicode NFC; CRLF, CR and Unicode line
separators become LF; control characters except LF and TAB and the BOM are removed (RLM/LRM and joiners are
kept); trailing spaces are removed from each line; outside code fences, runs of blank lines collapse to one;
the document is trimmed. If the document was longer than `maxDocumentCharacters`, it is cut at a paragraph
break first, and `sourceHash` is the hash of the text actually chunked. Offsets are given in Unicode code
points (Python string indices) and in UTF-8 bytes. `overlapWithPrevious` gives the range shared with the
previous chunk. The normalization code is `src/normalize.py`; with `saveNormalizedSources` the exact text is
stored per document.

#### IDs

- `documentId` = hash of (source key, source field, normalized-content hash). The source key is your `sourceId`,
  else the final URL, requested URL, canonical URL or `inputId`. Two records share a document ID only when both
  the key and the text are identical, so different input records are never merged by accident. The same URL
  with different content produces two documents and a `duplicate_source_key` warning.
- `chunkId` = hash of (document ID, chunking fingerprint, position, offsets, text hash). It is the same for
  unchanged content and unchanged chunking settings (tokenizer, sizes, overlap, normalization and chunker
  versions). Filters, limits and deduplication never change IDs.
- The output is deterministic: the same input and settings give byte-identical rows on every run.

### How chunking works

- Markdown is split into blocks with `markdown-it-py` (CommonMark plus GFM tables): headings, paragraphs,
  lists, code fences, tables, quotes and HTML blocks. Plain text is split at blank lines.
- A heading starts a new section; consecutive headings form one section. **Chunks never cross a section
  boundary**, and each chunk carries the full heading ancestry (`headingPath`).
- Blocks are packed up to `chunkSize`. A block that fits in `maxChunkSize` is never split.
- A larger block is split deterministically: list items, table rows (the header stays with its delimiter row)
  or code lines first, then sentences, then token windows snapped to whitespace, then raw token windows
  (never separating a combining mark from its letter). Each chunk holding a piece records
  `metadata.continuation` (block type, part N of M, split method, code language, and the table header for
  later parts).
- Overlap is applied only between chunks of the same section, starts at a sentence or word boundary, and is
  measured exactly; it never exceeds `chunkOverlap`.
- Text such as `<|endoftext|>` is counted as ordinary text, never as a tokenizer control token.

### Deduplication

- **exact** (default): a record whose normalized text is identical to an earlier one (for example the same page
  on a mirror URL) is not chunked again; the kept chunks list every source in `sourceReferences`. Chunks with
  byte-identical text (repeated footers, cookie banners) are emitted once; the report maps each removed chunk
  to the kept chunk ID.
- **near**: exact, plus removal of chunks that are almost the same. Candidates come from MinHash LSH over
  5-word shingles (lowercased, punctuation and diacritics ignored); a removal requires the **exact** Jaccard
  similarity of the two shingle sets to reach `nearDuplicateThreshold` (default 0.95). A pair is never merged
  when the two chunks differ in any number, price, percentage, date, email, URL or negation word
  (English and Hebrew). Every removal is listed with its similarity and the words that differed.
  Measured on the fixtures: the "$49" and "$59" versions of a plan page (similarity 0.945) are kept apart at
  thresholds 0.95, 0.9 and 0.8; a copy with only case and punctuation changes (similarity 1.0) is removed;
  a copy with one changed word ("every day" / "each day", 0.945) is kept at 0.95 and removed at 0.9.
  Near mode can therefore merge chunks that differ by a few ordinary words; use `exact` when every wording
  difference matters.

### Limits

- `maxDocuments` bounds the records read; the manifest reports records omitted by the limit.
- `maxDocumentCharacters` bounds each document (cut at a paragraph break, marked `partial`).
- `maxTotalCharacters` bounds the run; later documents are skipped as `input_budget_exhausted`.
- `maxChunks` bounds the output; documents are emitted whole, and the run stops before a document that
  would exceed the cap (`chunk_limit`).
- Datasets are read in pages sized to about 8 million characters of text, never all at once.
- Apify dataset items are at most 9 MB each, so one input record is bounded; the first page of a dataset is
  read with 10 items, later pages are sized from the text volume seen.

### Reliability

- Runs checkpoint after each batch (about 100 checkpoints per run). After a migration, crash or
  abort-and-resurrect, the run continues from the last checkpoint and never emits a chunk ID twice.
  Verified live: a run aborted after 4,200 of 13,500 rows and resurrected produced exactly the rows of a
  clean run.
- Website mode, while it was open, reattached to its crawl after an abort and resurrect instead of starting a
  second one, and produced the rows of a clean run.
- Datasets are read until an empty page. Apify's dataset `itemCount` is not trusted as a limit: right after
  a producer pushes items it can lag behind them (measured 1,200 vs 1,500).
- One document that fails to process is reported as `failed` (`processing_error`) and the run continues.
- A run where no document produced chunks ends as failed, with the reasons in `MANIFEST`.

### Pricing

**$0.002 per chunked document** (`document-chunked`), plus Apify's standard Actor start event ($0.00005 per GB of
run memory). Nothing else is charged.

One event covers one document that produced chunks, per started 100,000 characters **or** per started 250 chunks,
whichever is larger, never their sum:

| Document | Events |
| --- | --- |
| An ordinary page (a few thousand characters, a handful of chunks) | 1 |
| 100,000 characters in 40 chunks | 1 |
| 100,001 characters in 40 chunks | 2 |
| 5,000 characters in 251 chunks | 2 |
| 150,000 characters in 300 chunks | 2 |

Free: documents that are skipped (failed page, empty page, HTTP error, filtered out), documents collapsed as
duplicates of another document, and chunks removed as duplicates. A restarted or resurrected run never charges a
document twice, and a run stops before delivering a document its spending limit cannot pay for. The automatic
per-dataset-item event must stay off: the Actor refuses to run if it has a price, so diagnostic rows are never
billed.

So 1,000 ordinary pages cost about $2. Your Apify account also pays the usual platform storage for the dataset it
receives.

**Fair use.** Bring your own content: pages you crawled or own. This Actor does not fetch anything itself in this
release, it only reshapes records you pass in. Chunk text is copied from your source as is, so treat licensing of
that text as your responsibility.

### Security and untrusted content

- Page text is data. The Actor never follows instructions in it, runs no LLM and executes nothing from it.
  Downstream RAG systems should also treat retrieved chunks as untrusted content.
- Records and dataset modes make no web requests. The only network calls go to the Apify API for the input
  dataset and the run's own storages. No secrets are required.
- Limited Actor permissions: the run can read only the dataset you pick.

### Limitations

- Website mode is off in this release (see above); this Actor makes no web requests of its own.
- Markdown structure is only as good as the producer's markdown; plain text is chunked by paragraphs.
- Sentence splitting uses punctuation rules, not a language model; abbreviations can cause extra split
  points inside oversize paragraphs.
- Token counts are exact only for the named tiktoken encodings. Other embedding models (for example
  multilingual BERT-style models) tokenize differently; leave headroom or use `characters`.
- Near-duplicate removal is heuristic (see above); it is off by default.

### Development

```bash
pip install -r requirements.txt
python -m unittest discover -s test          # about 10 s
python -m src.cli examples/pagerecords.json --out examples/output --settings "$(cat examples/settings.json)"
python scripts/make_fixtures.py              # regenerate contracts/fixtures/page-records.v1.json
python scripts/live_smoke.py <actorId> inline dataset invalid --out report.json   # costs cents
python scripts/live_smoke.py <actorId> edges --out edges.json                     # billing edge measurement
## website, website_browser, website_resurrect, dataset_live_writer and attach_engine need website mode switched on
python scripts/check_producer_contract.py --commit <page-reader commit> [--dataset <crawl dataset>]
python scripts/deployment_check.py pre-merge|post-merge
```

The CLI runs the same pipeline locally: PageRecords in, `chunks.jsonl`, `documents.jsonl` and
`manifest.json` out. The committed example output is checked by the tests.

### Dependencies and licenses

Pinned in `requirements.txt`: `apify` 4.0.2 (Apache-2.0), `tiktoken` 0.14.0 (MIT; downloads its BPE files at
image build time and verifies their SHA-256), `markdown-it-py` 4.2.0 (MIT), `numpy` 2.5.3 (BSD-3-Clause, near
dedup only), `jsonschema` 4.26.0 (MIT). Main transitive dependencies: `regex` (Apache-2.0), `requests`
(Apache-2.0), `mdurl` (MIT), `referencing`, `rpds-py`, `attrs`, `jsonschema-specifications` (MIT). The
chunker, normalization and deduplication are LeadProof code; no third-party RAG or crawling service is called.

***

*Built by [LeadProof](https://leadproof.co) - verified local-business lead lists, built to order.*

# Actor input Schema

## `records` (type: `array`):

Pages to chunk, as PageRecord v1 objects. Minimum per record: a URL (finalUrl or requestedUrl) and markdown or text. Records with status failed or skipped are reported, not chunked. Optional sourceId sets your own stable document ID. Up to 10,000 records; use datasetId for larger inputs.

## `datasetId` (type: `string`):

An Apify dataset whose items are PageRecords, for example the output of a crawler or page reader run. If the run that writes it is still running, this run waits for it to finish (see Source wait) and never reads a dataset that is still growing. Only text fields are requested, never HTML.

## `sourceWaitSeconds` (type: `integer`):

How long to wait for the run that writes datasetId to finish. If it is still running after this, the run fails with the retryable SOURCE\_STILL\_RUNNING and reads nothing.

## `tokenizer` (type: `string`):

How sizes are measured. tiktoken counts are exact for the named encoding. The characters mode counts Unicode code points and is labeled as such in every row; it is not a model token count.

## `chunkSize` (type: `integer`):

Chunks are packed up to this many tokens (or characters).

## `maxChunkSize` (type: `integer`):

Hard limit, never exceeded. Defaults to the target size. Set it higher to keep tables, code blocks and list items whole when they are a bit larger than the target.

## `chunkOverlap` (type: `integer`):

Tokens repeated from the end of the previous chunk of the same section. Must be smaller than the target size. Default: 64, or one eighth of the target size if that is smaller.

## `sourceField` (type: `string`):

Markdown keeps headings, lists, tables and code fences, which the chunker uses as boundaries.

## `dedupeMode` (type: `string`):

Exact removes identical documents from different URLs (keeping every source URL on the kept chunks) and identical chunks such as repeated footers. Near also removes chunks whose 5-word shingle similarity is at least the threshold, unless they differ in any number, price, date, email, URL or negation.

## `nearDuplicateThreshold` (type: `number`):

Exact Jaccard similarity of 5-word shingles needed to remove a chunk in near mode. At 0.95 one changed word in a 200-word chunk is kept apart. Removed chunks and the words that differed are listed in the DOCUMENTS report.

## `includeUrlPatterns` (type: `array`):

Glob patterns (\* matches anything, including /). When set, only matching source URLs are chunked. Example: https://docs.example.com/\*

## `excludeUrlPatterns` (type: `array`):

Glob patterns of source URLs to skip, for example */tag/*.

## `languages` (type: `array`):

Primary language codes to keep, such as en or he. Records with a different known language are skipped; records without a language are kept.

## `minDocumentCharacters` (type: `integer`):

Documents with fewer normalized characters are skipped as too\_short.

## `maxDocuments` (type: `integer`):

Records read from the source. Further records are counted as omitted in the manifest.

## `maxDocumentCharacters` (type: `integer`):

Longer documents are truncated at a paragraph break and marked partial.

## `maxTotalCharacters` (type: `integer`):

Once reached, remaining documents are skipped as input\_budget\_exhausted.

## `maxChunks` (type: `integer`):

Output cap. Documents are emitted whole; the run stops before a document that would exceed the cap.

## `saveNormalizedSources` (type: `boolean`):

Store each chunked document's normalized text in the key-value store as source-<documentId>, so offsets can be checked against it. Adds one storage write per document.

## Actor input object example

````json
{
  "records": [
    {
      "schemaVersion": "1.0",
      "inputId": "example-en",
      "status": "succeeded",
      "finalUrl": "https://docs.example.com/sync/guide",
      "title": "Acme Sync Guide",
      "language": "en",
      "markdown": "# Acme Sync Guide\n\nAcme Sync copies files between laptops and the cloud.\n\n## Pricing\n\n| Plan | Price |\n|------|-------|\n| Basic | $10 |\n| Pro | $49 |\n\n## Installation\n\n```bash\ncurl -fsSL https://example.com/install.sh | sh\n```"
    },
    {
      "schemaVersion": "1.0",
      "inputId": "example-he",
      "status": "succeeded",
      "finalUrl": "https://gym.example.co.il/membership",
      "title": "מדריך מנוי",
      "language": "he",
      "markdown": "# מדריך מנוי\n\nניתן לבטל את המנוי בכל עת בהודעה של 30 יום מראש.\n\n## שעות פתיחה\n\n- ראשון עד חמישי: 06:00 עד 23:00\n- שבת: סגור"
    }
  ],
  "sourceWaitSeconds": 900,
  "tokenizer": "cl100k_base",
  "chunkSize": 512,
  "sourceField": "auto",
  "dedupeMode": "exact",
  "nearDuplicateThreshold": 0.95,
  "minDocumentCharacters": 1,
  "maxDocuments": 1000,
  "maxDocumentCharacters": 500000,
  "maxTotalCharacters": 20000000,
  "maxChunks": 20000,
  "saveNormalizedSources": false
}
````

# Actor output Schema

## `chunks` (type: `string`):

Full chunk records with offsets, source references and metadata.

## `jsonl` (type: `string`):

One chunk per line, for vector-store loaders.

## `manifest` (type: `string`):

Counts, skipped reasons, deduplication measurements, settings fingerprint and billing summary.

## `documents` (type: `string`):

One line per input record: chunked, skipped (with reason) or failed, and duplicates removed. Documents after the first 5,000 continue in the record DOCUMENTS-002.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

````javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "records": [
        {
            "schemaVersion": "1.0",
            "inputId": "example-en",
            "status": "succeeded",
            "finalUrl": "https://docs.example.com/sync/guide",
            "title": "Acme Sync Guide",
            "language": "en",
            "markdown": "# Acme Sync Guide\n\nAcme Sync copies files between laptops and the cloud.\n\n## Pricing\n\n| Plan | Price |\n|------|-------|\n| Basic | $10 |\n| Pro | $49 |\n\n## Installation\n\n```bash\ncurl -fsSL https://example.com/install.sh | sh\n```"
        },
        {
            "schemaVersion": "1.0",
            "inputId": "example-he",
            "status": "succeeded",
            "finalUrl": "https://gym.example.co.il/membership",
            "title": "מדריך מנוי",
            "language": "he",
            "markdown": "# מדריך מנוי\n\nניתן לבטל את המנוי בכל עת בהודעה של 30 יום מראש.\n\n## שעות פתיחה\n\n- ראשון עד חמישי: 06:00 עד 23:00\n- שבת: סגור"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("leadproof/rag-dataset-builder").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

````

## Python example

````python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "records": [
        {
            "schemaVersion": "1.0",
            "inputId": "example-en",
            "status": "succeeded",
            "finalUrl": "https://docs.example.com/sync/guide",
            "title": "Acme Sync Guide",
            "language": "en",
            "markdown": """# Acme Sync Guide

Acme Sync copies files between laptops and the cloud.

## Pricing

| Plan | Price |
|------|-------|
| Basic | $10 |
| Pro | $49 |

## Installation

```bash
curl -fsSL https://example.com/install.sh | sh
```""",
        },
        {
            "schemaVersion": "1.0",
            "inputId": "example-he",
            "status": "succeeded",
            "finalUrl": "https://gym.example.co.il/membership",
            "title": "מדריך מנוי",
            "language": "he",
            "markdown": """# מדריך מנוי

ניתן לבטל את המנוי בכל עת בהודעה של 30 יום מראש.

## שעות פתיחה

- ראשון עד חמישי: 06:00 עד 23:00
- שבת: סגור""",
        },
    ] }

# Run the Actor and wait for it to finish
run = client.actor("leadproof/rag-dataset-builder").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

````

## CLI example

````bash
echo '{
  "records": [
    {
      "schemaVersion": "1.0",
      "inputId": "example-en",
      "status": "succeeded",
      "finalUrl": "https://docs.example.com/sync/guide",
      "title": "Acme Sync Guide",
      "language": "en",
      "markdown": "# Acme Sync Guide\\n\\nAcme Sync copies files between laptops and the cloud.\\n\\n## Pricing\\n\\n| Plan | Price |\\n|------|-------|\\n| Basic | $10 |\\n| Pro | $49 |\\n\\n## Installation\\n\\n```bash\\ncurl -fsSL https://example.com/install.sh | sh\\n```"
    },
    {
      "schemaVersion": "1.0",
      "inputId": "example-he",
      "status": "succeeded",
      "finalUrl": "https://gym.example.co.il/membership",
      "title": "מדריך מנוי",
      "language": "he",
      "markdown": "# מדריך מנוי\\n\\nניתן לבטל את המנוי בכל עת בהודעה של 30 יום מראש.\\n\\n## שעות פתיחה\\n\\n- ראשון עד חמישי: 06:00 עד 23:00\\n- שבת: סגור"
    }
  ]
}' |
apify call leadproof/rag-dataset-builder --silent --output-dataset

````

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,leadproof/rag-dataset-builder"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/f6NtZ5ZtFXeIofCX4/builds/bMW9VD6OFHRMx5iXH/openapi.json