RAG Dataset Builder

Turn web page records into deduplicated, source-linked text chunks that are ready to embed and load into a retrieval system. Every chunk carries its source URL, its heading path, an exact token count, a stable ID and character offsets back into the source text.

This Actor prepares data. It does not create embeddings, run a vector database or answer questions. Chunks are not searchable until you embed them and load them into your own vector store or search engine.

What you give it

Exactly one source per run:

Source	Field	Notes
Inline PageRecords	records	Up to 10,000 objects. Minimum per record: a URL (finalUrl or requestedUrl) and markdown or text.
Apify dataset of PageRecords	datasetId	For example the output of a crawler or page reader run. If the run writing it is still running, this Actor waits for it to finish (sourceWaitSeconds). Read page by page; only text fields are requested, never HTML. The run gets read access to that dataset only.
Website	-	Not in this release. Crawling a site and chunking it in one run is implemented but switched off while its pricing is settled; a run that asks for it is refused. Crawl with any PageRecord producer and pass the dataset.

Records follow PageRecord v1 (contracts/page-record.v1.consumer.schema.json; the canonical schema belongs to the LeadProof Web Page Reader). Records with status failed or skipped, HTTP 4xx/5xx or no text are not chunked; each gets a reason in the document report. An optional sourceId on a record sets your own stable document identity (for example a product SKU or CMS ID).

Main settings (all validated; the run fails with a clear message on invalid values):

Field	Default	Meaning
tokenizer	cl100k_base	cl100k_base or o200k_base (tiktoken, exact counts), or characters (Unicode code points, labeled as not model tokens in every row).
chunkSize	512	Target size. Chunks are packed up to it.
maxChunkSize	= chunkSize	Hard limit, never exceeded. Raise it to keep tables, code blocks and list items whole when they are slightly larger than the target.
chunkOverlap	64 (or chunkSize/8)	Tokens repeated from the end of the previous chunk of the same section. Must be smaller than chunkSize.
sourceField	auto	auto uses markdown when present (keeps structure), else text.
dedupeMode	exact	none, exact, or near (see Deduplication).
includeUrlPatterns, excludeUrlPatterns	none	Glob patterns on the source URL (* matches anything, including /).
languages	none	Keep only these primary language codes (en, he); records without a declared language are kept.
maxDocuments, maxDocumentCharacters, maxTotalCharacters, maxChunks	1,000 / 500,000 / 20,000,000 / 20,000	Input and output caps (see Limits).
sourceWaitSeconds	900	How long to wait for the run that writes datasetId.
saveNormalizedSources	false	Also store each document's normalized text as source-<documentId> so offsets can be checked.

Website mode (not in this release)

Chunking a site in one run, by crawling it with the LeadProof Website Crawler first, is built and tested: one crawl per run with page, time and cost limits, the crawl reattached rather than repeated after a restart, and a crawl stopped by a limit reported as partial coverage. It is switched off here because the waiting time and the low-yield runs it can produce are not covered by this Actor's price yet. Input that asks for it is refused with a message, in the Console and through the API alike.

Until it returns: run a crawler (the LeadProof Website Crawler, or anything that writes PageRecord v1), then pass its dataset as datasetId. That is the same pipeline, with the crawl billed to you as its own run.

Sources are read only when finished

A source is never read while it is still being written, so a momentarily empty page can never be mistaken for the end of the data:

• datasetId: if the dataset belongs to an Apify run that is still running, this Actor waits for that run (sourceWaitSeconds, default 900). If it is still running after that, the run fails with the retryable SOURCE_STILL_RUNNING and reads nothing. If the producing run is not visible to this Actor, the dataset is read as it is, with the warning source_run_not_visible. Continuously written sources are not supported.
• website (off in this release): the crawl would be awaited to the end.
• The dataset's itemCount is never used as a read limit (Apify updates it asynchronously; measured 1,200 against 1,500 items right after a push). Pages are read until an empty page; after the producer has finished, a higher itemCount only triggers a few short retries.

What you get

1. Dataset: one row per chunk. Also available as JSONL: {{apiDefaultDatasetUrl}}/items?format=jsonl&clean=true (linked in the run's Output tab and in MANIFEST.outputs.chunksJsonl).
2. MANIFEST (key-value store): status, counts, skipped reasons, deduplication measurements, the settings and their fingerprints, billing summary.
3. DOCUMENTS (key-value store, JSONL): one line per input record: chunked, skipped with a reason, or failed; duplicate chunks removed and what they duplicated. Records after the first 5,000 continue in DOCUMENTS-002.

A real row (from examples/output/chunks.jsonl, shortened):

{
  "schemaVersion": "1.0", "inputId": "he-guide", "status": "succeeded", "warnings": [], "error": null,
  "chunkId": "chk_6920b44b7a5e599c08e7fd0d60a0181e",
  "documentId": "doc_1b17d4b5757827b0d20f8295c6818492",
  "sourceUrl": "https://gym.example.co.il/membership",
  "title": "מדריך מנוי",
  "headingPath": ["מדריך מנוי לחדר הכושר", "מחירים"],
  "text": "## מחירים\n\n| מסלול | מחיר חודשי | כניסות |\n|-------|-----------|--------|\n| בסיסי | 150 ₪ | 8 |\n| מלא | 290 ₪ | ללא הגבלה |",
  "tokenCount": 80,
  "tokenizer": {"name": "cl100k_base", "library": "tiktoken", "libraryVersion": "0.14.0", "unit": "token", "exactModelTokens": true},
  "chunkIndex": 1,
  "contentHash": "sha256:357696da...",
  "offsets": {"start": 116, "end": 239, "unit": "unicode_code_point", "utf8Start": 205, "utf8End": 374,
              "basis": "normalized_source_text", "sourceField": "markdown", "sourceHash": "sha256:ba614461...",
              "sourceCharacters": 449, "normalization": "rag-normalize-v1", "overlapWithPrevious": null},
  "sourceReferences": [{"inputId": "he-guide", "url": "https://gym.example.co.il/membership", "...": "..."}],
  "metadata": {"language": "he", "blockTypes": ["heading", "table"], "continuation": [], "documentChunkCount": 4, "...": "..."}
}

Row contract: contracts/chunk-record.v1.schema.json. Manifest contract: contracts/manifest.v1.schema.json.

Offsets and source text

text is always exactly normalizedSource[offsets.start:offsets.end]. normalizedSource is the record's offsets.sourceField (markdown or text) after rag-normalize-v1: Unicode NFC; CRLF, CR and Unicode line separators become LF; control characters except LF and TAB and the BOM are removed (RLM/LRM and joiners are kept); trailing spaces are removed from each line; outside code fences, runs of blank lines collapse to one; the document is trimmed. If the document was longer than maxDocumentCharacters, it is cut at a paragraph break first, and sourceHash is the hash of the text actually chunked. Offsets are given in Unicode code points (Python string indices) and in UTF-8 bytes. overlapWithPrevious gives the range shared with the previous chunk. The normalization code is src/normalize.py; with saveNormalizedSources the exact text is stored per document.

IDs

• documentId = hash of (source key, source field, normalized-content hash). The source key is your sourceId, else the final URL, requested URL, canonical URL or inputId. Two records share a document ID only when both the key and the text are identical, so different input records are never merged by accident. The same URL with different content produces two documents and a duplicate_source_key warning.
• chunkId = hash of (document ID, chunking fingerprint, position, offsets, text hash). It is the same for unchanged content and unchanged chunking settings (tokenizer, sizes, overlap, normalization and chunker versions). Filters, limits and deduplication never change IDs.
• The output is deterministic: the same input and settings give byte-identical rows on every run.

How chunking works

• Markdown is split into blocks with markdown-it-py (CommonMark plus GFM tables): headings, paragraphs, lists, code fences, tables, quotes and HTML blocks. Plain text is split at blank lines.
• A heading starts a new section; consecutive headings form one section. Chunks never cross a section boundary, and each chunk carries the full heading ancestry (headingPath).
• Blocks are packed up to chunkSize. A block that fits in maxChunkSize is never split.
• A larger block is split deterministically: list items, table rows (the header stays with its delimiter row) or code lines first, then sentences, then token windows snapped to whitespace, then raw token windows (never separating a combining mark from its letter). Each chunk holding a piece records metadata.continuation (block type, part N of M, split method, code language, and the table header for later parts).
• Overlap is applied only between chunks of the same section, starts at a sentence or word boundary, and is measured exactly; it never exceeds chunkOverlap.
• Text such as <|endoftext|> is counted as ordinary text, never as a tokenizer control token.

Deduplication

• exact (default): a record whose normalized text is identical to an earlier one (for example the same page on a mirror URL) is not chunked again; the kept chunks list every source in sourceReferences. Chunks with byte-identical text (repeated footers, cookie banners) are emitted once; the report maps each removed chunk to the kept chunk ID.
• near: exact, plus removal of chunks that are almost the same. Candidates come from MinHash LSH over 5-word shingles (lowercased, punctuation and diacritics ignored); a removal requires the exact Jaccard similarity of the two shingle sets to reach nearDuplicateThreshold (default 0.95). A pair is never merged when the two chunks differ in any number, price, percentage, date, email, URL or negation word (English and Hebrew). Every removal is listed with its similarity and the words that differed. Measured on the fixtures: the "$49" and "$59" versions of a plan page (similarity 0.945) are kept apart at thresholds 0.95, 0.9 and 0.8; a copy with only case and punctuation changes (similarity 1.0) is removed; a copy with one changed word ("every day" / "each day", 0.945) is kept at 0.95 and removed at 0.9. Near mode can therefore merge chunks that differ by a few ordinary words; use exact when every wording difference matters.

Limits

• maxDocuments bounds the records read; the manifest reports records omitted by the limit.
• maxDocumentCharacters bounds each document (cut at a paragraph break, marked partial).
• maxTotalCharacters bounds the run; later documents are skipped as input_budget_exhausted.
• maxChunks bounds the output; documents are emitted whole, and the run stops before a document that would exceed the cap (chunk_limit).
• Datasets are read in pages sized to about 8 million characters of text, never all at once.
• Apify dataset items are at most 9 MB each, so one input record is bounded; the first page of a dataset is read with 10 items, later pages are sized from the text volume seen.

Reliability

• Runs checkpoint after each batch (about 100 checkpoints per run). After a migration, crash or abort-and-resurrect, the run continues from the last checkpoint and never emits a chunk ID twice. Verified live: a run aborted after 4,200 of 13,500 rows and resurrected produced exactly the rows of a clean run.
• Website mode, while it was open, reattached to its crawl after an abort and resurrect instead of starting a second one, and produced the rows of a clean run.
• Datasets are read until an empty page. Apify's dataset itemCount is not trusted as a limit: right after a producer pushes items it can lag behind them (measured 1,200 vs 1,500).
• One document that fails to process is reported as failed (processing_error) and the run continues.
• A run where no document produced chunks ends as failed, with the reasons in MANIFEST.

Pricing

$0.002 per chunked document (document-chunked), plus Apify's standard Actor start event ($0.00005 per GB of run memory). Nothing else is charged.

One event covers one document that produced chunks, per started 100,000 characters or per started 250 chunks, whichever is larger, never their sum:

Document	Events
An ordinary page (a few thousand characters, a handful of chunks)	1
100,000 characters in 40 chunks	1
100,001 characters in 40 chunks	2
5,000 characters in 251 chunks	2
150,000 characters in 300 chunks	2

Free: documents that are skipped (failed page, empty page, HTTP error, filtered out), documents collapsed as duplicates of another document, and chunks removed as duplicates. A restarted or resurrected run never charges a document twice, and a run stops before delivering a document its spending limit cannot pay for. The automatic per-dataset-item event must stay off: the Actor refuses to run if it has a price, so diagnostic rows are never billed.

So 1,000 ordinary pages cost about $2. Your Apify account also pays the usual platform storage for the dataset it receives.

Fair use. Bring your own content: pages you crawled or own. This Actor does not fetch anything itself in this release, it only reshapes records you pass in. Chunk text is copied from your source as is, so treat licensing of that text as your responsibility.

Security and untrusted content

• Page text is data. The Actor never follows instructions in it, runs no LLM and executes nothing from it. Downstream RAG systems should also treat retrieved chunks as untrusted content.
• Records and dataset modes make no web requests. The only network calls go to the Apify API for the input dataset and the run's own storages. No secrets are required.
• Limited Actor permissions: the run can read only the dataset you pick.

Limitations

• Website mode is off in this release (see above); this Actor makes no web requests of its own.
• Markdown structure is only as good as the producer's markdown; plain text is chunked by paragraphs.
• Sentence splitting uses punctuation rules, not a language model; abbreviations can cause extra split points inside oversize paragraphs.
• Token counts are exact only for the named tiktoken encodings. Other embedding models (for example multilingual BERT-style models) tokenize differently; leave headroom or use characters.
• Near-duplicate removal is heuristic (see above); it is off by default.

Development

pip install -r requirements.txt
python -m unittest discover -s test          # about 10 s
python -m src.cli examples/pagerecords.json --out examples/output --settings "$(cat examples/settings.json)"
python scripts/make_fixtures.py              # regenerate contracts/fixtures/page-records.v1.json
python scripts/live_smoke.py <actorId> inline dataset invalid --out report.json   # costs cents
python scripts/live_smoke.py <actorId> edges --out edges.json                     # billing edge measurement
# website, website_browser, website_resurrect, dataset_live_writer and attach_engine need website mode switched on
python scripts/check_producer_contract.py --commit <page-reader commit> [--dataset <crawl dataset>]
python scripts/deployment_check.py pre-merge|post-merge

The CLI runs the same pipeline locally: PageRecords in, chunks.jsonl, documents.jsonl and manifest.json out. The committed example output is checked by the tests.

Dependencies and licenses

Pinned in requirements.txt: apify 4.0.2 (Apache-2.0), tiktoken 0.14.0 (MIT; downloads its BPE files at image build time and verifies their SHA-256), markdown-it-py 4.2.0 (MIT), numpy 2.5.3 (BSD-3-Clause, near dedup only), jsonschema 4.26.0 (MIT). Main transitive dependencies: regex (Apache-2.0), requests (Apache-2.0), mdurl (MIT), referencing, rpds-py, attrs, jsonschema-specifications (MIT). The chunker, normalization and deduplication are LeadProof code; no third-party RAG or crawling service is called.

---

Built by LeadProof - verified local-business lead lists, built to order.