RAG Text Chunker — Word Overlap & Character Offsets

Slug: rag-text-chunker
Category: AI / AI data preparation (not a chatbot)
Apify Store categories to select when publishing: AI, DEVELOPER_TOOLS (2 of 3 available). These are publishing choices, not automatic settings in actor.json.
Suggested pricing: per dataset result (one result = one chunk), with a maximum of 500 chunks per run. Configure pricing separately in Apify Console; this repository does not set fees.

What it does

Deterministically splits one user-provided text into overlapping, fixed-size groups of whitespace-delimited words. It emits one dataset item per chunk, including the precise start and end positions in the original input. The output can be used as input to RAG retrieval pipelines or text indexing systems. No embeddings, semantic boundaries, AI summarization, or language analysis are performed.

How the method works

1. Find each whitespace-delimited token with Python's re standard library.
2. Take up to chunk_size consecutive words per chunk.
3. Advance by chunk_size - overlap words and repeat, until the final token is included.
4. Copy each chunk as an exact substring of the original text. Report start (inclusive) and end (exclusive) as zero-based Python Unicode character offsets.
5. Create one default-dataset item for each chunk. chunk_id counts upward from 1.

No login, web fetch, LLM API, external service, or third-party library is needed for processing. The standard Apify platform token provided to an Actor run is used only for its default storage API; customers do not supply a key.

Input example

One short text:

{
  "text": "Small chunks help retrieval systems find relevant passages quickly.",
  "chunk_size": 5,
  "overlap": 2
}

Defaults are chunk_size: 100 and overlap: 20 if omitted. The Console's prefilled text is deliberately tiny.

Output example

Three items in the default dataset:

[
  {
    "chunk_id": 1,
    "text": "Small chunks help retrieval systems",
    "start": 0,
    "end": 35,
    "word_count": 5
  },
  {
    "chunk_id": 2,
    "text": "retrieval systems find relevant passages",
    "start": 18,
    "end": 58,
    "word_count": 5
  },
  {
    "chunk_id": 3,
    "text": "relevant passages quickly.",
    "start": 41,
    "end": 67,
    "word_count": 3
  }
]

The illustrative input above produces three chunks. text == original[start:end] for every item.

Field	Type	Meaning
chunk_id	integer	1-based chunk number within the run
text	string	Exact original substring
start	integer	Inclusive zero-based character offset
end	integer	Exclusive zero-based character offset
word_count	integer	Whitespace-delimited tokens in this chunk

Who this is for

Developers, data engineers, and RAG builders who need predictable chunk boundaries and offsets without relying on a tokenizer package or model provider.

Limits and edge cases

• One text per run, at most 50,000 Unicode characters. At most 500 chunks can be produced.
• chunk_size must be an integer from 1 to 5,000. overlap must be an integer from 0 to 4,999 and strictly less than chunk_size.
• Boundaries are based on whitespace, not sentences, subword tokens, semantic paragraphs, bytes, or grapheme clusters. Chinese/Japanese text without spaces may form a single token.
• Internal whitespace and line breaks are preserved because chunks are slices of the original input. Leading whitespace before the first token and trailing whitespace after the last token are not included.
• The final chunk can have fewer than chunk_size words; overlapping words intentionally repeat.
• Input errors exit nonzero with a simple ERROR: message in the run log and no traceback; they do not emit deceptive success rows.

Running locally (no Apify SDK)

From the actor's root folder, create storage/key_value_stores/default/INPUT.json with the input example above. Run:

$python src/main.py

Chunk items are written to storage/datasets/default/000000001.json, 000000002.json, etc. Files append by increasing item number, so clear storage/ between independent tests if needed.

Running on Apify

Build using the included Dockerfile. The Actor uses the Apify platform-provided APIFY_TOKEN with the run's ACTOR_DEFAULT_* identifiers to read input from its default key-value store and append output to its default dataset over the official Apify HTTP API. This is not an external AI API key and requires no user-provided credentials. Local output files alone would not publish cloud dataset results.

requirements.txt deliberately contains no packages. Python standard library only.