# Rag Text Chunker (`eichan/rag-text-chunker`) Actor

- **URL**: https://apify.com/eichan/rag-text-chunker.md
- **Developed by:** [Eichan Jolly](https://apify.com/eichan) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 60.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.01 / actor start

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RAG Text Chunker — Word Overlap & Character Offsets

**Slug:** `rag-text-chunker`\
**Category:** AI / AI data preparation (not a chatbot)\
**Apify Store categories to select when publishing:** `AI`, `DEVELOPER_TOOLS` (2 of 3 available). These are publishing choices, **not** automatic settings in `actor.json`.\
**Suggested pricing:** **per dataset result** (one result = one chunk), with a maximum of **500 chunks per run**. Configure pricing separately in Apify Console; this repository does not set fees.

### What it does

Deterministically splits **one user-provided text** into overlapping, fixed-size groups of whitespace-delimited words. It emits one dataset item per chunk, including the precise start and end positions in the original input. The output can be used as input to RAG retrieval pipelines or text indexing systems. **No embeddings, semantic boundaries, AI summarization, or language analysis are performed.**

### How the method works

1. Find each whitespace-delimited token with Python's `re` standard library.
2. Take up to `chunk_size` consecutive words per chunk.
3. Advance by `chunk_size - overlap` words and repeat, until the final token is included.
4. Copy each chunk as an **exact substring of the original text**. Report `start` (inclusive) and `end` (exclusive) as **zero-based Python Unicode character offsets**.
5. Create one default-dataset item for each chunk. `chunk_id` counts upward from 1.

No login, web fetch, LLM API, external service, or third-party library is needed for processing. The standard **Apify platform token provided to an Actor run** is used only for its default storage API; customers do not supply a key.

### Input example

One short text:

```json
{
  "text": "Small chunks help retrieval systems find relevant passages quickly.",
  "chunk_size": 5,
  "overlap": 2
}
```

Defaults are `chunk_size: 100` and `overlap: 20` if omitted. The Console's prefilled text is deliberately tiny.

### Output example

Three items in the **default dataset**:

```json
[
  {
    "chunk_id": 1,
    "text": "Small chunks help retrieval systems",
    "start": 0,
    "end": 35,
    "word_count": 5
  },
  {
    "chunk_id": 2,
    "text": "retrieval systems find relevant passages",
    "start": 18,
    "end": 58,
    "word_count": 5
  },
  {
    "chunk_id": 3,
    "text": "relevant passages quickly.",
    "start": 41,
    "end": 67,
    "word_count": 3
  }
]
```

The illustrative input above produces **three** chunks. `text == original[start:end]` for every item.

| Field | Type | Meaning |
| --- | --- | --- |
| `chunk_id` | integer | 1-based chunk number within the run |
| `text` | string | Exact original substring |
| `start` | integer | Inclusive zero-based character offset |
| `end` | integer | Exclusive zero-based character offset |
| `word_count` | integer | Whitespace-delimited tokens in this chunk |

### Who this is for

Developers, data engineers, and RAG builders who need predictable chunk boundaries and offsets without relying on a tokenizer package or model provider.

### Limits and edge cases

- **One text per run**, at most **50,000 Unicode characters**. At most **500 chunks** can be produced.
- `chunk_size` must be an integer from **1 to 5,000**. `overlap` must be an integer from **0 to 4,999** and **strictly less than** `chunk_size`.
- Boundaries are based on **whitespace**, not sentences, subword tokens, semantic paragraphs, bytes, or grapheme clusters. Chinese/Japanese text without spaces may form a single token.
- Internal whitespace and line breaks are preserved because chunks are slices of the original input. Leading whitespace before the first token and trailing whitespace after the last token are not included.
- The final chunk can have fewer than `chunk_size` words; overlapping words intentionally repeat.
- Input errors exit nonzero with a simple `ERROR:` message in the run log and no traceback; they do **not** emit deceptive success rows.

### Running locally (no Apify SDK)

From the actor's root folder, create `storage/key_value_stores/default/INPUT.json` with the input example above. Run:

```bash
python src/main.py
```

Chunk items are written to `storage/datasets/default/000000001.json`, `000000002.json`, etc. Files append by increasing item number, so clear `storage/` between independent tests if needed.

### Running on Apify

Build using the included `Dockerfile`. The Actor uses the Apify platform-provided `APIFY_TOKEN` with the run's `ACTOR_DEFAULT_*` identifiers to read input from its default key-value store and append output to its default dataset over the official Apify HTTP API. **This is not an external AI API key and requires no user-provided credentials.** Local output files alone would not publish cloud dataset results.

`requirements.txt` deliberately contains no packages. Python standard library only.

# Actor input Schema

## `text` (type: `string`):

One text, up to 50,000 Unicode characters (no URL fetching).

## `chunk_size` (type: `integer`):

Maximum number of whitespace-delimited words in a chunk.

## `overlap` (type: `integer`):

Words reused from the previous chunk; must be smaller than chunk_size.

## Actor input object example

```json
{
  "text": "Small chunks help retrieval systems find relevant passages quickly.",
  "chunk_size": 100,
  "overlap": 20
}
```

# Actor output Schema

## `dataset` (type: `string`):

Dataset containing the Actor's result items

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "text": "Small chunks help retrieval systems find relevant passages quickly."
};

// Run the Actor and wait for it to finish
const run = await client.actor("eichan/rag-text-chunker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "text": "Small chunks help retrieval systems find relevant passages quickly." }

# Run the Actor and wait for it to finish
run = client.actor("eichan/rag-text-chunker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "text": "Small chunks help retrieval systems find relevant passages quickly."
}' |
apify call eichan/rag-text-chunker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,eichan/rag-text-chunker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/XxdNXRmVKHsvdzQzP/builds/4g0aSxDfLJM7jNmsB/openapi.json