# Semantic RAG Document Chunker and Token Slicer (`n4d.h1dr/rag-smart-chunker`) Actor

Deterministic document chunker that splits text, markdown, and tables without cutting sentences or code blocks in half.

- **URL**: https://apify.com/n4d.h1dr/rag-smart-chunker.md
- **Developed by:** [Nadim Hider](https://apify.com/n4d.h1dr) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 document chunkeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Semantic RAG Document Chunker and Token Slicer

A high-performance document segmenter designed for Retrieval Augmented Generation (RAG) and vector databases. Uses exact token counting algorithms to slice large text, Markdown, or HTML files into context-aware chunks without breaking sentences, Markdown tables, or code blocks in half.

***

### Why Use This Actor?

Standard character-length chunkers blindly cut paragraphs and code blocks mid-sentence, causing retrieval distortion and embedding corruption. This tool provides:

- Exact token budgeting based on `cl100k_base` (OpenAI / Anthropic compatible).
- Semantic boundary preservation (code fences, markdown headings, list items).
- Heading breadcrumb propagation so each chunk retains its parent outline context.
- Configurable overlap token windows for smooth context transitions.

***

### Dual Output Envelope (Agent Native)

Every record contains both structured chunks with metadata and a concise markdown report:

```json
{
  "id": "doc-001",
  "success": true,
  "meta": {
    "processing_time_ms": 1.8,
    "items_count": 2
  },
  "data": {
    "document_id": "doc-001",
    "total_chunks": 2,
    "total_tokens": 140,
    "chunks": [
      {
        "chunk_index": 0,
        "content": "# Introduction to Autonomous RAG\nRetrieval Augmented Generation combines information retrieval with language modeling.",
        "token_count": 75,
        "headings": ["# Introduction to Autonomous RAG"]
      }
    ]
  },
  "markdown_summary": "### Document Chunking Summary\n- Document ID: doc-001\n- Chunks Generated: 2\n- Total Tokens: 140\n- Latency: 1.8ms\n"
}
```

***

### Input Schema

| Field | Type | Default | Description |
|---|---|---|---|
| `items` | Array of Objects | *Sample Document* | Array of objects with `id` and `text` to segment into chunks. |
| `max_tokens_per_chunk` | Integer | `200` | Maximum token limit per chunk. |
| `chunk_overlap_tokens` | Integer | `30` | Overlap window size between adjacent chunks. |

***

### Pay-Per-Event (PPE) Pricing

- Primary Event: `document-chunked`
- Cost: **$0.001 per document chunked** ($1.00 per 1,000 documents).
- Initial 5 seconds of compute are waived via `apify-actor-start`.

***

### Integration Code in 10+ Languages

#### cURL

```bash
curl -X POST "https://api.apify.com/v2/acts/n4d.h1dr~rag-smart-chunker/runs?token=YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"items": [{"id": "doc1", "text": "# Header\nLong text..."}], "max_tokens_per_chunk": 250}'
```

#### Python

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("n4d.h1dr/rag-smart-chunker").call(run_input={
    "items": [{"id": "doc1", "text": "Article content..."}],
    "max_tokens_per_chunk": 300
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"Chunks: {item['data']['total_chunks']}")
```

#### Node.js / TypeScript

```typescript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_API_TOKEN' });
const run = await client.actor('n4d.h1dr/rag-smart-chunker').call({
  items: [{ id: 'doc1', text: 'Text to split...' }]
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0].data.chunks);
```

#### Go

```go
package main
import (
    "bytes"
    "net/http"
)
func main() {
    url := "https://api.apify.com/v2/acts/n4d.h1dr~rag-smart-chunker/runs?token=YOUR_API_TOKEN"
    payload := []byte(`{"items": [{"id": "doc1", "text": "Content..."}]}`)
    http.Post(url, "application/json", bytes.NewBuffer(payload))
}
```

#### Ruby

```ruby
require 'net/http'
require 'json'
uri = URI('https://api.apify.com/v2/acts/n4d.h1dr~rag-smart-chunker/runs?token=YOUR_API_TOKEN')
req = Net::HTTP::Post.new(uri, 'Content-Type' => 'application/json')
req.body = { items: [{ id: 'doc1', text: 'Content to chunk' }] }.to_json
res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) { |http| http.request(req) }
```

#### PHP

```php
<?php
$ch = curl_init('https://api.apify.com/v2/acts/n4d.h1dr~rag-smart-chunker/runs?token=YOUR_API_TOKEN');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([
    'items' => [['id' => '1', 'text' => 'Long text to slice']]
]));
curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/json']);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$res = curl_exec($ch);
```

#### Java

```java
import java.net.URI;
import java.net.http.*;
public class Main {
    public static void main(String[] args) throws Exception {
        HttpClient client = HttpClient.newHttpClient();
        HttpRequest request = HttpRequest.newBuilder()
            .uri(URI.create("https://api.apify.com/v2/acts/n4d.h1dr~rag-smart-chunker/runs?token=YOUR_API_TOKEN"))
            .header("Content-Type", "application/json")
            .POST(HttpRequest.BodyPublishers.ofString("{\"items\": [{\"id\": \"1\", \"text\": \"Document...\"}]}"))
            .build();
        client.send(request, HttpResponse.BodyHandlers.ofString());
    }
}
```

#### C# (.NET)

```csharp
using System.Net.Http.Json;
var client = new HttpClient();
var payload = new { items = new[] { new { id = "1", text = "Document to chunk" } } };
await client.PostAsJsonAsync("https://api.apify.com/v2/acts/n4d.h1dr~rag-smart-chunker/runs?token=YOUR_API_TOKEN", payload);
```

#### Rust

```rust
use reqwest::Client;
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let client = Client::new();
    let res = client.post("https://api.apify.com/v2/acts/n4d.h1dr~rag-smart-chunker/runs?token=YOUR_API_TOKEN")
        .json(&serde_json::json!({"items": [{"id": "1", "text": "Document..."}]}))
        .send().await?;
    Ok(())
}
```

#### Swift

```swift
import Foundation
var request = URLRequest(url: URL(string: "https://api.apify.com/v2/acts/n4d.h1dr~rag-smart-chunker/runs?token=YOUR_API_TOKEN")!)
request.httpMethod = "POST"
request.setValue("application/json", forHTTPHeaderField: "Content-Type")
request.httpBody = try? JSONSerialization.data(withJSONObject: ["items": [["id": "1", "text": "RAG text"]]])
URLSession.shared.dataTask(with: request).resume()
```

#### n8n / Make.com Webhook Node

Add an HTTP Request node in n8n or Make:

- Method: `POST`
- URL: `https://api.apify.com/v2/acts/n4d.h1dr~rag-smart-chunker/runs?token=YOUR_API_TOKEN`
- Body Parameters: `{ "items": [ { "id": "={{ $json.id }}", "text": "={{ $json.raw_doc }}" } ], "max_tokens_per_chunk": 300 }`

# Actor input Schema

## `items` (type: `array`):

List of documents containing document text or markdown to segment into token exact chunks.

## `max_tokens_per_chunk` (type: `integer`):

Maximum token limit per chunk using cl100k_base tokenizer encoding.

## `chunk_overlap_tokens` (type: `integer`):

Number of overlapping tokens between sequential chunks to preserve cross sentence context.

## Actor input object example

````json
{
  "items": [
    {
      "id": "doc-001",
      "text": "# Introduction to Autonomous RAG\n\nRetrieval Augmented Generation combines information retrieval with language modeling.\n\n## Vector Embeddings\nEmbeddings convert semantic meaning into dense mathematical vectors.\n\n```python\ndef get_embedding(text):\n    return model.encode(text)\n```\n\n## Conclusion\nChunking preservation avoids truncating code blocks or tables."
    }
  ],
  "max_tokens_per_chunk": 200,
  "chunk_overlap_tokens": 30
}
````

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

````javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        {
            "id": "doc-001",
            "text": "# Introduction to Autonomous RAG\n\nRetrieval Augmented Generation combines information retrieval with language modeling.\n\n## Vector Embeddings\nEmbeddings convert semantic meaning into dense mathematical vectors.\n\n```python\ndef get_embedding(text):\n    return model.encode(text)\n```\n\n## Conclusion\nChunking preservation avoids truncating code blocks or tables."
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("n4d.h1dr/rag-smart-chunker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

````

## Python example

````python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "items": [{
            "id": "doc-001",
            "text": """# Introduction to Autonomous RAG

Retrieval Augmented Generation combines information retrieval with language modeling.

## Vector Embeddings
Embeddings convert semantic meaning into dense mathematical vectors.

```python
def get_embedding(text):
    return model.encode(text)
````

## Conclusion

Chunking preservation avoids truncating code blocks or tables.""",
}] }

# Run the Actor and wait for it to finish

run = client.actor("n4d.h1dr/rag-smart-chunker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)

print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

````

## CLI example

```bash
echo '{
  "items": [
    {
      "id": "doc-001",
      "text": "# Introduction to Autonomous RAG\\n\\nRetrieval Augmented Generation combines information retrieval with language modeling.\\n\\n## Vector Embeddings\\nEmbeddings convert semantic meaning into dense mathematical vectors.\\n\\n```python\\ndef get_embedding(text):\\n    return model.encode(text)\\n```\\n\\n## Conclusion\\nChunking preservation avoids truncating code blocks or tables."
    }
  ]
}' |
apify call n4d.h1dr/rag-smart-chunker --silent --output-dataset

````

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,n4d.h1dr/rag-smart-chunker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lvhJA8Y4s3ApFL5Kl/builds/DCk81jrMgQZJTjIhl/openapi.json