# Text Chunker & Embeddings for RAG (LangChain, LlamaIndex) (`steady_kahikatea/text-chunker-embeddings`) Actor

Split Markdown, text and web pages into heading-aware, token-sized chunks with overlap, and optionally embed them with a local CPU model (bge-small, MiniLM). No API key. Output works with LangChain, LlamaIndex, Pinecone, Qdrant and pgvector.

- **URL**: https://apify.com/steady_kahikatea/text-chunker-embeddings.md
- **Developed by:** [Nick Hellmich](https://apify.com/steady_kahikatea) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Text Chunker & Embeddings for RAG (LangChain, LlamaIndex)

Split Markdown, plain text and web pages into **heading-aware, token-sized chunks with overlap**, and optionally turn each chunk into a **vector embedding with a small model running inside this Actor on CPU**. No OpenAI key and no third-party API. The output drops straight into LangChain, LlamaIndex, n8n, Make, Zapier and vector databases (Pinecone, Qdrant, Weaviate, Chroma, pgvector).

Pairs with [Document to Markdown](https://apify.com/steady_kahikatea/document-to-markdown): convert PDFs/DOCX first, then pass that run's dataset ID here.

### Quick start

1. Click **Start**. The prefill chunks a short help-center article and embeds it with **bge-small-en-v1.5**.
2. Paste your own text into **Documents**, add **URLs**, or give a **dataset ID** from a document-to-markdown run.
3. Open the **Output** tab, or download every chunk as one **JSONL** file (`chunks.jsonl` in the key-value store).

### What you get (one dataset item per chunk)

```json
{
  "id": "5f525e2c5caa87629dd2",
  "text": "# Refund policy\n\nCustomers can request a full refund within 30 days of purchase. ...",
  "metadata": {"source": "help-center/refunds.md", "headings": ["Refund policy"], "section": "Refund policy",
               "chunkIndex": 0, "chunkCount": 3, "startChar": 0, "tokenCount": 35, "lang": "en"},
  "embeddingModel": "BAAI/bge-small-en-v1.5", "embeddingDim": 384,
  "embedding": [-0.033255, 0.030728, 0.014127, "... 384 numbers"]
}
```

`id` is stable for the same source, position and text, so re-running upserts instead of duplicating. `text` + `metadata` is exactly a LangChain `Document` / LlamaIndex `TextNode`. Vectors are L2-normalised: use cosine or dot product.

### How chunking works

- Splits at Markdown headings (code fences are respected) and stores the heading path as `section`.
- Packs paragraphs, then sentences, then words up to **Max tokens per chunk** (default 400 approximate subword tokens).
- Repeats about **Overlap tokens** (default 50) from the end of the previous chunk.
- The heading path is prepended to the text that gets embedded (not to `text`), which improves retrieval.

### Embedding models (all local, CPU)

| Option | Languages | Dim | Notes |
|---|---|---|---|
| none | – | – | Chunks only (cheapest) |
| BAAI/bge-small-en-v1.5 | English | 384 | Best quality for English retrieval |
| sentence-transformers/all-MiniLM-L6-v2 | English | 384 | About 3x faster and cheaper; reads the first 256 tokens of a chunk (use chunk size ≤ 256) |
| sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2 | 50+ | 384 | Multilingual; reads the first 128 tokens (use chunk size ≤ 128) |

**Important:** embed your search queries with the **same model**. With bge-small, that's `FastEmbedEmbeddings(model_name="BAAI/bge-small-en-v1.5")` in LangChain or `FastEmbedEmbedding` in LlamaIndex, both free and local.

### Pricing (pay per event)

| Event | Price | When |
|---|---|---|
| Document chunked | $0.001 | Per source document that produced chunks |
| Chunk created | $0.00002 ($0.02 per 1,000) | Per chunk in the output |
| Chunk embedded | $0.00022 ($0.22 per 1,000) | Per chunk embedded with bge-small or the multilingual model |
| Chunk embedded (fast) | $0.00007 ($0.07 per 1,000) | Per chunk embedded with all-MiniLM-L6-v2 |

Bronze, Silver and Gold Apify plans get 10%, 20% and 30% off. Example: 1,000 chunks from 10 documents with bge-small cost $0.25 ($0.10 with MiniLM, $0.03 chunks only).
Failed sources and empty input are never charged. Set *Maximum cost per run* to cap spending.

### Integrations

Steps follow Apify's integration docs: [Zapier](https://docs.apify.com/platform/integrations/zapier), [Make](https://docs.apify.com/platform/integrations/make), [n8n](https://docs.apify.com/platform/integrations/n8n), [LangChain](https://docs.apify.com/platform/integrations/langchain), [LlamaIndex](https://docs.apify.com/platform/integrations/llama-index).

#### n8n: new Drive file → searchable knowledge base

*Google Drive trigger (new file)* → *Apify: Run an Actor and get dataset* → `document-to-markdown` → *Apify: Run an Actor and get dataset* → this Actor with `{"documents": {{ $json }}, "embeddingModel": "BAAI/bge-small-en-v1.5"}` → *Qdrant / Pinecone / Postgres (pgvector) node*: upsert `id`, `embedding`, payload `text` + `metadata`.

#### Make: help-center pages → vector DB every night

*Schedule* → *Apify Run an Actor* (`{"urls": ["https://example.com/help/a", "..."]}`) → a second scenario: *Watch Actor Runs* → *Get Dataset Items* → *HTTP* upsert into your vector DB's REST API.

#### Zapier: new Notion page → chunks

*Notion New Database Item* → Apify *Run Actor* (async) with `{"documents": [{"text": "{{Content}}", "source": "{{Page URL}}"}]}` → second Zap: *Finished Actor run* → *Fetch dataset items* → store wherever you search from.

#### LangChain

```python
from langchain_apify import ApifyWrapper
from langchain_core.documents import Document
from langchain_community.embeddings.fastembed import FastEmbedEmbeddings
from langchain_community.vectorstores import FAISS

loader = ApifyWrapper().call_actor(
    actor_id="steady_kahikatea/text-chunker-embeddings",
    run_input={"urls": ["https://docs.apify.com/platform/integrations/langchain"], "embeddingModel": "BAAI/bge-small-en-v1.5"},
    dataset_mapping_function=lambda r: Document(page_content=r["text"] or "", metadata={**r["metadata"], "embedding": r.get("embedding")}),
)
docs = [d for d in loader.load() if d.page_content]
emb = FastEmbedEmbeddings(model_name="BAAI/bge-small-en-v1.5")   # same model, for queries
pairs = [(d.page_content, d.metadata.pop("embedding")) for d in docs]
store = FAISS.from_embeddings(pairs, emb, metadatas=[d.metadata for d in docs])
print(store.similarity_search("How do I load Apify data into LangChain?", k=3))
```

#### LlamaIndex

```python
from llama_index.readers.apify import ApifyActor
from llama_index.core import Document
reader = ApifyActor("<APIFY_TOKEN>")
docs = reader.load_data(
    actor_id="steady_kahikatea/text-chunker-embeddings",
    run_input={"datasetId": "<document-to-markdown defaultDatasetId>", "embeddingModel": "BAAI/bge-small-en-v1.5"},
    dataset_mapping_function=lambda r: Document(text=r["text"] or "", metadata=r["metadata"], embedding=r.get("embedding")),
)
```

### Limits and safety

- Up to 500 sources and 20,000 chunks per run; files up to 20 MB.
- URLs must be public http(s); private, internal and cloud-metadata addresses are blocked.
- PDFs, Word files and images aren't parsed here: use document-to-markdown first.
- Your text is never logged.

# Actor input Schema

## `documents` (type: `array`):

Strings, or objects like {"text": "...", "source": "faq.md", "metadata": {...}}. Fields markdown, text, content, page_content and pageContent are also recognised, so document-to-markdown output rows can be pasted as-is.

## `urls` (type: `array`):

Public http(s) links to .md/.txt files or web pages; Google Docs and Drive/Dropbox share links are converted. For PDF/DOCX, run document-to-markdown first and pass its dataset ID below.

## `datasetId` (type: `string`):

Reads text from each item (markdown/text/content field). Pass the defaultDatasetId of a previous run.

## `datasetTextField` (type: `string`):

Field to read from dataset items. Empty = auto-detect.

## `chunkSize` (type: `integer`):

Approximate subword tokens. 200-500 suits most RAG setups.

## `chunkOverlap` (type: `integer`):

Tokens repeated from the end of the previous chunk (capped at half the chunk size).

## `splitByHeadings` (type: `boolean`):

Never mix sections in one chunk, and record the heading path in metadata.

## `embeddingModel` (type: `string`):

Runs locally on CPU inside the Actor: no API key. Vectors are L2-normalised (use cosine or dot product).

## `includeEmbeddings` (type: `boolean`):

Turn off to keep only text and metadata.

## Actor input object example

```json
{
  "documents": [
    {
      "source": "help-center/refunds.md",
      "text": "# Refund policy\n\nCustomers can request a full refund within 30 days of purchase. Refunds go back to the original payment method within 5 business days.\n\n## Exceptions\n\nGift cards and downloadable software are not refundable once the license key has been revealed.\n\n# Shipping\n\nOrders ship within 2 business days. Express shipping arrives in 1 to 2 days in the continental US.",
      "metadata": {
        "lang": "en"
      }
    }
  ],
  "chunkSize": 400,
  "chunkOverlap": 50,
  "splitByHeadings": true,
  "embeddingModel": "BAAI/bge-small-en-v1.5",
  "includeEmbeddings": true
}
```

# Actor output Schema

## `chunks` (type: `string`):

No description

## `jsonl` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "documents": [
        {
            "source": "help-center/refunds.md",
            "text": "# Refund policy\n\nCustomers can request a full refund within 30 days of purchase. Refunds go back to the original payment method within 5 business days.\n\n## Exceptions\n\nGift cards and downloadable software are not refundable once the license key has been revealed.\n\n# Shipping\n\nOrders ship within 2 business days. Express shipping arrives in 1 to 2 days in the continental US.",
            "metadata": {
                "lang": "en"
            }
        }
    ],
    "embeddingModel": "BAAI/bge-small-en-v1.5"
};

// Run the Actor and wait for it to finish
const run = await client.actor("steady_kahikatea/text-chunker-embeddings").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "documents": [{
            "source": "help-center/refunds.md",
            "text": """# Refund policy

Customers can request a full refund within 30 days of purchase. Refunds go back to the original payment method within 5 business days.

## Exceptions

Gift cards and downloadable software are not refundable once the license key has been revealed.

# Shipping

Orders ship within 2 business days. Express shipping arrives in 1 to 2 days in the continental US.""",
            "metadata": { "lang": "en" },
        }],
    "embeddingModel": "BAAI/bge-small-en-v1.5",
}

# Run the Actor and wait for it to finish
run = client.actor("steady_kahikatea/text-chunker-embeddings").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "documents": [
    {
      "source": "help-center/refunds.md",
      "text": "# Refund policy\\n\\nCustomers can request a full refund within 30 days of purchase. Refunds go back to the original payment method within 5 business days.\\n\\n## Exceptions\\n\\nGift cards and downloadable software are not refundable once the license key has been revealed.\\n\\n# Shipping\\n\\nOrders ship within 2 business days. Express shipping arrives in 1 to 2 days in the continental US.",
      "metadata": {
        "lang": "en"
      }
    }
  ],
  "embeddingModel": "BAAI/bge-small-en-v1.5"
}' |
apify call steady_kahikatea/text-chunker-embeddings --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,steady_kahikatea/text-chunker-embeddings"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/DF1epcGhBQz5FZcjx/builds/pkmguKx3DTJ3vWYGc/openapi.json