Text Chunker: Split Text & Documents into Chunks for RAG and Embeddings

Split long text or documents into properly sized, boundary-aware chunks with overlap, ready to embed into a vector database. Choose recursive (structure-aware), sentence-safe, or fixed-token-window chunking, each with configurable size and overlap.

Powered by Chonkie (MIT licensed, 4,500+ GitHub stars), an open-source chunking library. Chunk sizes are counted in real BPE tokens (the same style of tokenizer used by common LLMs), not raw characters, so a "512-token chunk" means what it says.

Great for: text chunker, chunk text for embeddings, RAG chunking tool, split text into chunks, text splitter for LLM, feeding a vector database (Pinecone, Qdrant, Weaviate, Chroma, pgvector...), building a RAG pipeline, and preparing knowledge-base articles or converted documents for an AI chatbot.

What it does

Give it text directly, or a list of URLs to plain-text/Markdown files (for example, the Markdown output of a PDF-to-Markdown converter). It splits each one into chunks sized and overlapped the way you configure, and returns one row per chunk with its token count and position in the source document. Nothing is stored beyond the run.

Chunking strategies

chunkingStrategy	What it does
recursive (default)	Smart general-purpose default. Splits on paragraph breaks first, then sentences, then words, packing as much as fits into each chunk. Best balance of clean boundaries and consistent chunk size.
sentence	Never splits a sentence in half. Packs whole sentences up to the size limit, even if that means a chunk runs a little over when a single sentence is long. Best when exact sentence integrity matters more than exact chunk size.
token	Fastest option. Fixed-size sliding token windows with no awareness of sentence or word boundaries, may cut mid-word. Best when you need a strict, predictable token budget (e.g. to exactly match an embedding model's limit).

All three strategies support overlap: a run of tokens repeated at the boundary between consecutive chunks, so an idea split across a chunk boundary still appears whole in at least one chunk.

Example input

{
  "texts": [
    "Paste one or more long pieces of text here..."
  ],
  "documentUrls": [
    "https://example.com/knowledge-base-article.md"
  ],
  "chunkingStrategy": "recursive",
  "chunkSize": 512,
  "chunkOverlap": 50
}

Output

One row per chunk:

Field	Description
source	Which input this chunk came from (Text #1, or the document URL)
sourceType	text or url
chunkIndex / totalChunks	This chunk's position and the total chunk count for its source
text	The chunk's text: the payload to embed
tokenCount	Token count of this chunk
characterCount	Character count of this chunk
startIndex / endIndex	Character offsets into the original source text, for citation/traceability
status	ok, empty (blank input), or error

Pricing

Billed per chunk produced, plus a small flat per-run fee. Failed or empty inputs are not charged.

FAQ

How do I chunk text for a RAG pipeline? Paste your text into texts (or point documentUrls at a hosted .txt/.md file) and run. Each output row is one chunk, ready to embed and upsert into your vector database.

What chunk size should I use? 512 tokens with 50 tokens of overlap is a reasonable default for most embedding models. Use smaller chunks (128–256 tokens) for precise retrieval over short facts, or larger chunks (1000+ tokens) when matches need more surrounding context.

Why is chunk size measured in tokens, not characters? Embedding models and LLMs have context limits measured in tokens, not characters, and one token is roughly 4 characters of English text but varies by content. Sizing chunks in real tokens means they actually fit the budget you're planning for.

Does this cut sentences in half? Only the token strategy does, by design, for a strict fixed-size window. sentence never does. recursive respects sentence and paragraph boundaries for the main chunk split; only its overlap region can include a few tokens of the next chunk's opening as extra context.

Can I chunk the output of a PDF-to-Markdown converter? Yes. Run your PDF through a document-to-Markdown actor, then feed the resulting Markdown URL (or pasted text) into documentUrls or texts here.

Does this call any AI model or send my text anywhere for processing? No. Chunking is deterministic and local to the run: no embedding model, no LLM, no external API calls other than fetching a URL you provide.

Please note

Only chunk text/documents you have the right to process. Input is processed transiently and not retained beyond the run.

Built with Apify Python SDK + Chonkie.