# Markdown RAG Chunker (`codepoetry/markdown-rag-chunker`) Actor Chunk any document for RAG — PDF, HTML, Word, Excel, PPTX, Markdown and more. Header-aware splits with token counts and stable IDs. - **URL**: https://apify.com/codepoetry/markdown-rag-chunker.md - **Developed by:** [CodePoetry](https://apify.com/codepoetry) (community) - **Categories:** Open source, AI, Agents - **Stats:** 8 total users, 0 monthly users, 87.5% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing $20.00/month + usage To use this Actor, you pay a monthly rental fee to the developer. The rent is subtracted from your prepaid usage every month after the free trial period.You also pay for the Apify platform usage, which gets cheaper the higher Apify subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#rental-actors ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Markdown RAG Chunker **Markdown RAG Chunker** turns PDFs, web pages, Word, Excel, PowerPoint, and Markdown files into clean, **header-aware chunks** ready for embeddings and vector databases. Built for AI engineers shipping RAG pipelines who want predictable splits, stable IDs, and token estimates without writing chunking code. [Run the Actor in Apify Console](https://console.apify.com/actors/84h4GucI0gbo1SrN7?addFromActorId=84h4GucI0gbo1SrN7). ### What does Markdown RAG Chunker do? - **Multi-format input**: convert PDF, HTML, DOCX, XLSX, PPTX, CSV, JSON, XML, EPUB, and plain text to Markdown before chunking - **Header-aware splitting**: split by `#` through `######` so each chunk keeps its parent section in metadata - **Token-aware sizing**: optional `max_chunk_chars` re-splits only oversized sections, so retrieval stays focused - **Deterministic chunk IDs**: every chunk has a stable `chunk_id` for idempotent upserts and deduplication in [Pinecone](https://www.pinecone.io/), [Qdrant](https://qdrant.tech/), [Weaviate](https://weaviate.io/), [pgvector](https://github.com/pgvector/pgvector), and others - **Token count estimates**: `token_count` per chunk for embedding budget planning - **Run telemetry**: `metrics` payload with `input_file_type`, `chunk_count`, and `elapsed_ms` for monitoring - **Two input modes**: paste Markdown directly, or pass an HTTPS URL or `kvs://KEY` for any supported file - **Pay only when you use it**: pay-per-event pricing — no monthly rental Behind the scenes the Actor uses Microsoft's [markitdown](https://github.com/microsoft/markitdown) for format conversion and [LangChain's text splitters](https://python.langchain.com/docs/how_to/markdown_header_metadata_splitter/) (`MarkdownHeaderTextSplitter` plus `RecursiveCharacterTextSplitter`) for chunking. You get production-grade defaults without managing the dependencies yourself. ### Supported input formats | Category | Formats | | --- | --- | | Markdown and text | `.md`, `.txt` | | Web | HTML pages, single web page URLs | | PDF | `.pdf` | | Office | `.docx`, `.xlsx`, `.xls`, `.pptx` | | Data | `.csv`, `.json`, `.xml` | | Books | `.epub` | Provide any of these as an HTTPS URL or as an [Apify Key-Value Store](https://docs.apify.com/platform/storage/key-value-store) record (`kvs://KEY`). MIME type is detected automatically from the response and reported back in `metrics.input_file_type`. ### How to use Markdown RAG Chunker 1. Open the Actor and choose `input_mode`. Pick `text` to paste Markdown directly, or `file` to load any supported document from a URL or `kvs://KEY`. 2. Set `headers_to_split_on`. Most RAG pipelines work best with `["#", "##", "###"]` — that gives you section-level chunks while preserving page-level context in metadata. 3. Optionally set `max_chunk_chars` (for example, `1800` characters or about 450 tokens) to cap oversized sections. Only chunks above the cap are re-split, so well-sized sections are preserved untouched. 4. Run the Actor and read results from the dataset. Pipe each item's `content` into your embedding model and store the `chunk_id` and `metadata` alongside the vector. The full input form is documented under the [Input](https://apify.com/codepoetry/markdown-rag-chunker/input-schema) tab. Run output schema and field types live under the [Output](https://apify.com/codepoetry/markdown-rag-chunker/output-schema) tab. ### Output format ```json { "chunks": [ { "content": "Install the SDK with pip install ...", "metadata": { "Header 1": "Guide", "Header 2": "Install" }, "chunk_id": "f8b6be2adf7f6dbf", "char_count": 124, "token_count": 31 } ], "metrics": { "input_mode": "file", "input_file_type": "application/pdf", "input_chars": 4281, "chunk_count": 12, "elapsed_ms": 184 } } ```` | Field | Description | | --- | --- | | `content` | Chunk text to send to your embedding model | | `metadata` | Header hierarchy (`Header 1`, `Header 2`, ...) for context-aware retrieval | | `chunk_id` | Stable 16-char ID for idempotent upserts and deduplication | | `char_count` | Character length of `content` | | `token_count` | Approximate token count (~1 token per 4 characters) | | `metrics.input_file_type` | Detected source MIME type (for example, `application/pdf`) | | `metrics.chunk_count` | Total chunks produced in the run | | `metrics.elapsed_ms` | End-to-end processing time | ### How much does document chunking cost? Markdown RAG Chunker uses [pay-per-event pricing](https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event): you pay a small fixed amount only when a file is processed, with no monthly rental. Direct text input on the free tier is ideal for trying the chunker before wiring it into a pipeline. Detailed unit prices are listed under the [Pricing](https://apify.com/codepoetry/markdown-rag-chunker/pricing) tab. ### Use cases - **RAG over documentation**: split long product docs and changelogs into retrievable sections - **Knowledge bases**: ingest internal PDFs and Word docs into a vector store with stable IDs - **Customer support search**: chunk help center articles for semantic search - **AI agents**: feed large reference documents to agents in budget-friendly slices - **Crawler post-processing**: chain after [Website Content Crawler](https://apify.com/apify/website-content-crawler) to convert crawled pages into RAG-ready chunks ### FAQ #### How do I chunk a PDF for a vector database? Set `input_mode` to `file`, paste the PDF URL into `markdown_file`, and run the Actor. The PDF is converted to Markdown via [markitdown](https://github.com/microsoft/markitdown), split by header hierarchy, and each chunk gets a deterministic `chunk_id` you can use as the upsert key in [Pinecone](https://www.pinecone.io/), [Qdrant](https://qdrant.tech/), [Weaviate](https://weaviate.io/), or [pgvector](https://github.com/pgvector/pgvector). #### What is the difference between header-aware chunking and fixed-size chunking? Fixed-size chunking cuts text every N characters or tokens, often slicing through paragraphs and losing structural context. Header-aware chunking splits on Markdown headings, so each chunk maps to a logical section and the parent header chain is kept in `metadata`. This gives retrieval models real context — a chunk about "Authentication" still knows it lives under the "API Reference" section. #### Can I chain Markdown RAG Chunker with a web crawler? Yes. A common pipeline is **[Website Content Crawler](https://apify.com/apify/website-content-crawler) → Markdown RAG Chunker → embeddings → vector DB**. The crawler produces clean Markdown for each page, and this Actor splits that Markdown into RAG-ready chunks with stable IDs. #### Does it work with LangChain or LlamaIndex? Yes. The chunking is built on [LangChain text splitters](https://python.langchain.com/docs/how_to/markdown_header_metadata_splitter/), and the output (`content` + `metadata`) maps cleanly onto a LangChain [`Document`](https://python.langchain.com/api_reference/core/documents/langchain_core.documents.base.Document.html) or LlamaIndex [`Node`](https://docs.llamaindex.ai/en/stable/module_guides/loading/documents_and_nodes/). You can also use Apify's [LangChain](https://python.langchain.com/docs/integrations/providers/apify/) and [LlamaIndex](https://docs.apify.com/platform/integrations/llama) integrations directly inside both frameworks. #### How accurate is `token_count`? `token_count` is a fast estimate (~1 token per 4 characters) intended for budget planning and guardrails. For exact token counts, run your model's tokenizer over `content` after retrieval. #### Where do I find API examples? Use the [API](https://apify.com/codepoetry/markdown-rag-chunker/api), [Python](https://apify.com/codepoetry/markdown-rag-chunker/api/python), [JavaScript](https://apify.com/codepoetry/markdown-rag-chunker/api/javascript), [CLI](https://apify.com/codepoetry/markdown-rag-chunker/api/cli), [OpenAPI](https://apify.com/codepoetry/markdown-rag-chunker/api/openapi), and [MCP](https://apify.com/codepoetry/markdown-rag-chunker/api/mcp) tabs on this Actor's page — they include ready-to-paste code with the correct Actor ID and input shape for every supported client. # Actor input Schema ## `input_mode` (type: `string`): Use text to paste Markdown directly. Use file to load and convert a supported file format from an HTTPS URL or kvs://KEY. ## `markdown_text` (type: `string`): Paste Markdown content here when input\_mode is text. ## `markdown_file` (type: `string`): When input\_mode is file, provide an HTTPS URL to Markdown, HTML, PDF, Word, Excel, PowerPoint, CSV, JSON, XML, EPUB, and more. You can also provide an Apify Key-Value Store record like kvs://MARKDOWN\_INPUT. ## `headers_to_split_on` (type: `array`): Markdown header levels to split on, such as \['#', '##']. Content under headers not listed is merged into parent sections. ## `strip_headers` (type: `boolean`): When true, header text is moved to metadata only. Use true for embedding pipelines and false for human-readable chunks. ## `max_chunk_chars` (type: `integer`): Optional cap for chunk length in characters. Only oversized chunks are re-split. A common starting point is 1800 chars (about 450 tokens). ## `max_input_chars` (type: `integer`): Hard limit for input size in characters. Runs fail fast when exceeded to protect cost and runtime. ## `max_chunks` (type: `integer`): Safety cap on total output chunks. Increase only for very large documents. ## Actor input object example ```json { "input_mode": "file", "markdown_text": "# Header 1\nHello, World!\n## Header 2\nGoodbye, World!", "markdown_file": "https://en.wikipedia.org/wiki/Giant_panda", "headers_to_split_on": [ "#", "##", "###", "####", "#####", "######" ], "strip_headers": true, "max_input_chars": 500000, "max_chunks": 5000 } ``` # Actor output Schema ## `results` (type: `string`): Dataset items containing a chunks array (content, metadata, chunk\_id, char\_count, token\_count) and run metrics (input\_mode, input\_chars, chunk\_count, elapsed\_ms). # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "markdown_file": "https://en.wikipedia.org/wiki/Giant_panda" }; // Run the Actor and wait for it to finish const run = await client.actor("codepoetry/markdown-rag-chunker").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "markdown_file": "https://en.wikipedia.org/wiki/Giant_panda" } # Run the Actor and wait for it to finish run = client.actor("codepoetry/markdown-rag-chunker").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "markdown_file": "https://en.wikipedia.org/wiki/Giant_panda" }' | apify call codepoetry/markdown-rag-chunker --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=codepoetry/markdown-rag-chunker", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Markdown RAG Chunker", "description": "Chunk any document for RAG — PDF, HTML, Word, Excel, PPTX, Markdown and more. Header-aware splits with token counts and stable IDs.", "version": "1.0", "x-build-id": "il9UxYJEdyFw1pBPW" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/codepoetry~markdown-rag-chunker/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-codepoetry-markdown-rag-chunker", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/codepoetry~markdown-rag-chunker/runs": { "post": { "operationId": "runs-sync-codepoetry-markdown-rag-chunker", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/codepoetry~markdown-rag-chunker/run-sync": { "post": { "operationId": "run-sync-codepoetry-markdown-rag-chunker", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "input_mode": { "title": "Input Mode", "enum": [ "text", "file" ], "type": "string", "description": "Use text to paste Markdown directly. Use file to load and convert a supported file format from an HTTPS URL or kvs://KEY.", "default": "file" }, "markdown_text": { "title": "Markdown Text", "type": "string", "description": "Paste Markdown content here when input_mode is text.", "default": "# Header 1\nHello, World!\n## Header 2\nGoodbye, World!" }, "markdown_file": { "title": "Markdown File Source", "type": "string", "description": "When input_mode is file, provide an HTTPS URL to Markdown, HTML, PDF, Word, Excel, PowerPoint, CSV, JSON, XML, EPUB, and more. You can also provide an Apify Key-Value Store record like kvs://MARKDOWN_INPUT.", "default": "https://en.wikipedia.org/wiki/Giant_panda" }, "headers_to_split_on": { "title": "Headers to Split On", "type": "array", "description": "Markdown header levels to split on, such as ['#', '##']. Content under headers not listed is merged into parent sections.", "default": [ "#", "##", "###", "####", "#####", "######" ], "items": { "type": "string" } }, "strip_headers": { "title": "Strip Headers", "type": "boolean", "description": "When true, header text is moved to metadata only. Use true for embedding pipelines and false for human-readable chunks.", "default": true }, "max_chunk_chars": { "title": "Max Chunk Chars", "minimum": 1, "maximum": 100000, "type": "integer", "description": "Optional cap for chunk length in characters. Only oversized chunks are re-split. A common starting point is 1800 chars (about 450 tokens)." }, "max_input_chars": { "title": "Max Input Chars", "minimum": 1, "maximum": 5000000, "type": "integer", "description": "Hard limit for input size in characters. Runs fail fast when exceeded to protect cost and runtime.", "default": 500000 }, "max_chunks": { "title": "Max Chunks", "minimum": 1, "maximum": 50000, "type": "integer", "description": "Safety cap on total output chunks. Increase only for very large documents.", "default": 5000 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```