Technical Docs To Rag Markdown Chunker

Turn technical documentation into clean Markdown chunks your RAG pipeline can retrieve with confidence.

Extract documentation from Mintlify, Docusaurus, GitBook, VitePress, Nextra, ReadTheDocs, and other sites serving documentation as HTML or Markdown. Keep headings, lists, tables, links, and complete code blocks while removing navigation, sidebars, scripts, and page furniture. Each chunk includes its source, section context, language labels, estimated size, and a stable SHA-256 content hash.

Best for preparing documentation for retrieval and change detection. Start with one page and discovery disabled, then expand the path scope. Output is Markdown and chunk metadata; embeddings are created by your own pipeline, and token sizes are estimates.

Ready-to-run example

The Input form includes public example values. A fresh apify run uses the input schema defaults. To explicitly select the small sample, including when input was saved by an earlier local run, use:

apify run --input-file .actor/input.example.json

You can also paste .actor/input.example.json into the JSON input editor in Apify Console. Replace the example sources with your own targets and increase the limits when you are ready.

How to use

1. Add one or more public documentation URLs.
2. Leave discovery off to process exactly those pages, or enable it to discover pages within each URL's origin and path prefix.
3. Choose the maximum pages and chunk size. Run the Actor and export the dataset as JSON, CSV, or Excel.
4. Embed markdownChunk in your preferred embedding service and store the vector with its source and section metadata.

The prefilled sample uses https://docs.apify.com/actors, maxPages: 2, and the optional maxItems: 3 to save at most three rows quickly. Remove the optional output-row cap to extract all chunks from a page.

{
  "docsUrls": ["https://docs.apify.com/actors"],
  "crawlViaSitemap": false,
  "maxPages": 2,
  "maxItems": 3,
  "maxTokensPerChunk": 500,
  "overlapTokens": 50,
  "timeoutSecs": 15
}

Discovery checks /llms-full.txt, then /llms.txt, then /sitemap.xml. Full Markdown documents with explicit Source:, Source URL:, or URL: markers can be chunked without refetching their HTML. Other full-file formats fall back to Markdown links and sitemap discovery. Sitemap indexes are followed within the original origin, with at most 32 sitemap files per prefix. Only URLs matching the supplied path on a path boundary are selected: /docs includes /docs/guide, and excludes /docs-other. Failed discovery falls back to the original page. Duplicate pages are processed once per run.

What data can you extract?

Field	Meaning
sourceUrl	Final public page URL after safe redirects, or the full-text source marker
status	SUCCESS for a chunk, ERROR for a failed page
pageTitle	Main heading, document title, or path fallback
sectionBreadcrumb	Heading hierarchy, such as Guide > Setup > Windows
chunkIndex	Zero-based position within the page
totalChunksInPage	Total chunks generated before output and billing limits
estimatedTokens	UTF-8 byte count divided by four, rounded up
programmingLanguages	Distinct fenced-code language labels present in the chunk
hasCodeBlock	Whether the chunk contains a fenced code block
sha256Hash	SHA-256 of the exact stored markdownChunk
markdownChunk	Markdown with prepended Section: context
scrapedAt	UTC ISO 8601 timestamp
errorMessage	Safe domain and failure code; empty for successful chunks

Use sha256Hash to avoid re-embedding unchanged chunks. It excludes the timestamp and includes the section prefix. A changed chunking configuration can change chunk boundaries and hashes. totalChunksInPage describes the complete page even if a run cap saves only the first few chunks.

Input settings

Setting	Default	Allowed values
docsUrls	Required	1-2000 public HTTP/HTTPS URLs
crawlViaSitemap	false	Boolean
maxPages	2	1-2000 unique pages attempted
maxTokensPerChunk	500	100-4000 estimated tokens
overlapTokens	50	0-500, strictly below chunk size
timeoutSecs	15	3-60 seconds per logical request
proxyConfiguration	Optional	Apify proxy editor configuration
maxConcurrency	25	1-100; adjusted to assigned memory
maxItems	Unrestricted; sample 3	Optional 1-1,000,000 saved rows

Chunks follow heading boundaries. Long prose is divided within the token estimate, and trailing prose overlaps the next chunk when space permits. Complete code blocks stay together, including their opening and closing fences. Large tables repeat the header across chunks and keep each row intact. An indivisible code block or table row can exceed the target size. Long breadcrumbs are shortened only in the prepended text so prose chunks still fit; the full section hierarchy remains in sectionBreadcrumb.

The token count is a lightweight size estimate, not a model-specific tokenizer. Allow additional headroom when your embedding provider enforces a strict token limit. Tables use the first row as the Markdown header if no explicit HTML header is present.

Requests go directly to the site first. HTTP 403, 429, or 503 can trigger Apify datacenter fallback, trying SHADER and then the default datacenter group. Your configured groups, including residential groups, are used only when you explicitly enable Apify Proxy. Custom proxy URLs are used exclusively. Set useApifyProxy: false to stay direct. Discovery of the two optional llms files stays direct to avoid proxy spending on missing files.

Output example

Every row is flat. Language labels are a simple string array. A successful chunk has the following shape; the hash is computed from its exact Markdown at runtime.

{
  "sourceUrl": "https://docs.apify.com/actors",
  "status": "SUCCESS",
  "pageTitle": "Actors",
  "sectionBreadcrumb": "Actors",
  "chunkIndex": 0,
  "totalChunksInPage": 7,
  "estimatedTokens": 7,
  "programmingLanguages": [],
  "hasCodeBlock": false,
  "sha256Hash": "1ec5f5c8238a17d977b59cd8fb089b5bd107a1346dce10cb02303c6ad22b60ce",
  "markdownChunk": "Section: Actors\n\n# Actors",
  "scrapedAt": "2026-10-20T12:00:00.000Z",
  "errorMessage": ""
}

An unreachable page produces one ERROR row with empty Markdown, empty hash, zero chunk counters, and a safe message such as [docs.example.org] HTTP_ERROR (HTTP 403). Failed discovery probes are logged and do not create chargeable rows. Partial pages recovered after a size limit or eligible mid-stream disconnect produce normal successful chunks; their truncation is reported in run warnings and the final status.

Use the results in Clay, n8n, Make, or your API

• In n8n or Make, run this Actor, retrieve its dataset, filter status = SUCCESS, then send markdownChunk to your embedding provider and upsert the vector with its metadata.
• In Clay, import the dataset through an HTTP integration or CSV export and retain source URLs for citations.
• In Pinecone, Qdrant, or pgvector, store the embedding alongside sourceUrl, sectionBreadcrumb, sha256Hash, and chunkIndex. The Actor produces text and metadata; your pipeline creates the embeddings and performs database writes.
• For updates, compare stored hashes, re-embed changed chunks, and reconcile the previous page's chunk IDs after the complete page has been received. Do not delete old chunks based on a run stopped by an item or budget cap.

The Apify dataset items endpoint is available in the run's Output tab. JSON retains Markdown and language arrays most directly. CSV exports escape formula-like strings to make spreadsheet imports safer.

Pricing and limits

The custom pay-per-event event is page-chunked. One successfully saved chunk is one event. A page can produce several chunks. ERROR rows cost $0 in custom event fees and are saved without the event name. Platform compute, storage, data transfer, and proxy charges can still apply according to your plan.

For a concrete calculation, at a configured price of $0.001 per successful chunk, 1,000 successful chunks cost $1.00 in event fees, and 1,000 error rows cost $0.00 in event fees. This is an illustrative rate; the live Actor pricing shown in Apify Console is authoritative. Configure this event and its price in Console before monetizing the Actor; repository metadata does not publish Store pricing. See Apify pay-per-event documentation.

Hosted runs for non-paying Apify users stop after 25 saved rows, including errors. Local runs are unrestricted by the free-plan gate. Paid runs remain subject to your page limit, optional output-row cap, and maximum event charge budget. The writer reserves the remaining event budget before accepting successful rows and serializes all dataset writes.

The Actor defaults to 256 MB and a one-hour run timeout. It uses HTTP requests without a browser. Page and Markdown responses are capped at 4 MB after decompression; JSON at 2 MB; XML at 15 MB. A valid HTML prefix of at least 16 KiB can be recovered after a mid-stream timeout or reset. External cancellation does not salvage unfinished work. Completed buffered rows are flushed on Apify abort, SIGTERM, and SIGINT.

Concurrency is bounded by maxConcurrency, one worker per 64 MB of assigned memory, and available V8 heap headroom, giving up to four page workers at 256 MB. HTML depth is limited to 128 and retained elements to 60,000. Sitemaps reject DTDs and external entities. Dataset batches flush every 100 rows or five seconds, and earlier when buffered text reaches 2 MB.

FAQ and troubleshooting

Does this work with every documentation site?

It works with publicly accessible server-rendered HTML and Markdown. Client-only content that requires JavaScript, a login, or a CAPTCHA is unavailable through direct HTTP extraction. Try the site's Markdown URLs or enable discovery if it publishes llms.txt.

Why did discovery process only the supplied page?

The site may not publish llms files or a standard sitemap, may block those endpoints, or may have no URLs within your selected path. Use a broader documentation prefix or supply the exact pages you need. Cross-origin discovery and out-of-prefix page redirects are rejected.

Why is a code chunk larger than the target?

The Actor preserves the entire fenced code block, avoiding broken examples. Increase your embedding context size or handle oversized code separately in your ingestion pipeline. A single wide table row is also kept intact.

What causes an ERROR row?

Examples include HTTP 403/404/429, DNS failures, invalid certificates, empty documents, parser safety limits, and request timeouts before enough readable HTML was received. Certificates remain verified. Error messages include a public domain and safe code without raw exception text or internal paths.

Can I crawl a private network or a custom port?

Only public HTTP/HTTPS targets on default ports are accepted. Every DNS answer and redirect is validated. Local, private, reserved, link-local, and metadata-service addresses are blocked. DNS pinning preserves the original HTTP authority and TLS hostname, including when using a proxy tunnel.

Can I run with LIMITED_PERMISSIONS?

Yes. The Actor uses its input, default dataset, charging manager, and explicitly configured proxy services. It does not request the current user's profile. Paying status comes from the run environment, and local runs do not use the hosted free cap.

How do I run and validate the repository?

Use Node.js 22 or newer:

npm ci --include=dev --ignore-scripts
npm run build
npm test
npm run smoke

For a local Actor run, save your input as storage/key_value_stores/default/INPUT.json and run npm start. No environment file is required. npm run smoke makes one public direct request and reports extraction size, timings, code presence, and memory. Tests use node:test, including a real local proxy-tunnel test with an injected public target resolver; production target validation remains enabled.

docker build -t technical-docs-to-rag-markdown-chunker .
docker run --rm -v "$PWD/storage:/app/storage" technical-docs-to-rag-markdown-chunker

The multi-stage Node.js 22 Alpine image builds and tests the repository, removes development dependencies, and runs under the non-root node user. Provide a writable storage mount for local Docker runs.