# Technical Docs To Rag Markdown Chunker (`arezdev/technical-docs-to-rag-markdown-chunker`) Actor

Convert public documentation into clean Markdown and RAG chunks with source URLs, section context and content hashes. Discover pages through sitemaps and llms.txt, preserve code and tables, and export JSON for embedding and vector database pipelines.

- **URL**: https://apify.com/arezdev/technical-docs-to-rag-markdown-chunker.md
- **Developed by:** [Arezdev](https://apify.com/arezdev) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.20 / 1,000 rag chunk createds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**Turn technical documentation into clean Markdown chunks your RAG pipeline can retrieve with confidence.**

Extract documentation from Mintlify, Docusaurus, GitBook, VitePress, Nextra, ReadTheDocs, and other sites serving documentation as HTML or Markdown. Keep headings, lists, tables, links, and complete code blocks while removing navigation, sidebars, scripts, and page furniture. Each chunk includes its source, section context, language labels, estimated size, and a stable SHA-256 content hash.

Best for preparing documentation for retrieval and change detection. Start with one page and discovery disabled, then expand the path scope. Output is Markdown and chunk metadata; embeddings are created by your own pipeline, and token sizes are estimates.

### Ready-to-run example

The Input form includes public example values. A fresh `apify run` uses the input schema defaults. To explicitly select the small sample, including when input was saved by an earlier local run, use:

```sh
apify run --input-file .actor/input.example.json
```

You can also paste [.actor/input.example.json](.actor/input.example.json) into the JSON input editor in Apify Console. Replace the example sources with your own targets and increase the limits when you are ready.

### How to use

1. Add one or more public documentation URLs.
2. Leave discovery off to process exactly those pages, or enable it to discover pages within each URL's origin and path prefix.
3. Choose the maximum pages and chunk size. Run the Actor and export the dataset as JSON, CSV, or Excel.
4. Embed `markdownChunk` in your preferred embedding service and store the vector with its source and section metadata.

The prefilled sample uses `https://docs.apify.com/actors`, `maxPages: 2`, and the optional `maxItems: 3` to save at most three rows quickly. Remove the optional output-row cap to extract all chunks from a page.

```json
{
  "docsUrls": ["https://docs.apify.com/actors"],
  "crawlViaSitemap": false,
  "maxPages": 2,
  "maxItems": 3,
  "maxTokensPerChunk": 500,
  "overlapTokens": 50,
  "timeoutSecs": 15
}
```

Discovery checks `/llms-full.txt`, then `/llms.txt`, then `/sitemap.xml`. Full Markdown documents with explicit `Source:`, `Source URL:`, or `URL:` markers can be chunked without refetching their HTML. Other full-file formats fall back to Markdown links and sitemap discovery. Sitemap indexes are followed within the original origin, with at most 32 sitemap files per prefix. Only URLs matching the supplied path on a path boundary are selected: `/docs` includes `/docs/guide`, and excludes `/docs-other`. Failed discovery falls back to the original page. Duplicate pages are processed once per run.

### What data can you extract?

| Field | Meaning |
| --- | --- |
| `sourceUrl` | Final public page URL after safe redirects, or the full-text source marker |
| `status` | `SUCCESS` for a chunk, `ERROR` for a failed page |
| `pageTitle` | Main heading, document title, or path fallback |
| `sectionBreadcrumb` | Heading hierarchy, such as `Guide > Setup > Windows` |
| `chunkIndex` | Zero-based position within the page |
| `totalChunksInPage` | Total chunks generated before output and billing limits |
| `estimatedTokens` | UTF-8 byte count divided by four, rounded up |
| `programmingLanguages` | Distinct fenced-code language labels present in the chunk |
| `hasCodeBlock` | Whether the chunk contains a fenced code block |
| `sha256Hash` | SHA-256 of the exact stored `markdownChunk` |
| `markdownChunk` | Markdown with prepended `Section:` context |
| `scrapedAt` | UTC ISO 8601 timestamp |
| `errorMessage` | Safe domain and failure code; empty for successful chunks |

Use `sha256Hash` to avoid re-embedding unchanged chunks. It excludes the timestamp and includes the section prefix. A changed chunking configuration can change chunk boundaries and hashes. `totalChunksInPage` describes the complete page even if a run cap saves only the first few chunks.

### Input settings

| Setting | Default | Allowed values |
| --- | --- | --- |
| `docsUrls` | Required | 1-2000 public HTTP/HTTPS URLs |
| `crawlViaSitemap` | `false` | Boolean |
| `maxPages` | `2` | 1-2000 unique pages attempted |
| `maxTokensPerChunk` | `500` | 100-4000 estimated tokens |
| `overlapTokens` | `50` | 0-500, strictly below chunk size |
| `timeoutSecs` | `15` | 3-60 seconds per logical request |
| `proxyConfiguration` | Optional | Apify proxy editor configuration |
| `maxConcurrency` | `25` | 1-100; adjusted to assigned memory |
| `maxItems` | Unrestricted; sample `3` | Optional 1-1,000,000 saved rows |

Chunks follow heading boundaries. Long prose is divided within the token estimate, and trailing prose overlaps the next chunk when space permits. Complete code blocks stay together, including their opening and closing fences. Large tables repeat the header across chunks and keep each row intact. An indivisible code block or table row can exceed the target size. Long breadcrumbs are shortened only in the prepended text so prose chunks still fit; the full section hierarchy remains in `sectionBreadcrumb`.

The token count is a lightweight size estimate, not a model-specific tokenizer. Allow additional headroom when your embedding provider enforces a strict token limit. Tables use the first row as the Markdown header if no explicit HTML header is present.

Requests go directly to the site first. HTTP 403, 429, or 503 can trigger Apify datacenter fallback, trying SHADER and then the default datacenter group. Your configured groups, including residential groups, are used only when you explicitly enable Apify Proxy. Custom proxy URLs are used exclusively. Set `useApifyProxy: false` to stay direct. Discovery of the two optional `llms` files stays direct to avoid proxy spending on missing files.

### Output example

Every row is flat. Language labels are a simple string array. A successful chunk has the following shape; the hash is computed from its exact Markdown at runtime.

```json
{
  "sourceUrl": "https://docs.apify.com/actors",
  "status": "SUCCESS",
  "pageTitle": "Actors",
  "sectionBreadcrumb": "Actors",
  "chunkIndex": 0,
  "totalChunksInPage": 7,
  "estimatedTokens": 7,
  "programmingLanguages": [],
  "hasCodeBlock": false,
  "sha256Hash": "1ec5f5c8238a17d977b59cd8fb089b5bd107a1346dce10cb02303c6ad22b60ce",
  "markdownChunk": "Section: Actors\n\n# Actors",
  "scrapedAt": "2026-10-20T12:00:00.000Z",
  "errorMessage": ""
}
```

An unreachable page produces one `ERROR` row with empty Markdown, empty hash, zero chunk counters, and a safe message such as `[docs.example.org] HTTP_ERROR (HTTP 403)`. Failed discovery probes are logged and do not create chargeable rows. Partial pages recovered after a size limit or eligible mid-stream disconnect produce normal successful chunks; their truncation is reported in run warnings and the final status.

### Use the results in Clay, n8n, Make, or your API

- In n8n or Make, run this Actor, retrieve its dataset, filter `status = SUCCESS`, then send `markdownChunk` to your embedding provider and upsert the vector with its metadata.
- In Clay, import the dataset through an HTTP integration or CSV export and retain source URLs for citations.
- In Pinecone, Qdrant, or pgvector, store the embedding alongside `sourceUrl`, `sectionBreadcrumb`, `sha256Hash`, and `chunkIndex`. The Actor produces text and metadata; your pipeline creates the embeddings and performs database writes.
- For updates, compare stored hashes, re-embed changed chunks, and reconcile the previous page's chunk IDs after the complete page has been received. Do not delete old chunks based on a run stopped by an item or budget cap.

The Apify dataset items endpoint is available in the run's Output tab. JSON retains Markdown and language arrays most directly. CSV exports escape formula-like strings to make spreadsheet imports safer.

### Pricing and limits

The custom pay-per-event event is **`page-chunked`**. One successfully saved chunk is one event. A page can produce several chunks. **`ERROR` rows cost $0 in custom event fees** and are saved without the event name. Platform compute, storage, data transfer, and proxy charges can still apply according to your plan.

For a concrete calculation, at a configured price of **$0.001 per successful chunk**, 1,000 successful chunks cost **$1.00 in event fees**, and 1,000 error rows cost **$0.00 in event fees**. This is an illustrative rate; the live Actor pricing shown in Apify Console is authoritative. Configure this event and its price in Console before monetizing the Actor; repository metadata does not publish Store pricing. See [Apify pay-per-event documentation](https://docs.apify.com/sdk/js/docs/concepts/pay-per-event).

Hosted runs for non-paying Apify users stop after **25 saved rows**, including errors. Local runs are unrestricted by the free-plan gate. Paid runs remain subject to your page limit, optional output-row cap, and maximum event charge budget. The writer reserves the remaining event budget before accepting successful rows and serializes all dataset writes.

The Actor defaults to 256 MB and a one-hour run timeout. It uses HTTP requests without a browser. Page and Markdown responses are capped at 4 MB after decompression; JSON at 2 MB; XML at 15 MB. A valid HTML prefix of at least 16 KiB can be recovered after a mid-stream timeout or reset. External cancellation does not salvage unfinished work. Completed buffered rows are flushed on Apify abort, SIGTERM, and SIGINT.

Concurrency is bounded by `maxConcurrency`, one worker per 64 MB of assigned memory, and available V8 heap headroom, giving up to four page workers at 256 MB. HTML depth is limited to 128 and retained elements to 60,000. Sitemaps reject DTDs and external entities. Dataset batches flush every 100 rows or five seconds, and earlier when buffered text reaches 2 MB.

### FAQ and troubleshooting

**Does this work with every documentation site?**

It works with publicly accessible server-rendered HTML and Markdown. Client-only content that requires JavaScript, a login, or a CAPTCHA is unavailable through direct HTTP extraction. Try the site's Markdown URLs or enable discovery if it publishes `llms.txt`.

**Why did discovery process only the supplied page?**

The site may not publish llms files or a standard sitemap, may block those endpoints, or may have no URLs within your selected path. Use a broader documentation prefix or supply the exact pages you need. Cross-origin discovery and out-of-prefix page redirects are rejected.

**Why is a code chunk larger than the target?**

The Actor preserves the entire fenced code block, avoiding broken examples. Increase your embedding context size or handle oversized code separately in your ingestion pipeline. A single wide table row is also kept intact.

**What causes an ERROR row?**

Examples include HTTP 403/404/429, DNS failures, invalid certificates, empty documents, parser safety limits, and request timeouts before enough readable HTML was received. Certificates remain verified. Error messages include a public domain and safe code without raw exception text or internal paths.

**Can I crawl a private network or a custom port?**

Only public HTTP/HTTPS targets on default ports are accepted. Every DNS answer and redirect is validated. Local, private, reserved, link-local, and metadata-service addresses are blocked. DNS pinning preserves the original HTTP authority and TLS hostname, including when using a proxy tunnel.

**Can I run with LIMITED_PERMISSIONS?**

Yes. The Actor uses its input, default dataset, charging manager, and explicitly configured proxy services. It does not request the current user's profile. Paying status comes from the run environment, and local runs do not use the hosted free cap.

**How do I run and validate the repository?**

Use Node.js 22 or newer:

```sh
npm ci --include=dev --ignore-scripts
npm run build
npm test
npm run smoke
```

For a local Actor run, save your input as `storage/key_value_stores/default/INPUT.json` and run `npm start`. No environment file is required. `npm run smoke` makes one public direct request and reports extraction size, timings, code presence, and memory. Tests use `node:test`, including a real local proxy-tunnel test with an injected public target resolver; production target validation remains enabled.

```sh
docker build -t technical-docs-to-rag-markdown-chunker .
docker run --rm -v "$PWD/storage:/app/storage" technical-docs-to-rag-markdown-chunker
```

The multi-stage Node.js 22 Alpine image builds and tests the repository, removes development dependencies, and runs under the non-root `node` user. Provide a writable storage mount for local Docker runs.

# Actor input Schema

## `docsUrls` (type: `array`):

Public HTTP or HTTPS pages. With discovery enabled, each URL defines an origin and path prefix. Subpaths match on path boundaries. Up to 2000 URLs.

## `crawlViaSitemap` (type: `boolean`):

Check llms-full.txt, llms.txt, then sitemap.xml. Follow matching prefix URLs and nested sitemap indexes; otherwise process the supplied URL.

## `maxPages` (type: `integer`):

Maximum unique documentation pages attempted across all input URLs. A page can yield multiple billable chunks.

## `maxItems` (type: `integer`):

Optional additional cap on saved chunks and error rows. Omit for unlimited rows within the page, free-plan, and event-budget limits. The prefilled sample saves at most three rows.

## `maxTokensPerChunk` (type: `integer`):

Estimated tokens including section context. Uses UTF-8 bytes divided by four, rounded up. Fenced code blocks and indivisible table rows are kept whole and may exceed this target.

## `overlapTokens` (type: `integer`):

Maximum trailing prose context repeated in the next chunk when space permits. Must be smaller than maxTokensPerChunk. Code blocks and table rows are not duplicated.

## `timeoutSecs` (type: `integer`):

Total deadline for one logical request, including DNS, redirect validation, protocol retries, and proxy attempts.

## `maxConcurrency` (type: `integer`):

Upper bound for parallel page requests. Automatically reduced to one worker per 64 MB of assigned memory to bound parser memory use; four workers at 256 MB.

## `proxyConfiguration` (type: `object`):

Optional. Direct connections first, then datacenter fallback for 403, 429, or 503. Configured groups are used only when explicitly enabled. Custom proxy URLs are forced; useApifyProxy: false disables automatic proxies.

## Actor input object example

```json
{
  "docsUrls": [
    "https://docs.apify.com/actors"
  ],
  "crawlViaSitemap": false,
  "maxPages": 2,
  "maxItems": 3,
  "maxTokensPerChunk": 500,
  "overlapTokens": 50,
  "timeoutSecs": 15,
  "maxConcurrency": 25
}
```

# Actor output Schema

## `chunks` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "docsUrls": [
        "https://docs.apify.com/actors"
    ],
    "maxPages": 2,
    "maxItems": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("arezdev/technical-docs-to-rag-markdown-chunker").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "docsUrls": ["https://docs.apify.com/actors"],
    "maxPages": 2,
    "maxItems": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("arezdev/technical-docs-to-rag-markdown-chunker").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "docsUrls": [
    "https://docs.apify.com/actors"
  ],
  "maxPages": 2,
  "maxItems": 3
}' |
apify call arezdev/technical-docs-to-rag-markdown-chunker --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,arezdev/technical-docs-to-rag-markdown-chunker"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3iuQy0rcr2wWVGJLj/builds/ymF7p38CLTnWTbLug/openapi.json