# Docs → RAG Corpus Builder (LLM-ready Markdown + llms.txt) (`yasaslive/docs-rag-builder`) Actor

Crawl any documentation site into an embeddings-ready corpus: clean markdown chunks with heading-path metadata, generated llms.txt, corpus.jsonl, and optional OpenAI embeddings.

- **URL**: https://apify.com/yasaslive/docs-rag-builder.md
- **Developed by:** [Eonix Pvt Ltd](https://apify.com/yasaslive) (community)
- **Categories:** AI, Developer tools, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.00005 / actor start

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Docs → RAG Corpus Builder (LLM-ready Markdown + llms.txt)

Turn any documentation site into an **embeddings-ready RAG corpus** in one run: clean markdown
chunks with code blocks preserved, a heading path on every chunk, version awareness, a generated
[`llms.txt`](https://llmstxt.org), and optional OpenAI embeddings. Built to be called by AI agents
as easily as by humans.

**Who it's for:** teams building RAG chatbots and coding assistants over product docs, agent
builders who need fresh grounded context, and platform engineers keeping vector stores in sync
with living documentation.

### What it does

1. **Crawls** the docs site (Crawlee CheerioCrawler, plain HTTP — fast and cheap). Sitemap
   discovery runs first (`robots.txt` sitemaps, then `/sitemap.xml`); link crawling is the
   fallback, constrained by your include/exclude globs. Same-origin only, robots.txt respected,
   `Retry-After` honored on 429s.
2. **Extracts** the actual article content. Platform-specific selector heuristics for
   **Docusaurus, MkDocs Material, GitBook, ReadMe and Sphinx** (auto-detected), with a
   Mozilla Readability fallback for everything else. Nav, sidebars, footers, breadcrumbs,
   version badges and heading-anchor noise are stripped.
3. **Converts to markdown** with real fidelity: `<pre><code class="language-x">` becomes a
   ` ```x ` fenced block (line structure preserved even for prism/shiki line-span highlighting),
   tables become GFM tables.
4. **Chunks** on heading boundaries first, then packs to your token budget (cl100k estimate via
   js-tiktoken) with sentence-level overlap. **Fenced code blocks and tables are never split** —
   an oversized one becomes a single chunk flagged `oversized: true`.
5. **Dedupes** near-identical pages (print views, mirrors) via simhash and logs every skip.
6. **Ships artifacts**: one dataset record per chunk, plus `llms.txt`, `corpus.jsonl` and
   `MANIFEST.json` in the key-value store — and `embeddings.jsonl` when you provide an OpenAI key.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `startUrls` | array | *required* | Entry points. The crawl stays on these URLs' origins. |
| `includeGlobs` | string\[] | `[]` | Only crawl URLs matching at least one minimatch glob (matched against full URL and pathname, e.g. `/api/**`). |
| `excludeGlobs` | string\[] | `[]` | Skip URLs matching any glob. |
| `maxPages` | integer | `500` | Hard page cap (max 5,000). |
| `versionMode` | enum | `all` | `all` tags every page with its detected version; `latest-only` drops versioned URLs (unless pinned). |
| `versionPattern` | string | `/(v\d+(\.\d+)*\|\d+\.\d+)/` | Regex that detects the version segment in URL paths → `detectedVersion`. |
| `pinnedVersion` | string | — | In `latest-only` mode, keep URLs of exactly this version (e.g. `v2`). |
| `maxChunkTokens` | integer | `512` | Token budget per chunk. |
| `chunkOverlapTokens` | integer | `64` | Sentence-level overlap between consecutive chunks of a section. |
| `openaiApiKey` | secret string | — | Enables embeddings. Sent only to api.openai.com, never logged or stored. |
| `embeddingModel` | string | `text-embedding-3-small` | OpenAI embedding model. |
| `proxyConfiguration` | proxy | — | Apify Proxy or custom proxies. |

### Output

#### Dataset — one record per chunk

Real sample from a run against `https://docs.apify.com/` (`maxPages: 30`):

````json
{
  "id": "https://docs.apify.com/academy/advanced-web-scraping/crawling/crawling-sitemaps#7",
  "url": "https://docs.apify.com/academy/advanced-web-scraping/crawling/crawling-sitemaps",
  "title": "Crawling sitemaps",
  "headingPath": ["Crawling sitemaps", "Using Crawlee"],
  "detectedVersion": null,
  "chunkIndex": 7,
  "markdown": "## Using Crawlee\n\nFortunately, you don't have to worry about any of the above steps if you use [Crawlee](https://crawlee.dev), a scraping framework, which has rich traversing and parsing support for sitemap. It can traverse nested sitemaps, download, and parse compressed sitemaps, and extract URLs from them. You can get all the URLs in a few lines of code:\n\n```js\nimport { RobotsFile } from 'crawlee';\n\nconst robots = await RobotsFile.find('https://www.mysite.com');\n\nconst allWebsiteUrls = await robots.parseUrlsFromSitemaps();\n```",
  "tokenEstimate": 131,
  "oversized": false
}
````

The final dataset record is a run summary (`recordType: "summary"`) with counts and artifact URLs.
Records are validated against the actor's dataset schema on insert, and every run's **Output** tab
links the chunks and all four artifacts directly (actor output schema).

#### Key-value store artifacts

| Key | Contents |
|---|---|
| `llms.txt` | Site title, summary and a curated per-section link list in the [llmstxt.org](https://llmstxt.org) format. |
| `corpus.jsonl` | Every chunk, one JSON object per line — pipe straight into your ingestion job. |
| `MANIFEST.json` | Pages crawled, chunk/token totals, versions detected, full settings echo. |
| `embeddings.jsonl` | `{ "id", "vector" }` per chunk (only when `openaiApiKey` is set); `id` joins `corpus.jsonl`. |

`llms.txt` from the same real run:

```text
## Apify Documentation

> Documentation corpus generated from docs.apify.com — clean markdown chunks for RAG and LLM context.

### Overview

- [Apify Documentation](https://docs.apify.com/)

### Academy

- [Apify Academy](https://docs.apify.com/academy): Learn everything about web scraping and automation with our free courses that will turn you into an expert scraper developer.
- [Actor description & SEO description](https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description): Learn about Actor description and meta description. Where to set them and best practices for both content and length.
…
```

`MANIFEST.json` (excerpt, same run): 30 pages crawled in `sitemap` mode, 0 failed, 310 chunks,
1 oversized, 49,546 tokens total.

### Pricing (pay-per-event)

| Event | Charged when | Suggested price |
|---|---|---|
| `page-processed` | Per page successfully extracted, **after** its chunks are stored | **$1.50 / 1,000 pages** |
| `corpus-built` | Once per completed run, **after** `MANIFEST.json` is written | **$0.10 / run** |
| `embeddings-1k` | Per started 1,000 chunks embedded, **after** `embeddings.jsonl` is stored | **$0.20 / 1,000 chunks** |

Failed pages and failed runs are never charged. A default 500-page run costs about **$0.85**
(+ embeddings if enabled; OpenAI usage is billed to your own key).

### Use cases

1. **Docs chatbot in an afternoon.** Point the actor at `docs.yourproduct.com`, load
   `corpus.jsonl` + `embeddings.jsonl` into pgvector/Pinecone/Qdrant, and your support bot cites
   the exact section (`headingPath`) it answered from.
2. **Keep a coding agent current.** Schedule weekly runs over a fast-moving framework's docs with
   `versionMode: "latest-only"` so your agent stops recommending deprecated v1 APIs. Diff
   `MANIFEST.json` between runs to re-embed only what changed.
3. **Publish llms.txt for your own product.** Generate a spec-compliant `llms.txt` from your real
   docs structure and serve it at `/llms.txt` so ChatGPT, Claude and Perplexity ground themselves
   on your documentation instead of hallucinating it.

### Calling this actor from AI agents

The pipeline is exposed as one exported function (`runCorpusBuild(input)` in `src/pipeline.ts`),
so the actor is a single tool call for an agent:

- **Apify MCP server** — connect your agent (Claude Desktop, or any MCP client) to
  `https://mcp.apify.com` and add this actor. It appears as a callable tool: the agent passes the
  input JSON, waits for the run, then reads the dataset and `corpus.jsonl`/`llms.txt` from the
  key-value store. See [Apify MCP docs](https://docs.apify.com/platform/integrations/mcp).
- **Task runs** — create an Apify Task with your site preset (globs, budget, version pin) and let
  agents trigger it via `POST /v2/actor-tasks/:taskId/runs?token=…` — no input assembly needed.
- **API** — `POST /v2/acts/<you>~docs-rag-builder/runs` with the input as JSON body; poll the run,
  then `GET /v2/key-value-stores/:storeId/records/corpus.jsonl`.

### Running locally

```bash
npm install
npm run build     # tsc — zero errors
npm test          # node --test — extraction fixtures for all 5 doc platforms
apify run         # uses storage/key_value_stores/default/INPUT.json (docs.apify.com, 30 pages)
```

### FAQ

**Does it execute JavaScript?** No — it's HTTP + Cheerio by design, which makes it ~10× cheaper
and faster. JS-only SPAs without server-rendered content are out of scope for v0.1.

**How accurate are token counts?** Real cl100k\_base encoding via js-tiktoken — the same tokenizer
family OpenAI embedding models use, not a character heuristic.

**What does `oversized: true` mean?** The chunk exceeds `maxChunkTokens` because it is a single
indivisible code block or table. Splitting code mid-fence would poison retrieval, so it ships
whole and flagged; for embedding, oversized text is truncated to the model's input limit.

**Can it crawl multiple sites in one run?** Yes — add several `startUrls`; each URL's origin is
allowed and everything stays within that set.

**Why did a page I expected not appear?** Check `MANIFEST.json`: it records duplicate skips,
empty-content skips and failures per run, and the actor log names every skipped URL.

**Do I need an OpenAI key?** No — chunks, `llms.txt` and `corpus.jsonl` are produced without it.
The key only enables `embeddings.jsonl`.

### Changelog

#### 0.1.0

- Initial release: sitemap-first crawling, 5-platform extraction heuristics + Readability
  fallback, fence-safe heading-aware chunking, simhash dedupe, llms.txt/corpus.jsonl/MANIFEST
  artifacts, optional OpenAI embeddings, pay-per-event billing.

# Actor input Schema

## `startUrls` (type: `array`):

Documentation site entry points. The actor stays on the same origin(s) as these URLs. Sitemap discovery (/robots.txt + /sitemap.xml) is attempted first; link crawling is the fallback.

## `includeGlobs` (type: `array`):

Optional minimatch globs. When set, only URLs matching at least one glob are crawled. Globs are tested against both the full URL (e.g. `https://docs.example.com/api/**`) and the pathname (e.g. `/api/**`).

## `excludeGlobs` (type: `array`):

Optional minimatch globs. URLs matching any glob are skipped. Tested against both the full URL and the pathname.

## `maxPages` (type: `integer`):

Hard cap on the number of pages crawled.

## `versionMode` (type: `string`):

`all` crawls every matched URL and tags each with its detected docs version. `latest-only` drops URLs whose path matches the version pattern (unless they match the pinned version), keeping only unversioned = latest pages.

## `versionPattern` (type: `string`):

Regex applied to the URL path to detect a docs version segment. The first match (first capture group if present) becomes `detectedVersion` on every chunk.

## `pinnedVersion` (type: `string`):

In `latest-only` mode, versioned URLs whose detected version equals this value (e.g. `v2` or `3.1`) are kept instead of dropped.

## `maxChunkTokens` (type: `integer`):

Token budget per chunk (cl100k\_base estimate via js-tiktoken). Sections are split on heading boundaries first, then packed into chunks up to this size. Fenced code blocks are never split — oversized ones become single chunks flagged `oversized: true`.

## `chunkOverlapTokens` (type: `integer`):

Approximate token overlap carried between consecutive chunks split from the same section (trailing sentences only, never code).

## `openaiApiKey` (type: `string`):

When set, every chunk is embedded (batches of 100) and `embeddings.jsonl` is written to the key-value store. The key is only sent to api.openai.com and never stored or logged.

## `embeddingModel` (type: `string`):

OpenAI embedding model used when an API key is provided.

## `proxyConfiguration` (type: `object`):

Proxy to use for crawling. Apify Proxy (automatic) is recommended for reliability on protected sites.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "includeGlobs": [],
  "excludeGlobs": [],
  "maxPages": 500,
  "versionMode": "all",
  "versionPattern": "/(v\\d+(\\.\\d+)*|\\d+\\.\\d+)/",
  "maxChunkTokens": 512,
  "chunkOverlapTokens": 64,
  "embeddingModel": "text-embedding-3-small",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `chunks` (type: `string`):

One record per chunk: id, url, title, headingPath, detectedVersion, chunkIndex, markdown, tokenEstimate, oversized. The last record is the run summary.

## `llmsTxt` (type: `string`):

Generated llms.txt index of the crawled documentation (llmstxt.org format).

## `corpusJsonl` (type: `string`):

The full chunk corpus as JSONL — one chunk object per line, ready for ingestion.

## `manifest` (type: `string`):

Run manifest: pages crawled, chunk and token totals, versions detected, and the settings used.

## `embeddingsJsonl` (type: `string`):

One { id, vector } object per chunk; id joins corpus.jsonl and the dataset. Present only when an OpenAI API key was provided.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/"
        }
    ],
    "includeGlobs": [],
    "excludeGlobs": [],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("yasaslive/docs-rag-builder").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/" }],
    "includeGlobs": [],
    "excludeGlobs": [],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("yasaslive/docs-rag-builder").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/"
    }
  ],
  "includeGlobs": [],
  "excludeGlobs": [],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call yasaslive/docs-rag-builder --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,yasaslive/docs-rag-builder"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0rS1KNARdJjEW5Gdm/builds/Lvx3AUojv8SwYKsyO/openapi.json