# Docs-to-RAG Pipeline Builder (`optirefine/docs-to-rag-pipeline`) Actor

Crawl any documentation site into clean, chunked, embedding-ready markdown. Free embeddings via your own Cloudflare Workers AI account, or on-device. No paid API keys.

- **URL**: https://apify.com/optirefine/docs-to-rag-pipeline.md
- **Developed by:** [OptiRefine](https://apify.com/optirefine) (community)
- **Categories:** Automation, SEO tools, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 data extractions

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Docs-to-RAG Pipeline Builder

Point it at a documentation site. Get back clean, chunked, embedding-ready markdown.

Not a generic scraper with a markdown option — a pipeline built around the one thing that
makes a documentation corpus useful for retrieval: **every chunk knows where it came
from**, keeps its code fences intact, and carries the heading path that makes it
answerable on its own.

**No paid API keys anywhere.** Tokenisation, extraction and markdown conversion run
locally. Embeddings are optional and free either way: on-device, or through a Cloudflare
Worker you deploy on the free Workers AI tier.

***

### What it does

```
                     ┌─────────────┐
   start URLs ──────►│  sitemap.xml│──┐
                     └─────────────┘  │
                                      ▼
           ┌──────────────────────────────────────┐
           │  Pass 1  ·  CheerioCrawler (HTTP)    │
           └──────────────┬───────────────────────┘
                          │  content quality gate
                 ┌────────┴────────┐
        passes   │                 │  too thin / SPA shell
                 ▼                 ▼
                 │       ┌──────────────────────────────┐
                 │       │ Pass 2 · PlaywrightCrawler   │
                 │       └──────────────┬───────────────┘
                 └────────┬─────────────┘
                          ▼
   ┌───────────────────────────────────────────────────────┐
   │  Extract   explicit selector → generator profile       │
   │            → Defuddle → density heuristic → body       │
   │            (all scored; best wins)                     │
   ├───────────────────────────────────────────────────────┤
   │  Markdown  Turndown + GFM, code-fence languages,       │
   │            tables, admonitions → GitHub alerts         │
   ├───────────────────────────────────────────────────────┤
   │  Chunk     heading tree · breadcrumbs · never splits   │
   │            a code block or a table · real BPE tokens   │
   ├───────────────────────────────────────────────────────┤
   │  Dedup     exact hash + heading-scoped SimHash         │
   ├───────────────────────────────────────────────────────┤
   │  Embed     none · on-device · your Cloudflare Worker   │
   └───────────────────────────────────────────────────────┘
                          ▼
      dataset (one item per chunk) · corpus.jsonl · corpus.md
                  · llms.txt · llms-full.txt · run-report.json
```

### Quick start

#### On Apify

```json
{
  "startUrls": [{ "url": "https://docs.example.com" }],
  "maxCrawlPages": 500
}
```

`startUrls` is the entire minimum input; everything else has a documented default.
`maxCrawlPages` is shown here only because it is the one worth raising: it defaults
to `50`, which keeps a first run quick, and `500` takes most of a documentation site.

A run is also bounded by the clock, not only by the page count. The crawl stops
in time to write its output before the run's own timeout, so a site too big or
too slow for the time allowed returns a smaller corpus and says so in the summary,
rather than being killed mid-crawl with an empty dataset. Set `maxCrawlTimeSecs`
to choose that budget yourself.

#### Locally, no account needed

```bash
npm install
npm run build

npx docs-to-rag https://docs.example.com --out ./corpus
```

```
corpus/
├── chunks.jsonl        one JSON object per chunk, ready for a vector store
├── corpus.jsonl        the same, as a key-value-store artifact
├── corpus.md           every page as one readable markdown document
├── llms.txt            an index of the site, llmstxt.org style
├── llms-full.txt       the whole corpus as a single document
└── run-report.json     counts, extractors used, failures
```

Run `npx docs-to-rag --help` for the full flag list. Any field in
`INPUT_SCHEMA.json` works as `--kebab-case-name`.

***

### What a chunk looks like

```jsonc
{
  "id": "9f2c1a77b0e34d15",
  "url": "https://docs.example.com/api/authentication",
  "anchorUrl": "https://docs.example.com/api/authentication#oauth-2-0",
  "pageTitle": "Authentication",
  "breadcrumb": ["Authentication", "Authorization flows"],
  "heading": "OAuth 2.0",
  "headingLevel": 3,
  "chunkIndex": 2,
  "chunkCount": 5,

  // `text` is what you embed. It opens with the heading path, which is the
  // difference between a retrievable chunk and an orphaned paragraph.
  "text": "Authentication > Authorization flows > OAuth 2.0\n\n### OAuth 2.0\n\nSet `redirect_uri` to...",
  "rawText": "### OAuth 2.0\n\nSet `redirect_uri` to...",

  "tokenCount": 612,          // real cl100k_base tokens, counted offline
  "charCount": 2481,
  "contentHash": "1b0f...",
  "generator": "mkdocs-material",
  "extractor": "profile:mkdocs-material",
  "crawledAt": "2026-08-24T10:14:22.104Z",

  // present only when embeddings are enabled
  "embedding": [0.0123, -0.0456, ...],
  "embeddingModel": "@cf/baai/bge-m3",
  "embeddingDims": 1024
}
```

`anchorUrl` is a real deep link. Cite it and the reader lands on the exact section.

***

### The parts that took the work

#### The crawler escalates instead of guessing

Pass 1 fetches everything over plain HTTP with `CheerioCrawler`. Each page's extracted
content is scored, and pages that fail the gate — a JavaScript shell, an empty article, a
generator known to render client-side — are re-fetched in a real browser by
`PlaywrightCrawler`. You pay for Chromium only on the pages that need it.

This is deliberately **not** `AdaptivePlaywrightCrawler`: that class is still experimental,
hands the request handler a reduced context, and decides by sampling. Here the decision is
a pure function of the extracted content, so it is unit-tested and identical on every run.

#### Extraction degrades instead of failing silently

Four strategies run and are scored against each other:

1. your `mainContentSelector`, if you set one — it wins outright
2. a **documentation generator profile** — MkDocs (Material and default), Docusaurus,
   Sphinx (Read the Docs, alabaster, Furo, PyData), Starlight, VitePress, VuePress,
   Nextra, Antora, GitBook, Mintlify, Docsify, Docsy, Just the Docs, Redoc, ReadMe.io,
   plus best-effort profiles for Hugo Book and Slate
3. **Defuddle**, the extractor behind Obsidian Web Clipper
4. a link-density heuristic, then the bare `<body>`

Every profile except Hugo Book and Slate was verified against a live page from that
generator, saved as a committed fixture and asserted in the test suite; those two are
labelled unverified in `profiles.ts` because no reachable demo site existed when the table
was built. The scoring is what makes that safe: if a selector is wrong or goes stale, the
profile simply loses to Defuddle instead of quietly emitting an empty page.

#### Chunking that respects structure

- Splits on the heading tree, not a sliding window.
- **Never** splits inside a fenced code block or a table. An oversized code block is split
  at line boundaries and each piece is re-fenced with the original language; an oversized
  table is split at row boundaries with the header repeated.
- Overlap is never taken from inside a code fence — a half-open fence poisons everything
  downstream.
- Sections below `minChunkSize` are merged, because a fifteen-token orphan matches on
  stray terms and answers nothing.
- Sizes are real BPE token counts from `gpt-tokenizer`, offline, not characters ÷ 4.

#### Deduplication that knows what a heading means

Exact hashing catches byte-identical pages. Near-duplicate detection uses a 64-bit SimHash
with banded lookup, so it stays linear across tens of thousands of chunks.

Near-duplicate matching is scoped to the chunk's heading. Without that, "Install on Linux"
and "Install on Windows" with identical bodies differ by one word in a hundred, land inside
any useful similarity threshold, and one of them is silently lost. The default threshold is
conservative for the same reason: dropping real content is worse than keeping a duplicate.

#### Markdown fidelity

Code fences recover their language from `class="language-*"`, `lang-*`, `highlight-*`,
`data-language`, and wrapper divs — MkDocs, Sphinx, Docusaurus and Starlight each hide it
somewhere different. Line-number tables unwrap to the code cell instead of becoming a
markdown table. Admonitions become GitHub alerts. KaTeX keeps its TeX and drops the
duplicate visual markup. `snake_case` is not escaped to `snake\_case`, because CommonMark
does not treat intraword underscores as emphasis.

***

### Embeddings, free, two ways

| | `local` | `cloudflare-worker` |
| --- | --- | --- |
| Runs | on this machine / actor | your Cloudflare account |
| Network | model weights downloaded once, then none | one HTTPS call per batch |
| Cost | none | free tier, 10,000 Neurons/day |
| Default model | `Xenova/all-MiniLM-L6-v2` (384d) | `@cf/baai/bge-m3` (1024d) |
| Best for | offline, small corpora | better vectors, bigger crawls |

Deploy the Worker in about three minutes — see **[`worker/README.md`](worker/README.md)**
for the dashboard walkthrough and the `wrangler` one-liner. Then:

```json
{
  "startUrls": [{ "url": "https://docs.example.com" }],
  "embeddingProvider": "cloudflare-worker",
  "workerUrl": "https://docs-to-rag-worker.you.workers.dev",
  "workerApiKey": "<the API_TOKEN you set on the Worker>"
}
```

At bge-m3's rate the free daily allocation covers roughly **9 million tokens of
embeddings** — a 500-page site chunked at 800 tokens costs about 0.5% of one day.

If embedding fails, the crawl does not. Failed batches are counted and reported; the
markdown corpus is written either way. You lose vectors, never content.

#### Optional: LLM contextualisation

`contextualizeChunks: true` prepends one model-written sentence to each chunk situating it
in its document, before embedding — the "contextual retrieval" pattern. A chunk reading
"Set this to your callback URL" is nearly unretrievable; the same chunk led by "This
describes the redirect\_uri parameter of the OAuth authorisation endpoint" is not.

It costs one LLM call per chunk, so it is off by default and capped by
`maxContextualizedChunks` (500).

***

### Common recipes

**Only the docs, not the blog**

```json
{ "includeUrlGlobs": ["**/docs/**"], "excludeUrlGlobs": ["**/blog/**", "**/changelog/**"] }
```

**One version of versioned docs**

```json
{ "startUrls": [{ "url": "https://docs.example.com/en/stable/" }], "crawlScope": "same-path-prefix" }
```

**A fully client-rendered site** (Docsify, Redoc, an old React docs app)

```json
{ "renderingMode": "browser-only", "browserWaitForSelector": ".markdown-section" }
```

**Match your embedding model's window**

```json
{ "chunkSize": 512, "chunkOverlap": 64, "tokenizer": "o200k_base" }
```

**The site has an unusual layout**

```json
{ "mainContentSelector": "#article-body", "extraExcludeSelectors": [".version-banner", ".related-links"] }
```

### Full input reference

Every field, its type, default and description lives in
[`INPUT_SCHEMA.json`](INPUT_SCHEMA.json), which the Apify Console renders as a form. The
test suite asserts that file and the code agree on every default and every enum, so it
cannot drift.

Highlights:

| Field | Default | Notes |
| --- | --- | --- |
| `startUrls` | — | The only required field. |
| `crawlScope` | `same-hostname` | Also `same-domain`, `same-path-prefix`, `any`. |
| `maxCrawlPages` | `50` | Reported in the summary when it binds. Raise it for a whole site. |
| `maxCrawlTimeSecs` | `0` | Wall-clock ceiling on the crawl. `0` derives one from the run's timeout, keeping a reserve for embedding and output. |
| `renderingMode` | `auto` | HTTP first, browser where needed. |
| `escalateBelowWords` | `60` | Raise it on sites with many short reference pages: a genuinely short page gets a browser re-fetch that returns the same thin content. "Browser re-fetches" in the run summary tells you if that is happening. |
| `mainContentSelector` | — | Wins outright when set. |
| `chunkSize` / `chunkOverlap` | `800` / `100` | In `tokenizer` tokens. |
| `splitHeadingLevel` | `3` | Deeper headings stay inside their parent chunk. |
| `nearDuplicateThreshold` | `0.97` | Conservative on purpose. |
| `embeddingProvider` | `none` | `local` or `cloudflare-worker`. |
| `outputMode` | `chunks` | Or `pages`, or `both`. |

### Output

**Dataset** — one item per chunk (or per page, or both). Every record carries
`itemType: "chunk" | "page"`, so `outputMode: "both"` stays parseable: a loader filters on
that field rather than guessing from which keys are present. The Apify Console gets two
prebuilt views: *Chunks* for reading, *Provenance* for auditing where content came from.

**Key-value store** — `corpus.jsonl`, `corpus.md`, `llms.txt`, `llms-full.txt`,
`run-report.json`, and `OUTPUT` with the run summary.

The run ends with a readable summary and, when something looks off, specific advice:
the page budget bound the crawl, the generic `body` extractor won too often, most links
were rejected as out of scope, too many chunks were deduplicated. A crawl that quietly
returns 40 pages instead of 400 is the most common way a tool like this fails, and the
report is the defence.

### Development

```bash
npm install
npm run verify           # typecheck + full test suite
npm test                 # vitest
npm run build            # tsc -> dist/

## See which extractor wins on each committed fixture, and why
npx tsx scripts/extract-report.ts test/fixtures
npx tsx scripts/extract-report.ts test/fixtures --markdown
```

The content pipeline is pure functions over `{ url, html }`, so extraction, markdown
conversion and chunking are all tested offline against real saved documentation pages.
Crawler tests run against a synthetic site on localhost. Nothing in `npm test` touches the
network.

Architecture notes and the design rationale are in
[`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md).

### Deploying to Apify

```bash
npm install -g apify-cli
apify login
apify push
```

`.actor/actor.json`, `.actor/Dockerfile` and `INPUT_SCHEMA.json` are ready. The Docker base
image pins both the Node major and the exact Playwright version so the bundled Chromium
always matches the dependency.

### Licence

MIT.

# Actor input Schema

## `startUrls` (type: `array`):

Documentation root URLs. The crawler follows links from here, and also reads the site's sitemap unless you turn that off.

## `crawlScope` (type: `string`):

Which links are considered in-scope. 'Same hostname' is the safe default. 'Same domain' also follows subdomains. 'Same path prefix' restricts to the start URL's directory, which is the cleanest way to grab only /docs.

## `includeUrlGlobs` (type: `array`):

If set, only URLs matching one of these patterns are crawled. Supports \*, \*\*, ? and {a,b}. Example: **/docs/**

## `excludeUrlGlobs` (type: `array`):

URLs matching any of these are never crawled. Exclusions always beat inclusions. Example: **/blog/**

## `maxCrawlPages` (type: `integer`):

Hard page budget for the run. The default keeps a first run quick; raise it to take a whole site. Reached limits are reported in the run summary.

## `maxCrawlTimeSecs` (type: `integer`):

Stop crawling after this many seconds and write out whatever has been collected, so a slow site produces a partial corpus instead of nothing. Leave at 0 to derive the budget from the run's own timeout, keeping a reserve for embedding and output.

## `maxCrawlDepth` (type: `integer`):

How many links deep to follow from the start URLs. Sitemap-discovered pages count as depth 1.

## `respectRobotsTxt` (type: `boolean`):

Obey the site's robots.txt rules. Leave this on unless you own the site.

## `useSitemaps` (type: `boolean`):

Read sitemap.xml (discovered via robots.txt, then by convention) before crawling. This is what finds pages that no sidebar links to.

## `maxConcurrency` (type: `integer`):

Parallel HTTP requests. The browser pass automatically runs at a third of this.

## `requestTimeoutSecs` (type: `integer`):

Per-request timeout.

## `maxRequestRetries` (type: `integer`):

How many times to retry a failed request before giving up on it.

## `ignoreQueryParams` (type: `array`):

Extra query parameters to strip when deciding whether two URLs are the same page. Common tracking parameters (utm\_\*, gclid, fbclid, ...) are always stripped.

## `ignoreAllQueryParams` (type: `boolean`):

Treat ?a=1 and ?a=2 as the same page. Useful for docs sites that append view state to every link.

## `renderingMode` (type: `string`):

Auto fetches over plain HTTP first and automatically re-fetches in a real browser any page that comes back empty or too thin. Use browser-only for fully client-rendered docs (Docsify, Redoc), http-only for maximum speed on static sites.

## `escalateBelowWords` (type: `integer`):

In auto mode, a page whose extracted content has fewer words than this is re-fetched in a browser.

## `browserWaitMs` (type: `integer`):

Additional settle time after page load, for docs that hydrate late.

## `browserWaitForSelector` (type: `string`):

CSS selector to wait for before extracting, in browser mode. Leave empty unless a site needs it.

## `mainContentSelector` (type: `string`):

CSS selector for the article container. When set and non-empty it wins outright - use it when you know the site. Leave empty to let generator detection and content scoring choose.

## `excludeSelectors` (type: `array`):

CSS selectors stripped from every page. Replaces the built-in list entirely - use 'Extra exclude selectors' to add to it instead. Admonitions and callouts are protected from removal even if a selector would match them.

## `extraExcludeSelectors` (type: `array`):

Additional selectors to strip, on top of the defaults. This is usually what you want.

## `useGeneratorProfiles` (type: `boolean`):

Detect MkDocs, Docusaurus, Sphinx, Starlight, VitePress, Nextra, Antora, GitBook, Mintlify, Docsify, Docsy, Just the Docs and others, and use their known content containers. A stale profile loses to the generic extractors rather than emptying the page.

## `extractionFallback` (type: `string`):

What to fall back to when no profile matches. Defuddle is a readability-style extractor tuned for articles; heuristic scores containers by text and link density.

## `keepLinks` (type: `boolean`):

Keep inline links in the markdown. Turn off for the smallest possible corpus.

## `keepImages` (type: `boolean`):

Keep image references (with alt text) in the markdown.

## `keepTables` (type: `boolean`):

Convert HTML tables to GitHub-flavoured markdown tables. Turn off to drop them entirely.

## `absoluteUrls` (type: `boolean`):

Rewrite every relative link and image to an absolute URL, so chunks stay usable away from the source page.

## `chunkStrategy` (type: `string`):

Heading + recursive splits at headings and then splits oversized sections down to the chunk size. Heading only splits at headings and lets sections exceed the limit. Page emits one chunk per page.

## `chunkSize` (type: `integer`):

Maximum size of a chunk, in the unit below. 800 tokens suits most embedding models.

## `chunkOverlap` (type: `integer`):

How much of the end of a chunk to repeat at the start of the next. Overlap is never taken from inside a code block.

## `minChunkSize` (type: `integer`):

Chunks smaller than this are merged into a neighbour. Prevents one-line orphan chunks that match on stray terms and answer nothing.

## `chunkUnit` (type: `string`):

Measure chunks in real BPE tokens (offline, no API) or in characters.

## `tokenizer` (type: `string`):

BPE encoding used for token counts. cl100k\_base matches most current embedding models; o200k\_base matches the newest OpenAI models.

## `splitHeadingLevel` (type: `integer`):

Start a new chunk at headings up to this depth. Deeper headings stay inside their parent chunk, which keeps a section about one topic together.

## `includeBreadcrumbInChunk` (type: `boolean`):

Prefix each chunk with its heading path (Page > Section > Subsection). This is the single biggest retrieval-quality win in the whole pipeline.

## `deduplicate` (type: `boolean`):

Drop duplicate and near-duplicate pages and chunks. Versioned and translated documentation repeats itself heavily.

## `nearDuplicateThreshold` (type: `number`):

Similarity above which two chunks count as the same, from 0.5 to 1. The default is conservative on purpose: near-duplicate detection cannot distinguish a versioned mirror from a distinct section that shares boilerplate, and dropping real content is worse than keeping a duplicate. Lower it to 0.92 to collapse versioned mirrors, or raise it to 1 to drop only byte-identical chunks.

## `embeddingProvider` (type: `string`):

Leave as None to output chunks only and embed them yourself.

## `embeddingModel` (type: `string`):

Defaults to Xenova/all-MiniLM-L6-v2 for local, @cf/baai/bge-m3 for the Worker.

## `workerUrl` (type: `string`):

Base URL of your deployed Worker, for example https://docs-to-rag-worker.you.workers.dev. The actor calls /embed and /llm under it.

## `workerApiKey` (type: `string`):

The API\_TOKEN secret you set on the Worker. Sent as a bearer token.

## `embeddingEndpoint` (type: `string`):

Overrides {workerUrl}/embed. Point this at any endpoint that accepts { model, texts } and returns { vectors }.

## `embeddingBatchSize` (type: `integer`):

Chunks per embedding request.

## `embeddingConcurrency` (type: `integer`):

Parallel embedding requests. Keep this low on a free tier.

## `contextualizeChunks` (type: `boolean`):

Requires a Worker URL. Measurably improves retrieval on chunks that use pronouns or bare parameter names.

## `llmModel` (type: `string`):

Workers AI text-generation model used for contextualisation.

## `llmEndpoint` (type: `string`):

Overrides {workerUrl}/llm.

## `maxContextualizedChunks` (type: `integer`):

Hard cap on LLM calls, so a large crawl cannot exhaust a free daily allocation.

## `outputMode` (type: `string`):

Chunks pushes one dataset item per chunk, ready for a vector store. Pages pushes one item per page with its full markdown. Both pushes both.

## `saveMarkdownArtifacts` (type: `boolean`):

Write corpus.jsonl and corpus.md to the key-value store alongside the dataset.

## `saveLlmsTxt` (type: `boolean`):

Write llms.txt (an index of the crawled pages) and llms-full.txt (the whole corpus as one document) to the key-value store.

## `includeMarkdownInPageItems` (type: `boolean`):

Include each page's full markdown in page-mode dataset items. Turn off for a lighter dataset.

## `proxyConfiguration` (type: `object`):

Optional. Documentation sites rarely need a proxy; use one if you are rate-limited.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "crawlScope": "same-hostname",
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [
    "**/blog/**",
    "**/changelog/**"
  ],
  "maxCrawlPages": 50,
  "maxCrawlTimeSecs": 0,
  "maxCrawlDepth": 20,
  "respectRobotsTxt": true,
  "useSitemaps": true,
  "maxConcurrency": 10,
  "requestTimeoutSecs": 60,
  "maxRequestRetries": 3,
  "ignoreQueryParams": [],
  "ignoreAllQueryParams": false,
  "renderingMode": "auto",
  "escalateBelowWords": 60,
  "browserWaitMs": 0,
  "excludeSelectors": [
    "nav",
    "header",
    "footer",
    "aside",
    "script",
    "style",
    "noscript",
    "iframe",
    "template",
    "form",
    "[role=\"navigation\"]",
    "[role=\"banner\"]",
    "[role=\"contentinfo\"]",
    "[role=\"search\"]",
    "[aria-hidden=\"true\"]",
    ".sr-only",
    ".visually-hidden",
    ".skip-link",
    ".skip-to-content",
    ".breadcrumb",
    ".breadcrumbs",
    ".pagination",
    ".edit-page",
    ".edit-this-page",
    ".feedback",
    ".cookie-banner",
    ".announcement",
    ".advertisement",
    ".carbon-ads",
    "#carbonads",
    ".headerlink",
    ".header-anchor",
    "a.hash-link",
    ".anchor-link",
    "a.anchorjs-link",
    ".copybtn",
    ".copy-button",
    "[data-copy-button]"
  ],
  "extraExcludeSelectors": [],
  "useGeneratorProfiles": true,
  "extractionFallback": "defuddle",
  "keepLinks": true,
  "keepImages": true,
  "keepTables": true,
  "absoluteUrls": true,
  "chunkStrategy": "heading-recursive",
  "chunkSize": 800,
  "chunkOverlap": 100,
  "minChunkSize": 120,
  "chunkUnit": "token",
  "tokenizer": "cl100k_base",
  "splitHeadingLevel": 3,
  "includeBreadcrumbInChunk": true,
  "deduplicate": true,
  "nearDuplicateThreshold": 0.97,
  "embeddingProvider": "none",
  "embeddingBatchSize": 32,
  "embeddingConcurrency": 2,
  "contextualizeChunks": false,
  "llmModel": "@cf/meta/llama-3.1-8b-instruct-fast",
  "maxContextualizedChunks": 500,
  "outputMode": "chunks",
  "saveMarkdownArtifacts": true,
  "saveLlmsTxt": true,
  "includeMarkdownInPageItems": true
}
```

# Actor output Schema

## `chunks` (type: `string`):

One record per retrieval chunk: text with its heading path, a deep link to the source section, token counts, provenance, and vectors when an embedding provider is configured. This is what you load into a vector store.

## `corpusJsonl` (type: `string`):

Every chunk as newline-delimited JSON, in one file rather than a paginated dataset. Convenient for a single-shot import.

## `corpusMarkdown` (type: `string`):

Every crawled page as one readable markdown document, in crawl order. Useful for eyeballing extraction quality before you embed anything.

## `llmsTxt` (type: `string`):

An llmstxt.org-style index of the site: a title, a summary, and grouped links to every page.

## `llmsFullTxt` (type: `string`):

The entire corpus as a single document, ready to paste into a long-context model.

## `runReport` (type: `string`):

Counts for the run: pages crawled and kept, browser escalations, duplicates removed, chunks and tokens produced, which extractor handled each page, and any failures.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy"
        }
    ],
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [
        "**/blog/**",
        "**/changelog/**"
    ],
    "maxCrawlPages": 50,
    "extraExcludeSelectors": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("optirefine/docs-to-rag-pipeline").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/academy" }],
    "includeUrlGlobs": [],
    "excludeUrlGlobs": [
        "**/blog/**",
        "**/changelog/**",
    ],
    "maxCrawlPages": 50,
    "extraExcludeSelectors": [],
}

# Run the Actor and wait for it to finish
run = client.actor("optirefine/docs-to-rag-pipeline").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "includeUrlGlobs": [],
  "excludeUrlGlobs": [
    "**/blog/**",
    "**/changelog/**"
  ],
  "maxCrawlPages": 50,
  "extraExcludeSelectors": []
}' |
apify call optirefine/docs-to-rag-pipeline --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,optirefine/docs-to-rag-pipeline"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5IEBWNBBe5RZPlSBd/builds/viC1OUJLmLbQK79iv/openapi.json