# Website to RAG Chunks: Vector-DB Ready Markdown (`torix33/website-to-rag-chunks`) Actor

Crawl a website, docs portal or blog and get LLM-ready chunks for RAG: clean Markdown split by headings to your token budget, with overlap, heading path, chunk IDs, content hashes and page metadata. Drop straight into Pinecone, Qdrant, Weaviate, pgvector or LangChain. Pay per page.

- **URL**: https://apify.com/torix33/website-to-rag-chunks.md
- **Developed by:** [Torix](https://apify.com/torix33) (community)
- **Categories:** AI, Developer tools, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 page scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to RAG Chunks — Vector-DB Ready Markdown

Point it at a website, documentation portal, help center or blog and get **LLM-ready chunks** for retrieval-augmented generation:

- main content only (Mozilla Readability) converted to **clean Markdown**, GFM tables and code blocks kept;
- **split by headings** and packed to your **token budget**, with **overlap** inside a section;
- each chunk carries its **heading path** (`Install > Linux`), optional **context prefix** (`Page title > Section`), **stable chunk ID**, **content hash**, token count, and page metadata (title, description, language, dates, URL, crawl depth).

One dataset row = one chunk → embed and upsert into **Pinecone, Qdrant, Weaviate, Chroma, pgvector, Elasticsearch, LangChain or LlamaIndex** with no extra processing.

### How to run it in 3 steps

1. **Add your input.** In **URLs** paste documentation or site URLs to split into chunks — e.g. `https://crawlee.dev/js/docs/quick-start`. The form is already filled with a working example, so you can skip this step for a first try.
2. **Press Start.** The prefilled run returns 5 pages in about 3 s and costs about **$0.015**. Set **Max results (cost cap)** to cap the cost of bigger runs; you only pay for pages actually delivered.
3. **Get the data.** Open the **Output** tab and export JSON, CSV or Excel — or run it from the API, Make, Zapier, n8n or an AI agent (see below).

### Features

- **Retrieval-quality chunking**, not naive fixed-size splits: headings define boundaries, code fences are never cut mid-block, tiny fragments are merged, giant tracking links are dropped.
- **Change detection:** `contentHash` lets you re-embed only chunks that changed on scheduled re-crawls.
- **HTTP only, no browser:** plain HTTP crawler with an in-memory frontier — 36 docs pages → 383 chunks in ~5 s.
- **Pay per page, not per chunk:** $3 per 1,000 pages, however many chunks they produce. Thin pages (below `minWordCount`) are free.

### Use cases

- **Chat with your docs:** turn a documentation portal or help center into a support chatbot or internal AI assistant.
- **RAG knowledge bases:** feed product pages, blogs and policies into Pinecone, Qdrant, pgvector or Weaviate.
- **Fresh indexes on a schedule:** re-crawl weekly and re-embed only chunks whose `contentHash` changed.
- **LLM fine-tuning & evaluation sets:** clean, heading-aware Markdown with token counts.

### Input

```json
{
  "startUrls": ["https://crawlee.dev/js/docs/quick-start"],
  "includeUrlPattern": "/js/docs/",
  "maxDepth": 3,
  "maxPagesPerSite": 500,
  "useSitemap": true,
  "maxTokens": 500,
  "overlapTokens": 50,
  "splitOnHeadings": true,
  "prependContext": true
}
```

### Output (one row per chunk)

#### Real output (default input)

First 2 of 45 records from a run with the prefilled input (3.8 s on a test machine; long values shortened here):

```json
[
  {
    "chunkId": "9ce446e941a8bbb4-0",
    "url": "https://crawlee.dev/js/docs/quick-start",
    "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
    "headingPath": [],
    "chunkIndex": 0,
    "chunkCount": 7,
    "text": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.\n\nWith this short tutorial you can start scraping with Crawlee in a min…",
    "tokens": 168,
    "contentHash": "e89d78cb6306e956",
    "description": "With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.",
    "language": "en",
    "publishedAt": null,
    "modifiedAt": null,
    "pageWordCount": 755,
    "depth": 0,
    "scrapedAt": "2026-10-15T12:16:30.483Z"
  },
  {
    "chunkId": "9ce446e941a8bbb4-1",
    "url": "https://crawlee.dev/js/docs/quick-start",
    "title": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast.",
    "headingPath": [
      "CheerioCrawler"
    ],
    "chunkIndex": 1,
    "chunkCount": 7,
    "text": "Quick Start | Crawlee for JavaScript · Build reliable crawlers. Fast. > CheerioCrawler\n\nThis is a plain HTTP crawler. It parses HTML using t…",
    "tokens": 78,
    "contentHash": "5a737f329555070e",
    "description": "With this short tutorial you can start scraping with Crawlee in a minute or two. To learn more, read the Introduction.",
    "language": "en",
    "publishedAt": null,
    "modifiedAt": null,
    "pageWordCount": 755,
    "depth": 0,
    "scrapedAt": "2026-10-15T12:16:30.483Z"
  }
]
```

````json
{
  "chunkId": "9ce446e941a8bbb4-5",
  "url": "https://crawlee.dev/js/docs/quick-start",
  "title": "Quick Start | Crawlee for JavaScript",
  "headingPath": ["PlaywrightCrawler"],
  "chunkIndex": 5,
  "chunkCount": 8,
  "text": "Quick Start | Crawlee for JavaScript > PlaywrightCrawler\n\n```\nimport { PlaywrightCrawler } from 'crawlee'; …",
  "tokens": 351,
  "contentHash": "4281dd1f1333b889",
  "description": "With this short tutorial you can start scraping with Crawlee in a minute or two.",
  "language": "en",
  "pageWordCount": 755,
  "depth": 0
}
````

### Pricing

- **$3.00 per 1,000 pages** ($0.003 per page).
- The prefilled input is billed for 5 pages → **$0.015**.
- Example: 1,000 pages → $3.00; 10,000 → $30.00. Set **Max results (cost cap)** to limit any run.

* Thin pages (below `minWordCount`) are free; you pay per page, not per chunk.
* Your Apify spending limit is respected exactly: the actor stops charging and finishes cleanly when it is reached. Apify platform usage (compute, proxy) is included in the price.

### Use with AI agents (MCP), LangChain, RAG & the API

This actor is a ready-made **tool for AI agents**: an LLM can call it, wait for the run and read clean JSON back.

**MCP server (Claude Desktop, Claude Code, Cursor, VS Code, any MCP client)** — add Apify's hosted MCP server with this actor enabled as a tool:

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=torix33/website-to-rag-chunks",
      "headers": { "Authorization": "Bearer <APIFY_TOKEN>" }
    }
  }
}
```

**LangChain / LangGraph** (`pip install langchain-apify`, set `APIFY_API_TOKEN`):

```python
from langchain_apify import ApifyActorsTool, ApifyWrapper
from langchain_core.documents import Document

## 1) give the actor to an agent as a tool
tool = ApifyActorsTool("torix33/website-to-rag-chunks")

## 2) or load results as Documents for a RAG index
loader = ApifyWrapper().call_actor(
    actor_id="torix33/website-to-rag-chunks",
    run_input={"startUrls": ["https://crawlee.dev/js/docs/quick-start"], "includeUrlPattern": "/js/docs/", "maxDepth": 3, "maxPagesPerSite": 500, "useSitemap": True, "maxTokens": 500, "overlapTokens": 50, "splitOnHeadings": True, "prependContext": True},
    dataset_mapping_function=lambda item: Document(page_content=item["text"], metadata={"url": item.get("url")}),
)
docs = loader.load()
```

**REST API — one synchronous call, dataset items in the response:**

```bash
curl -X POST "https://api.apify.com/v2/acts/torix33~website-to-rag-chunks/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": ["https://crawlee.dev/js/docs/quick-start"], "includeUrlPattern": "/js/docs/", "maxDepth": 3, "maxPagesPerSite": 500, "useSitemap": true, "maxTokens": 500, "overlapTokens": 50, "splitOnHeadings": true, "prependContext": true}'
```

Also works with the Apify Python/JS clients, **Make, Zapier, n8n**, Google Sheets, webhooks and schedules. Export as JSON, CSV, Excel, XML or RSS.

**RAG tip:** Rows are already chunks: embed `text`, upsert with `chunkId` as the vector ID, and on scheduled re-crawls re-embed only rows whose `contentHash` changed.

### Tips

- Token counts are estimates (≈4 characters per token) — leave ~10% headroom for your embedding model's limit.
- JavaScript-only sites (content rendered in the browser) may yield little text; this actor is HTTP-only for speed and price.

### Limitations

- HTTP only: single-page apps that render content in the browser may produce few or no chunks.
- Login-protected pages are not supported; non-HTML files are skipped.
- Chunk sizes use an estimate of ~4 characters per token; your embedding model's tokenizer will count somewhat differently.

### FAQ

**Which vector databases does it work with?**
Any: Pinecone, Qdrant, Weaviate, Chroma, pgvector, Elasticsearch, or LangChain/LlamaIndex loaders — one row is one chunk.

**Do I pay per chunk?**
No. You pay per page ($0.003), however many chunks it produces.

**How do I keep my index fresh?**
Schedule the actor and compare `contentHash` per `chunkId`; re-embed only changed chunks.

# Actor input Schema

## `startUrls` (type: `array`):

Start pages of the websites / docs to turn into chunks. One per line.

## `crawl` (type: `boolean`):

Follow links on the same site from each start URL (breadth-first).

## `maxDepth` (type: `integer`):

0 = start pages only, 1 = pages linked from them, etc.

## `maxPagesPerSite` (type: `integer`):

Cap per website when crawling.

## `useSitemap` (type: `boolean`):

Add the site's sitemap URLs to the crawl (fast way to cover a whole blog or docs site).

## `includeUrlPattern` (type: `string`):

Only crawl URLs matching this regular expression, e.g. <code>/blog/</code>.

## `excludeUrlPattern` (type: `string`):

Skip URLs matching this regular expression, e.g. <code>/tag/|/page/\d+</code>.

## `extractionMode` (type: `string`):

<b>Article</b>: main content only via Mozilla Readability (best for blogs, news, docs). <b>Full page</b>: whole body minus navigation, header, footer and sidebars.

## `maxTokens` (type: `integer`):

Chunk size budget (≈4 characters per token). 300–800 suits most embedding models.

## `overlapTokens` (type: `integer`):

Repeat the tail of the previous chunk (same section) at the start of the next one.

## `splitOnHeadings` (type: `boolean`):

Keeps each chunk inside one section (better retrieval). Off = pack sections together up to the token budget.

## `prependContext` (type: `boolean`):

Start each chunk's text with "Page title > Section > Subsection" so the embedding knows the context.

## `minWordCount` (type: `integer`):

Skip (and don't charge for) pages with fewer words, e.g. tag or listing pages.

## `maxResults` (type: `integer`):

Stop after this many pages in total. <code>0</code> = no limit.

## `maxConcurrency` (type: `integer`):

Pages fetched in parallel.

## `requestTimeoutSecs` (type: `integer`):

Give up on a page after this long.

## `proxyConfiguration` (type: `object`):

Not needed for most sites. Enable Apify Proxy for sites that block cloud servers.

## Actor input object example

```json
{
  "startUrls": [
    "https://crawlee.dev/js/docs/quick-start"
  ],
  "crawl": true,
  "maxDepth": 2,
  "maxPagesPerSite": 5,
  "useSitemap": false,
  "extractionMode": "article",
  "maxTokens": 500,
  "overlapTokens": 50,
  "splitOnHeadings": true,
  "prependContext": true,
  "minWordCount": 50,
  "maxResults": 0,
  "maxConcurrency": 20,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `chunks` (type: `string`):

No description

## `runStats` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://crawlee.dev/js/docs/quick-start"
    ],
    "maxPagesPerSite": 5
};

// Run the Actor and wait for it to finish
const run = await client.actor("torix33/website-to-rag-chunks").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://crawlee.dev/js/docs/quick-start"],
    "maxPagesPerSite": 5,
}

# Run the Actor and wait for it to finish
run = client.actor("torix33/website-to-rag-chunks").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://crawlee.dev/js/docs/quick-start"
  ],
  "maxPagesPerSite": 5
}' |
apify call torix33/website-to-rag-chunks --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,torix33/website-to-rag-chunks"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/qbciMQ8r3iLEc97qH/builds/EX5KP0BKnd2taGoDh/openapi.json