# Website to RAG Chunks — RAG crawler, token-sized, cited (`webprobe-tools/website-to-rag-chunks`) Actor

Crawl a docs site, help center or blog and get embedding-ready chunks: main content as Markdown, split at headings to a token size you choose, each with its heading path, a citation URL to its section, a token count and a content hash. Plain HTTP. Built and maintained by an AI agent.

- **URL**: https://apify.com/webprobe-tools/website-to-rag-chunks.md
- **Developed by:** [WebProbe Tools](https://apify.com/webprobe-tools) (community)
- **Categories:** AI, Developer tools, Agents
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to RAG Chunks — RAG crawler, token-sized, cited

Crawl a documentation site, help center or blog and get chunks you can embed right away. Each page's main content becomes Markdown, which is split at its headings into chunks no larger than the token size you choose. Every chunk carries its heading path, a citation URL that points at its section, a token count, a stable id and content hashes. Navigation, footers and repeated boilerplate are left out.

Built and maintained by an AI agent.

### What it does

- **Extracts the main content only.** Menus, sidebars, cookie banners and footers are dropped. Headings, lists, tables and code blocks are kept as Markdown.
- **Chunks at headings, to a token budget.** A chunk never crosses into a parent's next section, and a code block is never split inside its fence. A very short section runs on into the next one, so a heading with one line under it doesn't become a chunk of its own. Token counts use OpenAI's `o200k_base` encoding.
- **Cites each chunk's section.** `citation` is the page URL plus the anchor of the heading the chunk starts under, for example `https://crawlee.dev/python/docs/quick-start#installation`. Your app can link the exact section an answer came from.
- **Tells same-named sections apart.** With `headingContext` on, each chunk starts with a line such as `[Quick start > Installation]`. A retriever can then separate the Installation section of one guide from another's.
- **Drops duplicates.** A page whose content matches one already chunked (a `/docs/next/` copy, or a URL with tracking parameters) is skipped and not charged. A chunk identical to one already output is dropped.
- **Supports incremental re-indexing.** `pageContentHash` changes only when a page's text changes, so a pipeline can skip re-embedding unchanged pages. `id` is stable across runs for the same page and text.
- **Finds pages links miss.** The crawl follows links breadth-first from your start URLs, then adds the in-scope URLs the site's sitemaps list. It respects robots.txt.
- **Fast and cheap:** plain HTTP, no browser, 512 MB. In a test run at 512 MB, 100 documentation pages took 84 seconds.

### Input

| Field | Default | What it does |
|---|---|---|
| `startUrls` | crawlee.dev's Python quick start | A site, a section, or a sitemap (`.xml`, `.xml.gz`). |
| `maxPages` | `20` | Pages to chunk. This bounds the run's cost. |
| `chunkTokens` | `512` | Largest chunk in tokens, heading line included. |
| `chunkOverlapTokens` | `50` | Whole paragraphs repeated from the end of the previous chunk of the same section. |
| `headingContext` | `true` | Start each chunk with its `[Page > Section]` line. |
| `scope` | `path` | `path`: the start URL's folder and below. `domain`: the whole host. |
| `includeGlobs` / `excludeGlobs` | changelogs and versioned docs excluded | URL patterns, e.g. `**/docs/v1/**`. |
| `useSitemaps` | `true` | Add the in-scope URLs from the site's sitemaps. |
| `dedupe` | `true` | Skip duplicate pages and drop duplicate chunks. |
| `keepLinks` | `false` | Keep `[text](url)` links. Off saves tokens: only the link text stays. |
| `respectRobotsTxt` | `true` | Skip what robots.txt disallows. |
| `maxConcurrency` | `5` | Pages fetched at once. |
| `timeoutSecs` | `20` | A slower page is skipped and not charged. |

```json
{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "maxPages": 500,
  "chunkTokens": 400,
  "excludeGlobs": ["**/changelog**", "**/v1/**"]
}
```

### Output

One dataset item per chunk, ready for a vector store: `text` is what to embed, the other fields are metadata.

```json
{
  "id": "d2d220b9bcf1db24f385240a",
  "url": "https://crawlee.dev/python/docs/introduction/setting-up",
  "title": "Setting up | Crawlee for Python · Fast, reliable Python web crawlers.",
  "chunkIndex": 2,
  "chunkCount": 8,
  "headingPath": ["Setting up", "Installing Crawlee"],
  "citation": "https://crawlee.dev/python/docs/introduction/setting-up#installing-crawlee",
  "tokens": 154,
  "text": "[Setting up > Installing Crawlee]\n\n## Installing Crawlee\n\nCrawlee is available as `crawlee` package on PyPI…",
  "contentHash": "f1ac9d771a1176dd",
  "pageContentHash": "dab95b14c2c68845",
  "language": "en"
}
```

The run's key-value store holds an `OUTPUT` summary. It gives the pages chunked with their token counts and hashes, the chunk and token totals, the duplicates dropped, and why any URL was skipped (`http 404`, `robots.txt`, `timeout`, `not a page (application/json)`…).

### Use it from an AI agent or a pipeline

- **MCP:** add it to [Apify's MCP server](https://mcp.apify.com) with `?tools=webprobe-tools/website-to-rag-chunks`, or let the agent find it with `search-actors`. Every input field has a description and a default.
- **API, one call:** `POST https://api.apify.com/v2/acts/webprobe-tools~website-to-rag-chunks/run-sync-get-dataset-items` with the input as JSON returns the chunks. Use it for small crawls; for large ones, start a run and read the dataset when it finishes.
- **Vector stores:** Apify's Pinecone, Qdrant and other vector database integrations can load this dataset: point the field to embed at `text` and keep the rest as metadata.

### Pricing

Free. The Actor charges nothing of its own; a run costs only the Apify platform usage it consumes. Pricing can change later; this page always shows the current price.

### Good to know

- It reads the HTML the server sends. Content that only JavaScript renders is not seen, and most documentation sites don't need it.
- Token counts follow OpenAI's `o200k_base` encoding. Other models' tokenizers count somewhat differently, so leave headroom under a hard limit.
- `maxPages` counts pages read, including duplicates that are skipped, so a run with duplicates chunks fewer pages than `maxPages`.
- Crawl your own sites, or public sites whose terms allow it. The Actor reads only public pages, sends no login, and respects robots.txt unless you turn that off.
- To read a handful of URLs at once and get only the passages that answer a question, use Web Page to LLM Context. To list every URL in a sitemap, use the Sitemap URL Extractor.

### Support

Open an issue on this Actor's page. Replies come from the AI agent that maintains it, signed as such.

# Actor input Schema

## `startUrls` (type: `array`):

Where to start: a site (https://docs.example.com), a section (https://example.com/docs/), or a sitemap (.xml or .xml.gz). A bare domain is read as https://. Only pages at or under each start URL's folder are crawled unless scope is "domain".

## `maxPages` (type: `integer`):

Stop after chunking this many pages. This bounds the run's cost.

## `chunkTokens` (type: `integer`):

Largest chunk, in OpenAI o200k_base tokens, heading line included. 256–1024 suits most embedding models.

## `chunkOverlapTokens` (type: `integer`):

Repeat up to this many tokens of whole paragraphs from the end of the previous chunk of the same section. 0 turns overlap off.

## `headingContext` (type: `boolean`):

Prefix every chunk with a line such as \[Quick start > Installation], so a retriever can tell apart chunks from same-named sections.

## `scope` (type: `string`):

path: only URLs at or under each start URL's folder. domain: the whole host.

## `includeGlobs` (type: `array`):

Crawl only URLs matching one of these, e.g. https://example.com/docs/\*\*. Empty: everything in scope.

## `excludeGlobs` (type: `array`):

Never crawl URLs matching one of these, e.g. **/changelog/**.

## `useSitemaps` (type: `boolean`):

Read the sitemaps robots.txt names (or /sitemap.xml); their in-scope URLs are crawled after the linked pages, so pages no link reaches are found too.

## `dedupe` (type: `boolean`):

Skip a page whose content matches one already chunked (it is not charged), and drop a chunk identical to one already output.

## `keepLinks` (type: `boolean`):

Keep Markdown links as [text](url). Off: only the link text stays, which saves tokens.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs the site's robots.txt disallows.

## `maxConcurrency` (type: `integer`):

How many pages are fetched at once.

## `timeoutSecs` (type: `integer`):

A page that does not answer in time is skipped and not charged.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://crawlee.dev/python/docs/quick-start"
    }
  ],
  "maxPages": 20,
  "chunkTokens": 512,
  "chunkOverlapTokens": 50,
  "headingContext": true,
  "scope": "path",
  "includeGlobs": [],
  "excludeGlobs": [
    "**/changelog**",
    "**/docs/0.*/**",
    "**/docs/next/**"
  ],
  "useSitemaps": true,
  "dedupe": true,
  "keepLinks": false,
  "respectRobotsTxt": true,
  "maxConcurrency": 5,
  "timeoutSecs": 20
}
```

# Actor output Schema

## `chunks` (type: `string`):

One item per chunk: id, text, tokens, heading path, citation URL and content hashes.

## `summary` (type: `string`):

Pages chunked with their token counts and hashes, duplicates dropped and URLs skipped.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://crawlee.dev/python/docs/quick-start"
        }
    ],
    "maxPages": 20,
    "excludeGlobs": [
        "**/changelog**",
        "**/docs/0.*/**",
        "**/docs/next/**"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("webprobe-tools/website-to-rag-chunks").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://crawlee.dev/python/docs/quick-start" }],
    "maxPages": 20,
    "excludeGlobs": [
        "**/changelog**",
        "**/docs/0.*/**",
        "**/docs/next/**",
    ],
}

# Run the Actor and wait for it to finish
run = client.actor("webprobe-tools/website-to-rag-chunks").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://crawlee.dev/python/docs/quick-start"
    }
  ],
  "maxPages": 20,
  "excludeGlobs": [
    "**/changelog**",
    "**/docs/0.*/**",
    "**/docs/next/**"
  ]
}' |
apify call webprobe-tools/website-to-rag-chunks --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,webprobe-tools/website-to-rag-chunks"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/KfaWXeq2LZmRbADN6/builds/jpLKvphz9x0ReibzY/openapi.json