# Website to RAG Chunks (`cynix_dev/web-to-rag-chunks`) Actor

Crawl any website and turn its pages into clean, chunked, metadata-rich Markdown records ready for RAG pipelines, vector stores, and custom GPTs.

- **URL**: https://apify.com/cynix\_dev/web-to-rag-chunks.md
- **Developed by:** [Cynix Dev](https://apify.com/cynix_dev) (community)
- **Categories:** AI, Developer tools
- **Stats:** 2 total users, 1 monthly users, 12.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.25 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Website to RAG Chunks

Crawl any website and get back **clean, chunked, metadata-rich Markdown** ready to embed — sized chunks with overlap, heading ancestry for citations, and token estimates. Built for RAG pipelines, vector stores and custom GPTs.

### What it does

Getting a website into a vector database normally means writing a crawler, stripping boilerplate, converting to Markdown, chunking sensibly and preserving enough metadata to cite sources. This Actor does all of it and hands you records that go straight into Pinecone, Qdrant, Weaviate, pgvector, Chroma or a custom GPT's knowledge file.

Crawl scope is yours to control: depth, page cap, same-domain restriction, include/exclude globs, and optional sitemap discovery so you can enumerate a documentation site properly instead of hoping links cover it. PDFs linked from the crawl can be extracted too.

### Features

- **Chunking that respects structure** — `chunkSize` with `chunkOverlap`, and `minChunkChars` to merge away useless fragments.
- **Heading path per chunk** — `headingPath` records the H1→H2→H3 ancestry, so citations can say exactly where text came from.
- **Token estimates** — `tokenEstimate` on every chunk for budgeting embedding and context costs.
- **Sitemap crawling** — `useSitemap` enumerates a site properly, honouring your include/exclude globs.
- **PDF extraction** — `extractPdfs` pulls text out of linked PDFs, with a per-file page cap.
- **Glob scoping** — `includeGlobs` / `excludeGlobs` to crawl `/docs/**` and skip `/blog/**`.
- **Metadata enrichment** — Open Graph, Twitter Card, JSON-LD and meta tags attached to chunks.
- **Content hashing** — `contentHash` per chunk makes deduplication and incremental re-indexing trivial.

### What people use it for

- Build a RAG knowledge base from product or API documentation.
- Feed a custom GPT or assistant with your own site's content.
- Populate a vector store for semantic search across a docs portal.
- Create an internal Q\&A bot over a knowledge base or handbook.
- Incremental re-indexing — hash comparison shows exactly which chunks changed.

### Choosing chunk size and overlap

There is no universally right answer, but these are sound starting points:

| Use case | `chunkSize` | `chunkOverlap` |
| --- | --- | --- |
| Precise Q\&A over docs | 500–800 | 100 |
| General RAG (default) | 1000 | 150 |
| Long-form summarisation | 2000+ | 200 |

Roughly four characters make one token, so `chunkSize: 1000` lands near 250 tokens. Overlap exists so a sentence split across a boundary is still retrievable from at least one chunk — without it, answers spanning a boundary get lost.

#### Crawling a documentation site properly

```json
{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "useSitemap": true,
  "includeGlobs": ["https://docs.example.com/**"],
  "excludeGlobs": ["**/changelog/**", "**/blog/**"],
  "maxPages": 500,
  "chunkSize": 1000,
  "chunkOverlap": 150
}
```

`useSitemap` is the difference between crawling a docs site and *hoping* your link graph reached every page.

#### Keeping an index fresh

`contentHash` identifies chunk content exactly. Store it alongside your vectors, re-run the Actor on a schedule, and re-embed only chunks whose hash changed — which is usually a tiny fraction and keeps embedding costs near zero.

### Input

`startUrls` is required. Defaults produce a small, safe crawl — raise `maxCrawlDepth` and `maxPages` deliberately once you know the shape of the site.

| Field | Type | Default | What it does |
| --- | --- | --- | --- |
| `startUrls` **(required)** | array | `[{"url": "https://docs.apify.com/platform"}]` | Public web pages to crawl and convert into RAG-ready chunks. |
| `maxCrawlDepth` | integer | `1` | How many link levels to follow from each start URL. 0 = only the start URLs. Range 0–10. |
| `maxPages` | integer | `50` | Hard cap on total pages crawled per run. Range 1–10000. |
| `sameDomainOnly` | boolean | `true` | Only follow links that stay on the same domain as the start URL. |
| `includeGlobs` | array | `[]` | Only enqueue URLs matching these glob patterns (e.g. https://site.com/docs/\*\*). |
| `excludeGlobs` | array | `[]` | Skip URLs matching these glob patterns. |
| `chunkSize` | integer | `1000` | Target maximum characters per chunk (~4 chars per token). Range 200–20000. |
| `chunkOverlap` | integer | `150` | Characters of overlap carried between consecutive chunks for context continuity. Range 0–5000. |
| `minChunkChars` | integer | `200` | Drop or merge chunks smaller than this size. Range 1–5000. |
| `useSitemap` | boolean | `false` | Fetch and parse sitemap.xml to discover all crawlable URLs before starting. Respects include/exclude globs. |
| `sitemapUrls` | array | `[]` | Additional sitemap URLs to fetch (e.g. https://site.com/sitemap-docs.xml). Auto-discovers /sitemap.xml if empty. |
| `extractPdfs` | boolean | `false` | Download and extract text from PDF links found during crawl (uses pdf-parse). Adds PDF chunks to dataset. |
| `maxPdfPages` | integer | `50` | Limit pages extracted per PDF (0 = all). Range 0–500. |
| `includeHeadingPath` | boolean | `true` | Add headingPath array to each chunk showing the H1->H2->H3... ancestry for better citation context. |
| `enrichMetadata` | boolean | `true` | Extract Open Graph, Twitter Card, JSON-LD, and meta tags as additional chunk metadata. |
| `proxyConfiguration` | object | see below | Apify Proxy configuration for blocked sites. |

#### Input example

```json
{
  "startUrls": [
    {
      "url": "https://example.com"
    }
  ],
  "maxCrawlDepth": 0,
  "maxPages": 1,
  "sameDomainOnly": true,
  "chunkSize": 500,
  "chunkOverlap": 50,
  "minChunkChars": 50,
  "includeHeadingPath": true,
  "enrichMetadata": true,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "proxyUrls": []
  },
  "useSitemap": false,
  "extractPdfs": false,
  "maxPdfPages": 50
}
```

### Output

One record per chunk, carrying its source URL and page title, its position (`chunkIndex` of `totalChunks`), the Markdown `text`, character count, token estimate, content hash and crawl timestamp.

Every dataset record contains: `url`, `title`, `chunkIndex`, `totalChunks`, `text`, `charCount`, `tokenEstimate`, `contentHash`, `crawledAt`, `headingPath`, `sourceType`, `metadata`.

#### Output example

A real record from a run of this Actor:

```json
{
  "url": "https://example.com/",
  "title": "Example Domain",
  "chunkIndex": 0,
  "totalChunks": 1,
  "text": "This domain is for use in documentation examples without needing permission. Avoid use in operations.\n\n[Learn more](https://iana.org/domains/example)",
  "charCount": 149,
  "tokenEstimate": 38,
  "contentHash": "a80242055d7489b4ce56b3193e7521cd92ef34fdc5b6aa666d7bef8cc29ccfd5",
  "crawledAt": "2026-08-22T15:28:17.922Z",
  "headingPath": [
    "Example Domain"
  ],
  "sourceType": "html",
  "metadata": {
    "viewport": "width=device-width, initial-scale=1"
  }
}
```

Export the dataset as JSON, CSV, Excel, XML or JSONL from the Console, or pull it programmatically through the Apify API and any of the official clients.

### How to use it

1. Click **Try for free** (or **Start** if you already have an Apify account).
2. Fill in the input fields described above — the defaults already produce a working run.
3. Press **Start** and watch the log; results stream into the dataset as they are found.
4. When the run finishes, open the **Output/Storage** tab and export as JSON, CSV or Excel.

Runs can be scheduled (hourly, daily, weekly) and wired into Slack, Google Sheets, Zapier, Make, webhooks or your own backend through Apify integrations. Everything the Console does is also available over the [Apify API](https://docs.apify.com/api/v2).

### Proxy configuration

This Actor accepts a standard Apify **proxy configuration** object. Residential proxy is the default because the target site rate-limits datacenter IP ranges; you can select a specific exit country or supply your own proxy URLs.

```json
{
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

### Pricing

This Actor is billed on Apify's **pay-per-event** model: a small charge when a run starts, plus a charge for each result written to the dataset. You only pay for records you actually receive — a run that finds nothing costs only the start event. Current rates are always shown on the **Pricing** tab of this page, and the run log prints your usage as it goes.

Free-plan credits from Apify cover a large amount of light usage, so you can evaluate the Actor before committing to anything.

### FAQ

#### What format is the chunk text in?

Markdown. Headings, lists, links and code blocks survive, which both embeds better than raw HTML and reads better when an LLM cites it.

#### How do I use `headingPath` for citations?

It's the heading ancestry of the chunk, e.g. `["API Reference", "Authentication", "API keys"]`. Store it as metadata and your assistant can cite "API Reference › Authentication › API keys" instead of just a bare URL.

#### Does it handle JavaScript-rendered sites?

It crawls and extracts page content, and works well on server-rendered and statically generated sites — which covers nearly all documentation. Heavily client-rendered apps with no server HTML may yield thin text; check a small run before committing to a large crawl.

#### Can I crawl a site that blocks datacenter IPs?

Yes — configure `proxyConfiguration` with Apify Proxy, RESIDENTIAL group if needed.

#### Will it crawl the entire internet by accident?

No. `sameDomainOnly` is on by default, `maxCrawlDepth` defaults to 1 and `maxPages` defaults to 50. You have to deliberately widen the scope.

#### How do PDFs get chunked?

Enable `extractPdfs` and linked PDFs are downloaded, text-extracted and chunked with the same settings as HTML pages. `maxPdfPages` caps very long documents.

### Other Actors by cynix\_dev

| Actor | What it does |
| --- | --- |
| [Dataset Drift & QA Monitor](https://apify.com/cynix_dev/dataset-drift-qa) | Stop finding out your scrapers broke three days late. Point this actor at any Apify dataset or JSON endpoint and it watches the … |
| [Page Change Monitor](https://apify.com/cynix_dev/page-change-monitor) | Monitor web pages for content changes. Diffs each run against the previous snapshot and emits structured change records with … |
| [OpenStreetMap Geocoder](https://apify.com/cynix_dev/osm-geocoder) | Forward and reverse geocoding via the free Komoot Photon / OpenStreetMap service. No API key, no scraping, ODbL data. |
| [arXiv Papers Extractor](https://apify.com/cynix_dev/arxiv-papers) | Search arXiv and extract papers as clean typed records: title, abstract, authors, categories, DOI, and direct PDF links. |
| [Page Change Monitor](https://apify.com/cynix_dev/page-change-monitor) | Monitor web pages for content changes. Diffs each run against the previous snapshot and emits structured change records with … |

### Legal and responsible use

This Actor collects only publicly available information. You are responsible for how you use the data, including compliance with the target site's Terms of Service, robots directives, copyright, and data protection law such as GDPR and CCPA. Do not use it to gather personal data without a lawful basis.

### Support and feedback

Found a bug, hit a site change, or need an extra field? Open a ticket on the **Issues** tab of this Actor — issues are read and fixed. Feature requests and custom-scraper enquiries are welcome through the same channel.

# Actor input Schema

## `startUrls` (type: `array`):

Public web pages to crawl and convert into RAG-ready chunks.

## `maxCrawlDepth` (type: `integer`):

How many link levels to follow from each start URL. 0 = only the start URLs.

## `maxPages` (type: `integer`):

Hard cap on total pages crawled per run.

## `sameDomainOnly` (type: `boolean`):

Only follow links that stay on the same domain as the start URL.

## `includeGlobs` (type: `array`):

Only enqueue URLs matching these glob patterns (e.g. https://site.com/docs/\*\*).

## `excludeGlobs` (type: `array`):

Skip URLs matching these glob patterns.

## `chunkSize` (type: `integer`):

Target maximum characters per chunk (~4 chars per token).

## `chunkOverlap` (type: `integer`):

Characters of overlap carried between consecutive chunks for context continuity.

## `minChunkChars` (type: `integer`):

Drop or merge chunks smaller than this size.

## `useSitemap` (type: `boolean`):

Fetch and parse sitemap.xml to discover all crawlable URLs before starting. Respects include/exclude globs.

## `sitemapUrls` (type: `array`):

Additional sitemap URLs to fetch (e.g. https://site.com/sitemap-docs.xml). Auto-discovers /sitemap.xml if empty.

## `extractPdfs` (type: `boolean`):

Download and extract text from PDF links found during crawl (uses pdf-parse). Adds PDF chunks to dataset.

## `maxPdfPages` (type: `integer`):

Limit pages extracted per PDF (0 = all).

## `includeHeadingPath` (type: `boolean`):

Add headingPath array to each chunk showing the H1->H2->H3... ancestry for better citation context.

## `enrichMetadata` (type: `boolean`):

Extract Open Graph, Twitter Card, JSON-LD, and meta tags as additional chunk metadata.

## `proxyConfiguration` (type: `object`):

Apify Proxy configuration for blocked sites.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform"
    }
  ],
  "maxCrawlDepth": 1,
  "maxPages": 50,
  "sameDomainOnly": true,
  "includeGlobs": [],
  "excludeGlobs": [],
  "chunkSize": 1000,
  "chunkOverlap": 150,
  "minChunkChars": 200,
  "useSitemap": false,
  "sitemapUrls": [],
  "extractPdfs": false,
  "maxPdfPages": 50,
  "includeHeadingPath": true,
  "enrichMetadata": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

One record per RAG-ready Markdown chunk with source metadata.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/platform"
        }
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("cynix_dev/web-to-rag-chunks").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/platform" }],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("cynix_dev/web-to-rag-chunks").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/platform"
    }
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call cynix_dev/web-to-rag-chunks --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,cynix_dev/web-to-rag-chunks"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/NP0wQ02FI5Gluye09/builds/2sgbohqCRMz4VjES3/openapi.json