# Website to RAG Chunks: Markdown, Tokens & Incremental Crawl (`nacenempresas/rag-website-chunks`) Actor

Turn any website, docs site, PDF or DOCX into clean, heading-aware RAG chunks with exact token counts. Incremental mode only returns new or changed pages. Pay per page, not per chunk.

- **URL**: https://apify.com/nacenempresas/rag-website-chunks.md
- **Developed by:** [NacenEmpresas](https://apify.com/nacenempresas) (community)
- **Categories:** AI, Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.25 / 1,000 pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to RAG Chunks: Markdown, Tokens & Incremental Crawl

Turn any website, documentation portal, PDF or Word file into **clean, heading-aware chunks with exact token counts**, ready to load into Pinecone, Qdrant, Weaviate, Chroma, pgvector or any LLM pipeline.

**$0.25 per 1,000 pages. Chunking included. You pay per page, never per chunk.**

### What you can do with it

- 🤖 **Build a RAG chatbot over your docs** – crawl `docs.yoursite.com`, upsert the chunks, done.
- 🔄 **Keep a vector index fresh** – schedule a daily run in *incremental mode*: only new or changed pages are returned and charged.
- 📄 **Ingest PDFs and DOCX** linked from a site (manuals, policies, papers) in the same run.
- 🧠 **Give AI agents a tool** – works through Apify's MCP server, so Claude, Cursor or any MCP client can call it.
- 📏 **Budget your embeddings** – every chunk has its exact token count (`cl100k_base` or `o200k_base`).

### Why this Actor

| | This Actor | Typical crawler | Typical "RAG" Actor |
| --- | --- | --- | --- |
| Chunking by headings → paragraphs → sentences | ✅ | ❌ do it yourself | ✅ |
| Chunks **never** exceed your limit | ✅ | – | ⚠️ often |
| Exact tokens (tiktoken) | ✅ | ❌ | ⚠️ estimated |
| Price model | **per page** | per page | per chunk (10× on long pages) |
| Browser only when a page comes back empty | ✅ auto | manual choice | ❌ |
| PDF / DOCX / llms.txt | ✅ | partial | ❌ |
| Incremental (ETag, Last-Modified, sitemap lastmod, content hash) | ✅ | date filter | ❌ |
| Sitemap discovery on by default | ✅ | ✅ | ❌ |
| Removes menus, footers, cookie banners repeated across pages | ✅ | ✅ | ❌ |
| Duplicate chunks & near-duplicate pages removed | ✅ | ❌ | ❌ |

### Quick start

1. Paste one or more URLs in **Start URLs** (a site, a docs folder, a sitemap or a PDF link).
2. Click **Start**. The defaults work: 512-token chunks, 64-token overlap, sitemap on, robots.txt respected.
3. Export the dataset as JSON/CSV, or send it straight to your vector database with an Apify integration.

### Output: one row per chunk

```json
{
  "id": "5f0c1a7e9b2d4c88a1e3f6b0d2c4e8a1",
  "url": "https://docs.example.com/api/auth",
  "title": "Authentication",
  "headingPath": ["API", "Authentication", "Tokens"],
  "chunkIndex": 2,
  "chunkCount": 5,
  "tokens": 498,
  "text": "API > Authentication > Tokens\n\nAccess tokens expire after one hour...",
  "contentHash": "sha256:9d1e…",
  "lastModified": "2026-10-30",
  "language": "en",
  "source": "html",
  "crawledAt": "2026-11-05T10:12:44+00:00"
}
```

- `id` is stable per URL and chunk position, so re-runs **upsert** instead of duplicating.
- `source` is `html`, `browser`, `pdf`, `docx`, `txt` or `md`.
- Each chunk starts with its section path, which improves retrieval quality.
- Set **Output** to `pages` for one row per page with full Markdown, or `both`.

Every run also saves to its key-value store:

- `llms.txt` – an index of the crawled site in the [llms.txt](https://llmstxt.org) format.
- `content.md` – the whole site as a single Markdown file.
- `SUMMARY` – pages returned, browser pages, documents, tokens, unchanged/skipped/failed URLs and charged events.

### Pricing (pay per event)

| Event | Price | When |
| --- | --- | --- |
| Page | **$0.25 / 1,000** | HTML, TXT or Markdown page with useful text, chunking included |
| Browser page | $2.00 / 1,000 | Page that needed JavaScript rendering (only when HTTP came back empty) |
| Document | $1.00 / 1,000 | PDF or DOCX file converted |
| Actor start | $0.05 / 1,000 runs | Apify's standard start fee |

**Never charged:** errors, 404s, empty pages, pages blocked by robots.txt or `noindex`, duplicates, and pages unchanged since your last incremental run. The run stops cleanly at your **Max pages** or at the spending limit you set.

Example: a 2,000-page documentation site costs about **$0.50**. Re-crawling it daily in incremental mode usually costs a few cents.

### Incremental mode (keep your index in sync)

Turn on **Incremental mode** and schedule the Actor (Apify → *Schedules*). Each run:

1. Re-uses the ETag / Last-Modified headers and sitemap `lastmod` dates from the previous run to avoid downloading what did not change.
2. Compares a content fingerprint of every downloaded page.
3. Returns and charges **only new or changed pages**.

State is kept in a named key-value store in your account (derived from your start URLs, or set **State store name**).

### Use with your stack

**Python (apify-client)**

```python
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("nacenempresas/rag-website-chunks").call(run_input={
    "startUrls": [{"url": "https://docs.example.com"}],
    "chunkSize": 512,
    "incremental": True,
})
for chunk in client.dataset(run["defaultDatasetId"]).iterate_items():
    vector_db.upsert(id=chunk["id"], text=chunk["text"], metadata={"url": chunk["url"], "section": chunk["headingPath"]})
```

**LangChain**

```python
from langchain_apify import ApifyDatasetLoader
from langchain_core.documents import Document

loader = ApifyDatasetLoader(
    dataset_id="<DATASET_ID>",
    dataset_mapping_function=lambda c: Document(page_content=c["text"], metadata={"source": c["url"], "section": " > ".join(c["headingPath"])}),
)
docs = loader.load()  # already chunked – no text splitter needed
```

**No code:** use Apify's Pinecone, Qdrant or Make/Zapier/n8n integrations on the dataset.

### Input options

| Field | Default | What it does |
| --- | --- | --- |
| Start URLs | – | Sites, folders, sitemaps or document links |
| Max pages | 100 | Pages/documents returned and charged per run |
| Chunk size / overlap | 512 / 64 tokens | Hard upper limit per chunk and overlap inside a section |
| Tokenizer | cl100k_base | `o200k_base` for GPT-4o and newer, or a fast approximation |
| Output | chunks | `chunks`, `pages` or `both` |
| Incremental mode | off | Only new or changed pages |
| Use sitemap | on | Finds pages that are not linked |
| Stay within path | on | Only crawl under the start folder |
| Include / exclude globs | – | e.g. `https://site.com/docs/**`, `**/changelog/**` |
| Linked documents | PDF, DOCX, TXT, MD | Converted and chunked too |
| JavaScript rendering | auto | Browser only for pages that come back empty |
| Remove boilerplate | on | Drops repeated menus/footers, duplicate chunks and near-duplicate pages |
| Respect robots.txt | on | Skips disallowed URLs |
| Proxy | off | Use your own or Apify Proxy for sites that block datacenter IPs |

### FAQ

**Does it handle JavaScript sites?** Yes. Every page is fetched over fast HTTP first; only pages that come back (almost) empty are re-rendered in headless Chromium and billed as *browser page*. Set *JavaScript rendering* to `always` for apps that need it everywhere.

**Why per page and not per chunk?** Per-chunk pricing punishes long pages and small chunk sizes. Here a long page costs the same as a short one, so you can pick the chunk size that is best for retrieval, not for your bill.

**Is the token count exact?** Yes, it uses OpenAI's `tiktoken` with the encoding you choose. Most embedding models tokenize similarly, so it is a reliable budget for any provider.

**Can it log in or bypass paywalls?** No. It reads public pages only and respects robots.txt by default.

**Is it legal to crawl a site?** You decide which sites to read and are responsible for how you use the data. Respect each site's terms and copyright. The Actor does not search for or extract personal data.

### Support

Found a page that comes back wrong? Open an issue on the **Issues** tab with the URL – we answer within 48 hours. Changes are listed in the changelog.

# Changelog

This Actor's version history is a separate document: https://apify.com/nacenempresas/rag-website-chunks/changelog.md

# Actor input Schema

## `startUrls` (type: `array`):

Websites, docs sites, sitemaps or direct links to PDF/DOCX/TXT/MD/llms.txt files. Pages under the same path are discovered automatically via links and the sitemap.

## `maxPages` (type: `integer`):

Maximum number of pages/documents returned (and charged) in this run. The crawl also stops at your run's spending limit.

## `chunkSize` (type: `integer`):

Maximum tokens per chunk. No chunk ever exceeds this limit: text is split by section, then paragraph, then sentence.

## `chunkOverlap` (type: `integer`):

Tokens repeated between consecutive chunks of the same section (max half of chunk size).

## `tokenizer` (type: `string`):

cl100k_base (GPT-4, GPT-3.5, text-embedding-3), o200k_base (GPT-4o and newer) or a fast approximation.

## `output` (type: `string`):

chunks: one row per chunk (ready for a vector database). pages: one row per page with full Markdown. both: both row types with a `type` field.

## `incremental` (type: `boolean`):

Remembers a fingerprint of every page between runs. Unchanged pages are skipped and not charged. Ideal for scheduled runs that keep a vector index fresh.

## `stateStoreName` (type: `string`):

Optional name of the key-value store that keeps incremental state. Leave empty to derive it from the start URLs.

## `useSitemap` (type: `boolean`):

Discover pages from robots.txt sitemaps and /sitemap.xml, including pages that are not linked.

## `stayWithinPath` (type: `boolean`):

Only crawl URLs under the start URL's folder (e.g. /docs/). Disable to crawl the whole domain.

## `includeGlobs` (type: `array`):

Only crawl URLs matching these patterns, e.g. https://example.com/docs/\*\*

## `excludeGlobs` (type: `array`):

Never crawl URLs matching these patterns, e.g. **/changelog/**

## `maxDepth` (type: `integer`):

How many link hops from the start URL to follow.

## `includeFiles` (type: `array`):

Linked files that are downloaded and chunked too.

## `jsFallback` (type: `string`):

auto: fast HTTP first, headless browser only for pages that come back (almost) empty. always: render every page. never: HTTP only.

## `removeBoilerplate` (type: `boolean`):

Drop menus, footers and cookie notices that repeat across pages, plus duplicate chunks and near-duplicate pages.

## `respectRobotsTxt` (type: `boolean`):

Skip URLs disallowed by the site's robots.txt.

## `maxConcurrency` (type: `integer`):

Parallel HTTP requests.

## `proxyConfiguration` (type: `object`):

Optional. Use a proxy for sites that block datacenter IPs.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "maxPages": 50,
  "chunkSize": 512,
  "chunkOverlap": 64,
  "tokenizer": "cl100k_base",
  "output": "chunks",
  "incremental": false,
  "useSitemap": true,
  "stayWithinPath": true,
  "maxDepth": 20,
  "includeFiles": [
    "PDF",
    "DOCX",
    "TXT",
    "MD"
  ],
  "jsFallback": "auto",
  "removeBoilerplate": true,
  "respectRobotsTxt": true,
  "maxConcurrency": 10,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `chunks` (type: `string`):

One row per chunk (or per page with output=pages), ready to upsert into a vector database.

## `llmsTxt` (type: `string`):

Index of the crawled pages in the llms.txt format.

## `contentMarkdown` (type: `string`):

All crawled pages concatenated as a single Markdown file.

## `summary` (type: `string`):

Pages returned, browser pages, documents, tokens, skipped and failed URLs and charged events.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy"
        }
    ],
    "maxPages": 50
};

// Run the Actor and wait for it to finish
const run = await client.actor("nacenempresas/rag-website-chunks").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/academy" }],
    "maxPages": 50,
}

# Run the Actor and wait for it to finish
run = client.actor("nacenempresas/rag-website-chunks").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "maxPages": 50
}' |
apify call nacenempresas/rag-website-chunks --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nacenempresas/rag-website-chunks"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/e3I9wAiMoykTbS2j4/builds/aHCl3LOeRL3PDaaHc/openapi.json