# Website to RAG Chunks - Re-crawl and Update Only What Changed (`that_red_bird/website-to-rag-incremental`) Actor

Stop re-embedding your whole site. Crawl docs or any site into heading-aware, token-counted RAG chunks; on re-runs get only new and changed chunks plus the ids to delete from your vector DB. Sitemap and robots.txt aware. JSON, JSONL, Markdown. No embedding or API key.

- **URL**: https://apify.com/that_red_bird/website-to-rag-incremental.md
- **Developed by:** [mohamed alaya](https://apify.com/that_red_bird) (community)
- **Categories:** AI, Agents, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $10.00 / 1,000 site crawleds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Website to RAG Chunks - Re-crawl and Update Only What Changed

Turn a website or docs site into **embedding-ready RAG chunks**, then keep your vector database fresh
by re-running the Actor on a schedule: after the first run it outputs **only the chunks that were added or
edited, plus the ids of the chunks to delete**. You re-embed a handful of chunks instead of the whole site.

No embeddings are created here and no API key is needed. The Actor delivers clean text, metadata and
stable ids; you bring your own embedding model and vector store (recipes below).
(Formerly "RAG Dataset Builder".)

### Why incremental

Most website-to-RAG crawlers hand you the entire site again on every run. For a docs site with 5,000
pages that means re-embedding thousands of chunks to pick up the three pages that changed, and no
clean way to remove chunks of pages that were deleted. This Actor remembers the last crawl of each site
and gives you exactly the diff:

| Run | What you get |
|---|---|
| First run | Every chunk, marked `changeType: "added"` |
| Later runs | Only new/edited chunks (`operation: "upsert"`) and `operation: "delete"` rows with the ids to remove |
| Nothing changed | Just a summary row saying your index is already in sync |

Chunk ids are **content-addressed** (`<page hash>-<text hash>`). The same text always gets the same id,
so upserts are idempotent. Edited text gets a new id and the old id appears in the delete list.

### What you get per chunk

| Field | Meaning |
|---|---|
| `id` | Stable content-based id. Use it as your vector id |
| `operation` / `changeType` | `upsert` + `added`/`updated`/`unchanged`, or `delete` |
| `text` | Markdown of the chunk (headings, lists, tables and fenced code kept intact) |
| `url` / `sourceUrl` | Canonical URL (from `<link rel=canonical>` when same-origin) and the URL actually fetched |
| `pageTitle`, `language` | From the page |
| `headingPath`, `headingPathString` | Where the chunk sits, e.g. `API clients > JavaScript API client` |
| `chunkIndex`, `chunkCount` | Position within the page |
| `tokenCount` | **Estimate** (about 4 characters per token). Not a real tokenizer, expect roughly +/-15% against cl100k |
| `contentHash`, `pageHash` | SHA-256 prefixes for your own change tracking |
| `metadata` | Ready-made payload object for your vector DB (url, title, headingPath, language, contentHash...) |

#### Sample output

```json
{
  "type": "chunk", "operation": "upsert", "changeType": "added",
  "id": "1cade10765-d0c6204cead5",
  "url": "https://docs.apify.com/api",
  "pageTitle": "Apify Documentation", "language": "en",
  "headingPathString": "API clients > JavaScript API client",
  "chunkIndex": 3, "chunkCount": 7, "tokenCount": 119,
  "text": "### JavaScript API client\n\nFor web browser, JavaScript/TypeScript applications, Node.js, Deno, or Bun...",
  "contentHash": "d0c6204cead5b3e0",
  "metadata": { "url": "https://docs.apify.com/api", "headingPath": "API clients > JavaScript API client", "chunkIndex": 3 }
}
```

```json
{ "type": "delete", "operation": "delete", "id": "d9c3495eed-f91590ecfee9",
  "url": "https://example.com/install", "reason": "content-changed" }
```

```json
{ "type": "summary", "mode": "incremental", "pagesAdded": 1, "pagesChanged": 1, "pagesUnchanged": 1,
  "pagesRemoved": 1, "itemsUpserted": 2, "itemsDeleted": 3, "itemsUnchanged": 2,
  "itemsInIndexAfterSync": 4, "tokensUpserted": 93,
  "syncHint": "Upsert the \"upsert\" rows and delete the ids in the \"delete\" rows from your vector DB." }
```

The same changes are also written to the key-value store as `CHANGES.jsonl` (one JSON object per line,
good for bulk loaders), optionally `CHANGES.json` / `CHANGES.md`, plus `DELETED_IDS.json` (just the ids).

### How the incremental sync works

1. The site is crawled (sitemap first, including sitemaps named in `robots.txt`; otherwise links up to 3 levels deep).
2. Each page becomes markdown, is split on headings with a token budget, and near-duplicate chunks across the site are merged.
3. Every chunk gets its content-based id and is compared with the state saved by the last run (stored in a **named key-value store**, default `website-to-rag-state`, one record per site).
4. New ids are upserts, ids that disappeared are deletes. The state is saved **after** the output, so a run that dies half way is simply re-emitted next time (at-least-once).

Safety rules built in:

- A page that **fails with a temporary error** keeps its old state. Its chunks are never deleted because of a timeout.
- A page is deleted only when it returns **404/410**, or when it is **no longer in the sitemap** (and that is only checked when the sitemap was used).
- Pages beyond `maxPages` are never treated as deleted. Lowering `maxPages` will not wipe your index.
- Changing chunk settings (`targetTokens`, `overlapTokens`...) changes the ids, so the next run re-emits everything and deletes the old ids. The summary says `settingsChanged: true`.

### Input

| Field | Default | What it does |
|---|---|---|
| `siteUrl` | `https://docs.apify.com` | Site or docs section. End with a folder (`https://example.com/docs/`) to crawl only that folder |
| `maxPages` | 25 | Pages fetched per run |
| `onlyChanges` | true | Later runs output only changes. Off = full export every time (deletes are still listed) |
| `removeMissingPages` | true | Emit deletes for pages that left the sitemap |
| `resetState` | false | Treat this run as the first one |
| `outputUnit` | chunks | `chunks`, or `pages` for one clean markdown row per page (still with change detection) |
| `outputFormats` | jsonl | Extra files in the key-value store: `jsonl`, `json`, `markdown` |
| `includePatterns` / `excludePatterns` | empty | URL substring filters |
| `respectRobotsTxt` | true | Honour `Disallow` for `User-agent: *` |
| `targetTokens` / `maxTokens` / `overlapTokens` / `minChunkTokens` | 350 / 512 / 40 / 15 | Chunk sizing |
| `dedupThreshold`, `shingleSize` | 85, 5 | Near-duplicate merging across the whole crawl |
| `webhookUrl` | empty | Slack/Discord/any webhook, called only when a re-run finds changes |
| `stateStoreName` | `website-to-rag-state` | Use a different name per project |

Run with empty input `{}` to try it on the default docs site.

### Recipes

All snippets use the Apify Python client. `run_input` can be your own site; schedule the Actor (daily or
weekly) and run the sync step after each run.

```python
from apify_client import ApifyClient
client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("that_red_bird/website-to-rag-incremental").call(run_input={"siteUrl": "https://docs.example.com"})
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
upserts = [r for r in rows if r["type"] == "chunk"]
deletes = [r["id"] for r in rows if r["type"] == "delete"]
vectors = embed([r["text"] for r in upserts])   # your embedding model
```

**Pinecone**

```python
index.upsert(vectors=[(r["id"], v, {**r["metadata"], "text": r["text"]}) for r, v in zip(upserts, vectors)])
if deletes: index.delete(ids=deletes)
```

**Qdrant** (point ids must be UUIDs or integers, so derive one from the chunk id)

```python
import uuid
pid = lambda s: str(uuid.uuid5(uuid.NAMESPACE_URL, s))
client.upsert("docs", points=[PointStruct(id=pid(r["id"]), vector=v, payload={**r["metadata"], "text": r["text"], "chunk_id": r["id"]}) for r, v in zip(upserts, vectors)])
if deletes: client.delete("docs", points_selector=PointIdsList(points=[pid(i) for i in deletes]))
```

**pgvector**

```sql
INSERT INTO chunks (id, url, text, metadata, embedding) VALUES (%s, %s, %s, %s, %s)
ON CONFLICT (id) DO UPDATE SET text = EXCLUDED.text, metadata = EXCLUDED.metadata, embedding = EXCLUDED.embedding;
DELETE FROM chunks WHERE id = ANY(%s);
```

**Weaviate** (object uuid via `generate_uuid5(chunk_id)`): batch-insert the upserts with `properties={**metadata, "text": text}` and `collection.data.delete_by_id(generate_uuid5(i))` for each delete.

**LangChain**

```python
vectorstore.add_documents([Document(page_content=r["text"], metadata=r["metadata"]) for r in upserts], ids=[r["id"] for r in upserts])
if deletes: vectorstore.delete(ids=deletes)
```

**LlamaIndex**: build `TextNode(id_=r["id"], text=r["text"], metadata=r["metadata"])` for each upsert, `index.insert_nodes(nodes)`, and remove deletes with `index.delete_nodes(deletes)` (needs a vector store that supports node deletion).

These are sketches against current client libraries; check argument names for the version you use.

### Pricing

Pay per event, no subscription:

| Event | Price | When |
|---|---|---|
| Site crawled | $0.01 | once per run |
| Page processed | $0.002 | per page fetched and converted |
| Chunk produced | $0.002 | per chunk **output** (unchanged chunks on a re-run are free unless you turn off `onlyChanges`) |

A 100-page docs site is roughly $2 for the first run (about 950 chunks). A weekly re-run where 5 pages changed
is roughly $0.25 (100 pages fetched + a few dozen changed chunks). Deletes, errors and the summary are not charged.
Pages that fail to load are not charged.

### Use cases

- Keep a **support bot / docs assistant** index in sync with your product docs without re-embedding everything.
- Feed an **AI agent or MCP server** with a knowledge base that updates itself on a schedule.
- Build a **competitor or regulation knowledge base** and get a precise list of what changed.
- Export a site as **per-page markdown** (`outputUnit: pages`) for fine-tuning data or a static search index.

### FAQ

**Does it create embeddings?** No. It outputs chunks, metadata and ids. Use any embedding model.

**Does it run JavaScript?** No. It reads server-rendered HTML. Pages that are empty shells without JavaScript produce little or no text.

**How does it detect changes?** By hashing the final chunk text. Cosmetic changes that alter the text (a reworded sentence) count as changes; changes to the page chrome (menus, footers) do not, because only the main content is extracted.

**Why did I get more upserts than I expected after editing one paragraph?** Chunk overlap copies the tail of a chunk into the next one, so an edit can change the following chunk too. Set `overlapTokens` to 0 for the tightest diffs.

**What if my sitemap is huge?** Set `maxPages` and `includePatterns`. Pages outside the limit are left alone, never deleted.

**Is robots.txt respected?** Yes by default (`User-agent: *` rules). It does not read rate-limit hints.

**Can I run several sites?** Yes, run the Actor once per site. State is kept per site and per output unit in the state store.

### Integrations

Works with Make, Zapier, n8n, webhooks, the Apify API/clients and Apify's MCP server so an agent can call it. Use `webhookUrl` for a Slack or Discord ping when a re-run finds changes, or Apify's own run-succeeded webhooks to trigger your sync job.

If this Actor saves you time, a review on the Store page helps others find it. Questions or a site that does not crawl well? Open an issue on the Actor page.

# Actor input Schema

## `siteUrl` (type: `string`):

The site or docs section to turn into RAG chunks, e.g. https://docs.example.com. End it with a slash and a folder (https://example.com/docs/) to crawl only that folder. Pages come from sitemap.xml (also those listed in robots.txt), or from following same-origin links if there is no sitemap.

## `maxPages` (type: `integer`):

Upper bound on pages fetched per run. Pages beyond this limit are never treated as deleted, so lowering it will not wipe your index.

## `onlyChanges` (type: `boolean`):

On every run after the first, output only new and edited chunks (operation upsert) plus the ids to delete (operation delete). Turn off to also get the unchanged chunks again (full export). The first run for a site always outputs everything.

## `removeMissingPages` (type: `boolean`):

When a page that was crawled before is gone from the sitemap, or returns 404/410, output delete rows for its chunk ids. Pages that only failed with a temporary error keep their state and are never deleted.

## `resetState` (type: `boolean`):

Forget the stored crawl for this site and output everything as new. Use after you rebuilt your vector DB from scratch.

## `stateStoreName` (type: `string`):

Named key-value store that remembers what each site looked like at the last run. Use a different name per project or client to keep their states separate.

## `outputUnit` (type: `string`):

chunks: embedding-ready chunks split on headings with a token budget. pages: one clean markdown row per page (no chunking), still with change detection.

## `outputFormats` (type: `array`):

Besides the dataset, also write the changes to the key-value store as CHANGES.jsonl (one JSON object per line, ready for bulk loaders), CHANGES.json or CHANGES.md. DELETED_IDS.json is always written.

## `includePatterns` (type: `array`):

Keep only URLs containing one of these substrings, e.g. /docs/, /guides/. Empty means all URLs.

## `excludePatterns` (type: `array`):

Drop URLs containing any of these substrings, e.g. /tag/, /author/, ?page=.

## `respectRobotsTxt` (type: `boolean`):

Skip pages that robots.txt disallows for all crawlers (User-agent: \*) and read Sitemap: lines from it.

## `useSitemap` (type: `boolean`):

Prefer the sitemap (including sitemap indexes) for discovery. Falls back to following same-origin links up to 3 levels deep when there is none.

## `targetTokens` (type: `integer`):

Preferred chunk size when a section has to be split. A whole section under the max is kept as one chunk. Token counts are estimates (about 4 characters per token), not a real tokenizer.

## `maxTokens` (type: `integer`):

Hard cap per chunk, except a single code block or table bigger than the cap, which is kept whole rather than cut in half. Must be at least the target.

## `overlapTokens` (type: `integer`):

How much of the end of a chunk is repeated at the start of the next chunk of the same page, on a whole sentence or block. 0 disables overlap.

## `minChunkTokens` (type: `integer`):

Chunks estimated below this size are dropped as too small to be useful, e.g. a lone heading with no text.

## `dedupThreshold` (type: `integer`):

Chunks anywhere in the crawl whose shingle similarity is at or above this percentage are merged into one, which removes repeated boilerplate. 85 means 0.85. Higher means fewer merges.

## `shingleSize` (type: `integer`):

Length of the word n-grams used to fingerprint chunks for near-duplicate detection. Smaller is noisier, larger is stricter.

## `includeLinks` (type: `boolean`):

Keep [text](url) links in the extracted markdown. Turn off for pure prose without link noise.

## `includeDuplicateRows` (type: `boolean`):

Output one row per near-duplicate chunk that was merged away (type duplicate), with the surviving chunk id and similarity. Off keeps the dataset a clean corpus.

## `webhookUrl` (type: `string`):

Optional Slack, Discord or generic webhook. It receives a short message only when a re-run finds changes (never on the first run). Leave empty to disable.

## `concurrency` (type: `integer`):

How many pages to fetch in parallel.

## `proxyConfiguration` (type: `object`):

Optional Apify proxy for sites that block datacenter IPs. Off by default.

## Actor input object example

```json
{
  "siteUrl": "https://docs.apify.com",
  "maxPages": 25,
  "onlyChanges": true,
  "removeMissingPages": true,
  "resetState": false,
  "stateStoreName": "website-to-rag-state",
  "outputUnit": "chunks",
  "outputFormats": [
    "jsonl"
  ],
  "includePatterns": [],
  "excludePatterns": [],
  "respectRobotsTxt": true,
  "useSitemap": true,
  "targetTokens": 350,
  "maxTokens": 512,
  "overlapTokens": 40,
  "minChunkTokens": 15,
  "dedupThreshold": 85,
  "shingleSize": 5,
  "includeLinks": true,
  "includeDuplicateRows": false,
  "webhookUrl": "",
  "concurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `downloadCsv` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `changesJsonl` (type: `string`):

No description

## `deletedIds` (type: `string`):

No description

## `count` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "siteUrl": "https://docs.apify.com",
    "stateStoreName": "website-to-rag-state",
    "outputUnit": "chunks",
    "outputFormats": [
        "jsonl"
    ],
    "includePatterns": [],
    "excludePatterns": [],
    "webhookUrl": "",
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("that_red_bird/website-to-rag-incremental").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "siteUrl": "https://docs.apify.com",
    "stateStoreName": "website-to-rag-state",
    "outputUnit": "chunks",
    "outputFormats": ["jsonl"],
    "includePatterns": [],
    "excludePatterns": [],
    "webhookUrl": "",
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("that_red_bird/website-to-rag-incremental").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "siteUrl": "https://docs.apify.com",
  "stateStoreName": "website-to-rag-state",
  "outputUnit": "chunks",
  "outputFormats": [
    "jsonl"
  ],
  "includePatterns": [],
  "excludePatterns": [],
  "webhookUrl": "",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call that_red_bird/website-to-rag-incremental --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,that_red_bird/website-to-rag-incremental"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/tWc76qb46OPyA1Rz2/builds/LcF4PYh5Pmfs4wfPF/openapi.json