Website to RAG Chunks - Re-crawl and Update Only What Changed

Turn a website or docs site into embedding-ready RAG chunks, then keep your vector database fresh by re-running the Actor on a schedule: after the first run it outputs only the chunks that were added or edited, plus the ids of the chunks to delete. You re-embed a handful of chunks instead of the whole site.

No embeddings are created here and no API key is needed. The Actor delivers clean text, metadata and stable ids; you bring your own embedding model and vector store (recipes below). (Formerly "RAG Dataset Builder".)

Why incremental

Most website-to-RAG crawlers hand you the entire site again on every run. For a docs site with 5,000 pages that means re-embedding thousands of chunks to pick up the three pages that changed, and no clean way to remove chunks of pages that were deleted. This Actor remembers the last crawl of each site and gives you exactly the diff:

Run	What you get
First run	Every chunk, marked changeType: "added"
Later runs	Only new/edited chunks (operation: "upsert") and operation: "delete" rows with the ids to remove
Nothing changed	Just a summary row saying your index is already in sync

Chunk ids are content-addressed (<page hash>-<text hash>). The same text always gets the same id, so upserts are idempotent. Edited text gets a new id and the old id appears in the delete list.

What you get per chunk

Field	Meaning
id	Stable content-based id. Use it as your vector id
operation / changeType	upsert + added/updated/unchanged, or delete
text	Markdown of the chunk (headings, lists, tables and fenced code kept intact)
url / sourceUrl	Canonical URL (from <link rel=canonical> when same-origin) and the URL actually fetched
pageTitle, language	From the page
headingPath, headingPathString	Where the chunk sits, e.g. API clients > JavaScript API client
chunkIndex, chunkCount	Position within the page
tokenCount	Estimate (about 4 characters per token). Not a real tokenizer, expect roughly +/-15% against cl100k
contentHash, pageHash	SHA-256 prefixes for your own change tracking
metadata	Ready-made payload object for your vector DB (url, title, headingPath, language, contentHash...)

Sample output

{
  "type": "chunk", "operation": "upsert", "changeType": "added",
  "id": "1cade10765-d0c6204cead5",
  "url": "https://docs.apify.com/api",
  "pageTitle": "Apify Documentation", "language": "en",
  "headingPathString": "API clients > JavaScript API client",
  "chunkIndex": 3, "chunkCount": 7, "tokenCount": 119,
  "text": "### JavaScript API client\n\nFor web browser, JavaScript/TypeScript applications, Node.js, Deno, or Bun...",
  "contentHash": "d0c6204cead5b3e0",
  "metadata": { "url": "https://docs.apify.com/api", "headingPath": "API clients > JavaScript API client", "chunkIndex": 3 }
}

{ "type": "delete", "operation": "delete", "id": "d9c3495eed-f91590ecfee9",
  "url": "https://example.com/install", "reason": "content-changed" }

{ "type": "summary", "mode": "incremental", "pagesAdded": 1, "pagesChanged": 1, "pagesUnchanged": 1,
  "pagesRemoved": 1, "itemsUpserted": 2, "itemsDeleted": 3, "itemsUnchanged": 2,
  "itemsInIndexAfterSync": 4, "tokensUpserted": 93,
  "syncHint": "Upsert the \"upsert\" rows and delete the ids in the \"delete\" rows from your vector DB." }

The same changes are also written to the key-value store as CHANGES.jsonl (one JSON object per line, good for bulk loaders), optionally CHANGES.json / CHANGES.md, plus DELETED_IDS.json (just the ids).

How the incremental sync works

1. The site is crawled (sitemap first, including sitemaps named in robots.txt; otherwise links up to 3 levels deep).
2. Each page becomes markdown, is split on headings with a token budget, and near-duplicate chunks across the site are merged.
3. Every chunk gets its content-based id and is compared with the state saved by the last run (stored in a named key-value store, default website-to-rag-state, one record per site).
4. New ids are upserts, ids that disappeared are deletes. The state is saved after the output, so a run that dies half way is simply re-emitted next time (at-least-once).

Safety rules built in:

• A page that fails with a temporary error keeps its old state. Its chunks are never deleted because of a timeout.
• A page is deleted only when it returns 404/410, or when it is no longer in the sitemap (and that is only checked when the sitemap was used).
• Pages beyond maxPages are never treated as deleted. Lowering maxPages will not wipe your index.
• Changing chunk settings (targetTokens, overlapTokens...) changes the ids, so the next run re-emits everything and deletes the old ids. The summary says settingsChanged: true.

Input

Field	Default	What it does
siteUrl	https://docs.apify.com	Site or docs section. End with a folder (https://example.com/docs/) to crawl only that folder
maxPages	25	Pages fetched per run
onlyChanges	true	Later runs output only changes. Off = full export every time (deletes are still listed)
removeMissingPages	true	Emit deletes for pages that left the sitemap
resetState	false	Treat this run as the first one
outputUnit	chunks	chunks, or pages for one clean markdown row per page (still with change detection)
outputFormats	jsonl	Extra files in the key-value store: jsonl, json, markdown
includePatterns / excludePatterns	empty	URL substring filters
respectRobotsTxt	true	Honour Disallow for User-agent: *
targetTokens / maxTokens / overlapTokens / minChunkTokens	350 / 512 / 40 / 15	Chunk sizing
dedupThreshold, shingleSize	85, 5	Near-duplicate merging across the whole crawl
webhookUrl	empty	Slack/Discord/any webhook, called only when a re-run finds changes
stateStoreName	website-to-rag-state	Use a different name per project

Run with empty input {} to try it on the default docs site.

Recipes

All snippets use the Apify Python client. run_input can be your own site; schedule the Actor (daily or weekly) and run the sync step after each run.

from apify_client import ApifyClient
client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("that_red_bird/website-to-rag-incremental").call(run_input={"siteUrl": "https://docs.example.com"})
rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
upserts = [r for r in rows if r["type"] == "chunk"]
deletes = [r["id"] for r in rows if r["type"] == "delete"]
vectors = embed([r["text"] for r in upserts])   # your embedding model

Pinecone

index.upsert(vectors=[(r["id"], v, {**r["metadata"], "text": r["text"]}) for r, v in zip(upserts, vectors)])
if deletes: index.delete(ids=deletes)

Qdrant (point ids must be UUIDs or integers, so derive one from the chunk id)

import uuid
pid = lambda s: str(uuid.uuid5(uuid.NAMESPACE_URL, s))
client.upsert("docs", points=[PointStruct(id=pid(r["id"]), vector=v, payload={**r["metadata"], "text": r["text"], "chunk_id": r["id"]}) for r, v in zip(upserts, vectors)])
if deletes: client.delete("docs", points_selector=PointIdsList(points=[pid(i) for i in deletes]))

pgvector

INSERT INTO chunks (id, url, text, metadata, embedding) VALUES (%s, %s, %s, %s, %s)
ON CONFLICT (id) DO UPDATE SET text = EXCLUDED.text, metadata = EXCLUDED.metadata, embedding = EXCLUDED.embedding;
DELETE FROM chunks WHERE id = ANY(%s);

Weaviate (object uuid via generate_uuid5(chunk_id)): batch-insert the upserts with properties={**metadata, "text": text} and collection.data.delete_by_id(generate_uuid5(i)) for each delete.

LangChain

vectorstore.add_documents([Document(page_content=r["text"], metadata=r["metadata"]) for r in upserts], ids=[r["id"] for r in upserts])
if deletes: vectorstore.delete(ids=deletes)

LlamaIndex: build TextNode(id_=r["id"], text=r["text"], metadata=r["metadata"]) for each upsert, index.insert_nodes(nodes), and remove deletes with index.delete_nodes(deletes) (needs a vector store that supports node deletion).

These are sketches against current client libraries; check argument names for the version you use.

Pricing

Pay per event, no subscription:

Event	Price	When
Site crawled	$0.01	once per run
Page processed	$0.002	per page fetched and converted
Chunk produced	$0.002	per chunk output (unchanged chunks on a re-run are free unless you turn off onlyChanges)

A 100-page docs site is roughly $2 for the first run (about 950 chunks). A weekly re-run where 5 pages changed is roughly $0.25 (100 pages fetched + a few dozen changed chunks). Deletes, errors and the summary are not charged. Pages that fail to load are not charged.

Use cases

• Keep a support bot / docs assistant index in sync with your product docs without re-embedding everything.
• Feed an AI agent or MCP server with a knowledge base that updates itself on a schedule.
• Build a competitor or regulation knowledge base and get a precise list of what changed.
• Export a site as per-page markdown (outputUnit: pages) for fine-tuning data or a static search index.

FAQ

Does it create embeddings? No. It outputs chunks, metadata and ids. Use any embedding model.

Does it run JavaScript? No. It reads server-rendered HTML. Pages that are empty shells without JavaScript produce little or no text.

How does it detect changes? By hashing the final chunk text. Cosmetic changes that alter the text (a reworded sentence) count as changes; changes to the page chrome (menus, footers) do not, because only the main content is extracted.

Why did I get more upserts than I expected after editing one paragraph? Chunk overlap copies the tail of a chunk into the next one, so an edit can change the following chunk too. Set overlapTokens to 0 for the tightest diffs.

What if my sitemap is huge? Set maxPages and includePatterns. Pages outside the limit are left alone, never deleted.

Is robots.txt respected? Yes by default (User-agent: * rules). It does not read rate-limit hints.

Can I run several sites? Yes, run the Actor once per site. State is kept per site and per output unit in the state store.

Integrations

Works with Make, Zapier, n8n, webhooks, the Apify API/clients and Apify's MCP server so an agent can call it. Use webhookUrl for a Slack or Discord ping when a re-run finds changes, or Apify's own run-succeeded webhooks to trigger your sync job.

If this Actor saves you time, a review on the Store page helps others find it. Questions or a site that does not crawl well? Open an issue on the Actor page.