RAG Sync Monitor: Changed Chunks Only

Keep your vector database in sync with a website, without re-embedding the whole site every time.

RAG Sync Monitor crawls a docs site, help center, knowledge base or linked PDFs and outputs only the chunks that were added, changed or removed since the last run. Each chunk has a stable ID, so you apply the output directly as upsert and delete operations in any vector database: Pinecone, Qdrant, Weaviate, Chroma, pgvector, Elasticsearch, or your own pipeline.

• ✅ Only what changed: new and edited chunks as upserts, removed sections and pages as deletes
• ✅ Cheap to run on a schedule: unchanged pages are skipped, and a quiet site costs only the page checks
• ✅ Ready for citations: every chunk keeps its URL, page title and section heading

Quickstart

1. Try it (2 minutes). Paste your docs URL into Start URLs, keep Max pages at 25 for a first try, and click Start. When the run finishes, open its Output tab: every row is an upsert with the chunk text, its URL and its section. Run it again and you'll see only what changed, which on a quiet site is usually nothing. A first try like this costs about $0.08.

2. Sync your vector database. Ready-to-run scripts for Chroma, Pinecone, Qdrant and Postgres + pgvector (free local embeddings, a nightly scheduled-job template) and an n8n workflow are in the open-source examples repo: rag-sync-examples

3. Or call it from your own code:

from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("cipherwise/rag-sync-monitor").call(run_input={
    "startUrls": [{"url": "https://docs.example.com/"}],
    "stateKey": "example-docs",   # keep the same name for every run of this site
})
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item["op"], item["chunk_id"], item["url"])   # apply deletes, then upserts

See Apply the changes to your vector database for a complete Chroma example. The same pattern works for Pinecone, Qdrant and pgvector: delete by ID, then upsert.

4. Automate it. Create an Apify Schedule (daily is typical) with the same input and stateKey, and connect the output to your pipeline with a webhook, n8n, Make or Zapier.

Common setups

Copy one of these into the JSON view of the input and change the URL.

Daily sync of a docs site

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "stateKey": "example-docs",
  "maxPages": 500
}

Docs plus the PDFs and Word files they link to (chunked per page, so answers can cite "Page 3")

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "stateKey": "example-docs-with-files",
  "includeFiles": ["pdf", "docx"]
}

A help center that builds its pages with JavaScript (set run memory to 1024 MB)

{
  "startUrls": [{ "url": "https://help.example.com/" }],
  "stateKey": "example-help-center",
  "renderJavaScript": "auto"
}

Moving an existing sync to a new schedule or state name: record the baseline without re-sending everything (only when your index already holds this Actor's chunks)

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "stateKey": "example-docs",
  "emitBaseline": false
}

Whole pages for your own text splitter (one stable ID per URL, re-sent when the page changes)

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "stateKey": "example-docs-pages",
  "granularity": "page"
}

Only one version of versioned docs, found through the sitemap

{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "stateKey": "example-docs-latest",
  "useSitemaps": true,
  "excludeUrlPatterns": ["/docs/\\d", "/docs/next/"]
}

Docs behind a login (header and cookie fields are stored encrypted)

{
  "startUrls": [{ "url": "https://internal-docs.example.com/" }],
  "stateKey": "internal-docs",
  "customHttpHeaders": { "Authorization": "Bearer <token>" }
}

Why use it

• Stop paying to re-embed unchanged content. Typical docs sites change a few pages a week. Re-crawling and re-embedding everything wastes embedding spend and indexing time.
• No stale or deleted answers. Pages that disappear (404/410) or sections that are removed produce delete records, so your chatbot stops citing content that no longer exists.
• Chunk-level, not page-level. If one section of a long page changes, you get that one chunk, not the whole page.
• Cheap repeat runs. Uses ETag / Last-Modified conditional requests, so unchanged pages usually aren't even downloaded.
• Vector-DB agnostic. Plain JSON output. Works with LangChain, LlamaIndex, n8n, Make, Zapier or a 20-line script.
• Pages, JavaScript apps and documents. Static HTML by default; a headless browser for JavaScript-built pages when you need it; linked PDF and Word files, chunked by page for citations.
• Docs behind a login. Custom headers and cookies (stored encrypted) are sent only to the site you crawl, never to third parties.

Compared with other approaches

	General crawler	Page-level sync	RAG Sync Monitor
Output each run	Every page	Every changed page, in full	Only added, changed and removed chunks
One paragraph edited on a long page	Whole site again	Whole page again	That one chunk
Deleted pages and sections	You diff to notice	Varies by tool	delete records with stable IDs
Unchanged pages	Downloaded and exported again	Usually downloaded again	Skipped via ETag/Last-Modified (HTTP 304)
Chunking	Your job	Varies	Heading-aware chunks, or whole pages (granularity: "page")

Pricing

You pay per event; there's no monthly fee.

Event	Price	When it's charged
Page check	$0.0005 ($0.50 per 1,000)	Each page or file checked, including unchanged pages (HTTP 304) and removed pages. Redirects, duplicate (canonical) pages, failed requests and non-HTML links aren't charged.
Chunk change	$0.0002 ($0.20 per 1,000)	Each upsert or delete record output. Unchanged chunks are never output or charged.
Browser render	$0.003 ($3 per 1,000)	Only when renderJavaScript is on and a page actually needed the headless browser, on top of the page check.
Actor start	$0.00005	Once per run (Apify's standard start fee).

What typical runs cost (a typical docs page splits into 10–15 chunks):

Run	Approximate cost
First try: 25 pages	about $0.08
First sync of a 500-page docs site (about 6,000 chunks)	about $1.45
Daily re-sync of that site, 20 chunks changed	about $0.25 per run, or about $7.60 per month
Daily re-sync when nothing changed	about $0.25 per run (page checks only)
First sync of a 100-page JavaScript help center (about 1,000 chunks)	about $0.55
A 50-page PDF (about 60 chunks)	about $0.01

Set a maximum cost per run in the run options. The Actor stops cleanly at the limit, never outputs half a page's changes, and picks up the rest on the next run.

If maxPages is smaller than the site, each run checks a different slice: newly found pages and the least recently checked known pages go first, so the whole site is covered over a few runs.

Output

Each dataset item is one operation to apply.

Field	In	Meaning
op	all	upsert or delete
chunk_id	all	Stable ID: use it as the vector ID in your database
url	all	Source page or file
text	upsert	The chunk's Markdown (a section's first chunk starts with its heading)
title	upsert	Page or document title
heading_path	upsert	Section path, e.g. Install > Configuration (PDFs: Document > Page 3)
chunk_index	upsert	Position of the chunk on its page
page_status	upsert	new (page seen for the first time) or changed
source_type	upsert	html, pdf or docx
description, language	upsert	The page's meta description and language, when present
reason	delete	removed_from_page, page_gone (404/410), redirected or duplicate_of_canonical
synced_at	all	When the change was detected

{
  "op": "upsert",
  "chunk_id": "037ec235493f055422c37060",
  "url": "https://docs.example.com/guide/install",
  "title": "Install | Example Docs",
  "heading_path": "Install > Configuration",
  "chunk_index": 3,
  "text": "## Configuration\n\nSet the API key ...",
  "page_status": "changed",
  "source_type": "html",
  "description": "How to install Example on Linux, macOS and Windows.",
  "language": "en",
  "synced_at": "2026-10-18T13:47:16+00:00"
}

{
  "op": "delete",
  "chunk_id": "a91f0c2b77e4d1903b6e5c11",
  "url": "https://docs.example.com/guide/old-page",
  "reason": "page_gone",
  "synced_at": "2026-10-18T13:47:16+00:00"
}

Each run also saves a summary to the OUTPUT record of its key-value store: pages checked, new, changed, unchanged, gone and rendered, chunk upserts and deletes, chunksUnchanged (chunks you didn't have to re-embed), pages still pending, warnings and errors.

Apply the changes to your vector database

Python example with Chroma and apify-client 3.x (the same pattern works for any database with upsert and delete by ID):

from apify_client import ApifyClient
import chromadb

apify = ApifyClient("<APIFY_TOKEN>")
run = apify.actor("cipherwise/rag-sync-monitor").call(run_input={
    "startUrls": [{"url": "https://docs.example.com/"}],
    "stateKey": "example-docs",
})

collection = chromadb.PersistentClient("./db").get_or_create_collection("docs")
upserts, deletes = [], []
for item in apify.dataset(run.default_dataset_id).iterate_items():
    (upserts if item["op"] == "upsert" else deletes).append(item)

if deletes:
    collection.delete(ids=[d["chunk_id"] for d in deletes])
if upserts:
    collection.upsert(
        ids=[u["chunk_id"] for u in upserts],
        documents=[u["text"] for u in upserts],
        metadatas=[{"url": u["url"], "title": u["title"], "section": u["heading_path"]} for u in upserts],
    )

Apply deletes before upserts. Both operations are idempotent, so re-applying a run's output is safe. Complete, tested scripts for Pinecone, Qdrant and pgvector, plus an n8n workflow, are in rag-sync-examples.

How it works

1. First run (baseline): crawls the site, splits each page into chunks along its headings, and outputs every chunk as an upsert. The chunk IDs are saved under a sync state name (stateKey).
2. Every later run: re-checks known pages and discovers new ones, re-chunks the pages that changed, and compares the chunk IDs with the saved state:
   • new or edited chunk → upsert
   • chunk that no longer exists on the page → delete (reason: removed_from_page)
   • page that returns 404/410 → delete for all its chunks (reason: page_gone)
   • page that now redirects → its old chunks are deleted (reason: redirected) and the target page is synced
3. Unchanged chunks are not output.

Chunk IDs are derived from the URL, the heading path and the chunk text, so the same content always gets the same ID.

Reliability and safety

• Progress is saved as the run goes, including before Apify moves a run to another server, so a restarted run doesn't send or charge for the same changes twice.
• The spending limit never leaves a page half-synced. Pages that didn't fit are picked up on the next run.
• Temporary errors (timeouts, 429, 5xx) are retried up to 3 times. Pages that still fail keep their previous state and are retried next run; nothing is deleted because of an error.
• Repeatable output: tested on about 40 documentation platforms to confirm that re-downloading an unchanged site produces zero changes.
• Polite crawling: robots.txt is respected by default, concurrency is capped, and requests identify themselves as RAGSyncMonitor.
• Your data stays yours. Content is never sent to an AI or embedding service; you embed it yourself. Login headers and cookies are stored encrypted and sent only to the site you crawl.
• Test safely: dryRun shows what would change without updating the saved baseline.

FAQ

If someone inserts a paragraph near the top of a page, does every chunk after it change? No. IDs come from the content and its section, not its position. Only the edited chunk changes. A very large insertion into a long section can re-chunk the rest of that section, but never other sections.

What happens when a page moves to a new URL? If the old URL redirects, its chunks are deleted and the page is synced at its new address. If it just disappears (404), its chunks are deleted. Either way your index stays correct; the moved content is embedded once at its new URL.

How do I sync several sites, or several indexes of one site? Give each one its own stateKey. Each name keeps its own baseline.

My site is already in my vector database. Will the first run re-send everything? Yes, if it was indexed with another tool: its chunk IDs differ, so do a normal first run into a fresh collection and switch over. If the index already holds this Actor's chunks (for example, you're moving to a new stateKey or schedule), set emitBaseline to false: the first run records the baseline silently, and later runs send only changes.

How do I start over? Use a new stateKey and a fresh index. Do the same if you change chunkSize, chunkOverlap or the content selectors, since those change how pages are chunked.

Can I use my own chunker? Yes: set granularity to page to get each changed page as one Markdown document with a stable ID per URL.

Does it work on JavaScript-heavy sites? Set renderJavaScript to auto and run with 1024 MB of memory. Only pages with too little static text use the browser and are charged the browser-render fee; once three pages on a site needed it, the whole site is rendered.

What does a run cost when nothing changed? Only the page checks ($0.50 per 1,000 pages). No chunks are output, so there's nothing to embed.

Input reference

Field	Description	Default
startUrls	Where to start crawling, e.g. https://docs.example.com/	required (or sitemapUrls)
sitemapUrls	Optional sitemap.xml / sitemap index URLs	–
useSitemaps	Also load sitemaps from robots.txt or /sitemap.xml	false
urlPrefixes	Only crawl URLs starting with these	directory of each start URL
includeUrlPatterns	If set, URLs must also match one of these regexes	–
excludeUrlPatterns	Regexes for URLs to skip	–
crawlLinks	Follow in-scope links to find new pages	true
maxCrawlDepth	Links to follow from the start URLs (0 = start URLs only)	no limit
maxPages	Max pages checked per run	200 (the form starts at 25)
stateKey	Name of the saved baseline. Use one per site / per index	derived from first start URL
emitBaseline	Output all chunks on the first run. Turn off only if your index already holds this Actor's chunks	true
chunkSize / chunkOverlap	Max characters per chunk and overlap within a section	1500 / 150
granularity	chunk (changed chunks) or page (changed pages as full Markdown, one stable ID per URL)	chunk
contentSelector	CSS selector for the main content	main, article or body
removeSelectors	Extra CSS selectors to strip (e.g. .sidebar, .edit-page)	–
removeRepeatedText	Learn lines repeated on most pages (menus, footers, banners, rotating sponsor lists) and remove them. Learned once, then frozen; the run after learning cleans up already-synced pages once	false
renderJavaScript	never, auto (browser only for pages with too little static text; a whole site switches to the browser after 3 such pages) or always. Use 1024 MB memory for a browser	never
dynamicContentWaitSecs / waitForSelector	Browser only: how long / for what to wait before extracting	10 / –
includeFiles	Sync linked pdf and/or docx files on the same host	–
useCanonicalUrls	Sync the page's rel=canonical URL instead of duplicates	true
keepUrlFragments	Treat #/route URLs as separate pages (hash-routed apps)	false
maxConcurrency	Parallel requests	5
proxyConfiguration	Apify Proxy or your own proxies	none
customHttpHeaders / cookies	Sent only to the crawled hosts, stored encrypted	–
forceRefetch	Ignore ETag/Last-Modified and re-download everything	false
dryRun	Output the changes without saving the baseline	false
respectRobotsTxt	Skip pages disallowed by robots.txt	true

Tested on

Docusaurus, MkDocs / Material for MkDocs, Sphinx and Read the Docs, GitBook, Mintlify, VitePress, Astro Starlight, Hugo, Next.js docs, large developer-platform and API docs, MDN, react.dev, Intercom help centers and Wikipedia, including non-English (Japanese) docs.

Also tested: mdBook, Django, Kubernetes, Microsoft Learn, AWS docs, Cloudflare, Supabase, Tailwind, Vue, Laravel, PostgreSQL, GOV.UK, WordPress, Ghost, LangChain and Anthropic docs, hash-routed Docsify apps and JavaScript help centers (with renderJavaScript), and PDF/DOCX files.

Limits

• Pages are fetched as plain HTML by default (fast and cheap). JavaScript-built sites need renderJavaScript and 1024 MB of memory (pages render 2 at a time; more memory renders up to 4 at a time). Rendered pages are re-rendered on every run, since their HTML shell can stay the same while the content changes.
• Scanned PDFs without a text layer produce no text (no OCR). Files larger than 30 MB and pages larger than 10 MB are skipped.
• Sync state is stored (compressed) in your account's rag-sync-monitor-state key-value store. One state name comfortably handles tens of thousands of pages; split very large sites with urlPrefixes and separate state names.
• Versioned or translated docs (e.g. /docs/3.2.1/…, /docs/next/…, /zh-CN/…) publish many copies of the same page. Index only the version you want with excludeUrlPatterns, e.g. ["/docs/\\d", "/docs/next/", "^https://[^/]+/[a-z]{2}(-[A-Z]{2})?/"].
• Only crawl sites you are allowed to crawl. The Actor respects robots.txt by default.

Feedback and support

Found a site that doesn't extract cleanly, or need a feature (moving content between URLs without re-embedding, direct Pinecone/Qdrant writes, more file types)? Open an issue on the Issues tab.

If RAG Sync Monitor saves you embedding costs, a short review on this page helps other people find it.