RAG Sync Monitor: Changed Chunks Only
Pricing
from $0.50 / 1,000 page checks
Incremental RAG sync - Keep your vector DB in sync with any docs site, help center or PDFs. Outputs only the chunks added, changed or removed since the last run, with stable IDs for upserts and deletes. Pinecone, Qdrant, Chroma, pgvector.
Pages to start crawling from, e.g. the root of a docs site. Only pages under the URL prefixes are crawled.
Optional sitemap.xml or sitemap index URLs. Pages listed in them are added to the crawl.
Also load sitemaps listed in each start host's robots.txt (or /sitemap.xml), so pages that aren't linked from navigation are found too.
Only URLs starting with one of these are crawled. Defaults to the directory of each start URL.
Regular expressions; matching URLs are skipped, e.g. "/changelog/" or "\?page=".
If set, a URL must also match one of these regular expressions to be synced, e.g. "/guides/" or "/api/v2/".
Discover new pages by following in-scope links. Turn off to sync only start URLs and sitemap pages.
How many links away from the start URLs to follow. 0 = only the start URLs (and sitemap pages). Empty = no limit.
Upper limit on pages checked in one run. The form starts at 25 so a first try is quick and cheap; raise it for a full sync (the default when called via API is 200). Unchanged pages are cheap to check thanks to conditional requests.
Name of the saved baseline that runs are compared against. Defaults to the first start URL. Use a different name per site or per vector index. Change it to start a fresh baseline.
On the first run for a state name, output every chunk as an upsert. Turn off if the site is already indexed and you only want future changes.
Chunks follow headings and are packed by paragraph up to this size. Changing it re-chunks every page on the next run.
Characters repeated from the previous chunk within the same section.
Chunks: changed chunks are output. Pages: each changed page is output once as full Markdown, with a stable ID per URL.
Optional CSS selector for the main content, e.g. "main .markdown". Defaults to main, article or body.
Extra elements to strip before chunking, e.g. ".sidebar", ".edit-this-page". Nav, header, footer and scripts are always removed.
Learns lines that appear on most pages of the site (menus, footers, banners, rotating sponsor lists) and removes them from chunks. Headings and code are never removed. When new lines are learned, the next run re-downloads every page once to apply the cleanup.
For sites that build their content in the browser (some help centers and single-page apps). Auto renders only pages with too little static text. Pages that need the browser are charged an extra 'page rendered' event. Use 1024 MB of run memory.
Browser only: how long to wait for the page to finish loading content.
Browser only: wait until this element appears before extracting, e.g. "article".
Also sync linked PDF and Word files found on the same host. PDF chunks are grouped by page ("Page 3") for citations. Scanned PDFs without a text layer produce no text.
If a page declares another URL as canonical (rel=canonical), sync that URL instead, which removes mirrored and duplicate pages.
For hash-routed apps (URLs like /#/guide). Plain #section anchors are always ignored. Usually combined with Render JavaScript.
Parallel requests. Keep it low to be polite to the site.
Use Apify Proxy (or your own) for sites that block cloud IP addresses.
Extra headers sent with every request to the crawled hosts only, e.g. {"Authorization": "Bearer ..."} for docs behind a login. Stored encrypted.
Re-download every page even if the server says it is unchanged. Output is still only real chunk changes.
Output what would change without updating the saved baseline, so the next run reports the same changes. Useful for testing a pipeline.
