# RAG Sync Monitor: Changed Chunks Only (`cipherwise/rag-sync-monitor`) Actor

Incremental RAG sync - Keep your vector DB in sync with any docs site, help center or PDFs. Outputs only the chunks added, changed or removed since the last run, with stable IDs for upserts and deletes. Pinecone, Qdrant, Chroma, pgvector.

- **URL**: https://apify.com/cipherwise/rag-sync-monitor.md
- **Developed by:** [Cipher Wise](https://apify.com/cipherwise) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.50 / 1,000 page checks

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RAG Sync Monitor: Changed Chunks Only

**Keep your vector database in sync with a website, without re-embedding the whole site every time.**

RAG Sync Monitor crawls a docs site, help center, knowledge base or linked PDFs and outputs **only the chunks that were added, changed or removed since the last run**. Each chunk has a stable ID, so you apply the output directly as `upsert` and `delete` operations in any vector database: Pinecone, Qdrant, Weaviate, Chroma, pgvector, Elasticsearch, or your own pipeline.

- ✅ **Only what changed:** new and edited chunks as upserts, removed sections and pages as deletes
- ✅ **Cheap to run on a schedule:** unchanged pages are skipped, and a quiet site costs only the page checks
- ✅ **Ready for citations:** every chunk keeps its URL, page title and section heading

### Quickstart

**1. Try it (2 minutes).** Paste your docs URL into **Start URLs**, keep **Max pages** at 25 for a first try, and click **Start**. When the run finishes, open its **Output** tab: every row is an `upsert` with the chunk text, its URL and its section. Run it again and you'll see only what changed, which on a quiet site is usually nothing. A first try like this costs about $0.08.

**2. Sync your vector database.** Ready-to-run scripts for **Chroma, Pinecone, Qdrant and Postgres + pgvector** (free local embeddings, a nightly scheduled-job template) and an **n8n workflow** are in the open-source examples repo: [rag-sync-examples](https://github.com/Arjun-M-291297/rag-sync-examples)

**3. Or call it from your own code:**

```python
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("cipherwise/rag-sync-monitor").call(run_input={
    "startUrls": [{"url": "https://docs.example.com/"}],
    "stateKey": "example-docs",   # keep the same name for every run of this site
})
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item["op"], item["chunk_id"], item["url"])   # apply deletes, then upserts
```

See [Apply the changes to your vector database](#apply-the-changes-to-your-vector-database) for a complete Chroma example. The same pattern works for Pinecone, Qdrant and pgvector: delete by ID, then upsert.

**4. Automate it.** Create an Apify **Schedule** (daily is typical) with the same input and `stateKey`, and connect the output to your pipeline with a webhook, n8n, Make or Zapier.

### Common setups

Copy one of these into the **JSON** view of the input and change the URL.

**Daily sync of a docs site**

```json
{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "stateKey": "example-docs",
  "maxPages": 500
}
```

**Docs plus the PDFs and Word files they link to** (chunked per page, so answers can cite "Page 3")

```json
{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "stateKey": "example-docs-with-files",
  "includeFiles": ["pdf", "docx"]
}
```

**A help center that builds its pages with JavaScript** (set run memory to 1024 MB)

```json
{
  "startUrls": [{ "url": "https://help.example.com/" }],
  "stateKey": "example-help-center",
  "renderJavaScript": "auto"
}
```

**Moving an existing sync to a new schedule or state name:** record the baseline without re-sending everything (only when your index already holds this Actor's chunks)

```json
{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "stateKey": "example-docs",
  "emitBaseline": false
}
```

**Whole pages for your own text splitter** (one stable ID per URL, re-sent when the page changes)

```json
{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "stateKey": "example-docs-pages",
  "granularity": "page"
}
```

**Only one version of versioned docs, found through the sitemap**

```json
{
  "startUrls": [{ "url": "https://docs.example.com/" }],
  "stateKey": "example-docs-latest",
  "useSitemaps": true,
  "excludeUrlPatterns": ["/docs/\\d", "/docs/next/"]
}
```

**Docs behind a login** (header and cookie fields are stored encrypted)

```json
{
  "startUrls": [{ "url": "https://internal-docs.example.com/" }],
  "stateKey": "internal-docs",
  "customHttpHeaders": { "Authorization": "Bearer <token>" }
}
```

### Why use it

- **Stop paying to re-embed unchanged content.** Typical docs sites change a few pages a week. Re-crawling and re-embedding everything wastes embedding spend and indexing time.
- **No stale or deleted answers.** Pages that disappear (404/410) or sections that are removed produce `delete` records, so your chatbot stops citing content that no longer exists.
- **Chunk-level, not page-level.** If one section of a long page changes, you get that one chunk, not the whole page.
- **Cheap repeat runs.** Uses `ETag` / `Last-Modified` conditional requests, so unchanged pages usually aren't even downloaded.
- **Vector-DB agnostic.** Plain JSON output. Works with LangChain, LlamaIndex, n8n, Make, Zapier or a 20-line script.
- **Pages, JavaScript apps and documents.** Static HTML by default; a headless browser for JavaScript-built pages when you need it; linked PDF and Word files, chunked by page for citations.
- **Docs behind a login.** Custom headers and cookies (stored encrypted) are sent only to the site you crawl, never to third parties.

### Compared with other approaches

| | General crawler | Page-level sync | RAG Sync Monitor |
|---|---|---|---|
| Output each run | Every page | Every changed page, in full | Only added, changed and removed **chunks** |
| One paragraph edited on a long page | Whole site again | Whole page again | That one chunk |
| Deleted pages and sections | You diff to notice | Varies by tool | `delete` records with stable IDs |
| Unchanged pages | Downloaded and exported again | Usually downloaded again | Skipped via ETag/Last-Modified (HTTP 304) |
| Chunking | Your job | Varies | Heading-aware chunks, or whole pages (`granularity: "page"`) |

### Pricing

You pay per event; there's no monthly fee.

| Event | Price | When it's charged |
|---|---|---|
| Page check | $0.0005 ($0.50 per 1,000) | Each page or file checked, including unchanged pages (HTTP 304) and removed pages. Redirects, duplicate (canonical) pages, failed requests and non-HTML links aren't charged. |
| Chunk change | $0.0002 ($0.20 per 1,000) | Each `upsert` or `delete` record output. Unchanged chunks are never output or charged. |
| Browser render | $0.003 ($3 per 1,000) | Only when `renderJavaScript` is on and a page actually needed the headless browser, on top of the page check. |
| Actor start | $0.00005 | Once per run (Apify's standard start fee). |

**What typical runs cost** (a typical docs page splits into 10–15 chunks):

| Run | Approximate cost |
|---|---|
| First try: 25 pages | about $0.08 |
| First sync of a 500-page docs site (about 6,000 chunks) | about $1.45 |
| Daily re-sync of that site, 20 chunks changed | about $0.25 per run, or about $7.60 per month |
| Daily re-sync when nothing changed | about $0.25 per run (page checks only) |
| First sync of a 100-page JavaScript help center (about 1,000 chunks) | about $0.55 |
| A 50-page PDF (about 60 chunks) | about $0.01 |

Set a **maximum cost per run** in the run options. The Actor stops cleanly at the limit, never outputs half a page's changes, and picks up the rest on the next run.

If `maxPages` is smaller than the site, each run checks a different slice: newly found pages and the least recently checked known pages go first, so the whole site is covered over a few runs.

### Output

Each dataset item is one operation to apply.

| Field | In | Meaning |
|---|---|---|
| `op` | all | `upsert` or `delete` |
| `chunk_id` | all | Stable ID: use it as the vector ID in your database |
| `url` | all | Source page or file |
| `text` | upsert | The chunk's Markdown (a section's first chunk starts with its heading) |
| `title` | upsert | Page or document title |
| `heading_path` | upsert | Section path, e.g. `Install > Configuration` (PDFs: `Document > Page 3`) |
| `chunk_index` | upsert | Position of the chunk on its page |
| `page_status` | upsert | `new` (page seen for the first time) or `changed` |
| `source_type` | upsert | `html`, `pdf` or `docx` |
| `description`, `language` | upsert | The page's meta description and language, when present |
| `reason` | delete | `removed_from_page`, `page_gone` (404/410), `redirected` or `duplicate_of_canonical` |
| `synced_at` | all | When the change was detected |

```json
{
  "op": "upsert",
  "chunk_id": "037ec235493f055422c37060",
  "url": "https://docs.example.com/guide/install",
  "title": "Install | Example Docs",
  "heading_path": "Install > Configuration",
  "chunk_index": 3,
  "text": "## Configuration\n\nSet the API key ...",
  "page_status": "changed",
  "source_type": "html",
  "description": "How to install Example on Linux, macOS and Windows.",
  "language": "en",
  "synced_at": "2026-10-18T13:47:16+00:00"
}
```

```json
{
  "op": "delete",
  "chunk_id": "a91f0c2b77e4d1903b6e5c11",
  "url": "https://docs.example.com/guide/old-page",
  "reason": "page_gone",
  "synced_at": "2026-10-18T13:47:16+00:00"
}
```

Each run also saves a summary to the `OUTPUT` record of its key-value store: pages checked, new, changed, unchanged, gone and rendered, chunk upserts and deletes, `chunksUnchanged` (chunks you didn't have to re-embed), pages still pending, warnings and errors.

### Apply the changes to your vector database

Python example with Chroma and `apify-client` 3.x (the same pattern works for any database with upsert and delete by ID):

```python
from apify_client import ApifyClient
import chromadb

apify = ApifyClient("<APIFY_TOKEN>")
run = apify.actor("cipherwise/rag-sync-monitor").call(run_input={
    "startUrls": [{"url": "https://docs.example.com/"}],
    "stateKey": "example-docs",
})

collection = chromadb.PersistentClient("./db").get_or_create_collection("docs")
upserts, deletes = [], []
for item in apify.dataset(run.default_dataset_id).iterate_items():
    (upserts if item["op"] == "upsert" else deletes).append(item)

if deletes:
    collection.delete(ids=[d["chunk_id"] for d in deletes])
if upserts:
    collection.upsert(
        ids=[u["chunk_id"] for u in upserts],
        documents=[u["text"] for u in upserts],
        metadatas=[{"url": u["url"], "title": u["title"], "section": u["heading_path"]} for u in upserts],
    )
```

Apply deletes before upserts. Both operations are idempotent, so re-applying a run's output is safe. Complete, tested scripts for Pinecone, Qdrant and pgvector, plus an n8n workflow, are in [rag-sync-examples](https://github.com/Arjun-M-291297/rag-sync-examples).

### How it works

1. **First run (baseline):** crawls the site, splits each page into chunks along its headings, and outputs every chunk as an `upsert`. The chunk IDs are saved under a *sync state name* (`stateKey`).
2. **Every later run:** re-checks known pages and discovers new ones, re-chunks the pages that changed, and compares the chunk IDs with the saved state:
   - new or edited chunk → `upsert`
   - chunk that no longer exists on the page → `delete` (`reason: removed_from_page`)
   - page that returns 404/410 → `delete` for all its chunks (`reason: page_gone`)
   - page that now redirects → its old chunks are deleted (`reason: redirected`) and the target page is synced
3. Unchanged chunks are not output.

Chunk IDs are derived from the URL, the heading path and the chunk text, so the same content always gets the same ID.

### Reliability and safety

- **Progress is saved as the run goes**, including before Apify moves a run to another server, so a restarted run doesn't send or charge for the same changes twice.
- **The spending limit never leaves a page half-synced.** Pages that didn't fit are picked up on the next run.
- **Temporary errors** (timeouts, 429, 5xx) are retried up to 3 times. Pages that still fail keep their previous state and are retried next run; nothing is deleted because of an error.
- **Repeatable output:** tested on about 40 documentation platforms to confirm that re-downloading an unchanged site produces zero changes.
- **Polite crawling:** robots.txt is respected by default, concurrency is capped, and requests identify themselves as `RAGSyncMonitor`.
- **Your data stays yours.** Content is never sent to an AI or embedding service; you embed it yourself. Login headers and cookies are stored encrypted and sent only to the site you crawl.
- **Test safely:** `dryRun` shows what would change without updating the saved baseline.

### FAQ

**If someone inserts a paragraph near the top of a page, does every chunk after it change?**
No. IDs come from the content and its section, not its position. Only the edited chunk changes. A very large insertion into a long section can re-chunk the rest of that section, but never other sections.

**What happens when a page moves to a new URL?**
If the old URL redirects, its chunks are deleted and the page is synced at its new address. If it just disappears (404), its chunks are deleted. Either way your index stays correct; the moved content is embedded once at its new URL.

**How do I sync several sites, or several indexes of one site?**
Give each one its own `stateKey`. Each name keeps its own baseline.

**My site is already in my vector database. Will the first run re-send everything?**
Yes, if it was indexed with another tool: its chunk IDs differ, so do a normal first run into a fresh collection and switch over. If the index already holds this Actor's chunks (for example, you're moving to a new `stateKey` or schedule), set `emitBaseline` to `false`: the first run records the baseline silently, and later runs send only changes.

**How do I start over?**
Use a new `stateKey` and a fresh index. Do the same if you change `chunkSize`, `chunkOverlap` or the content selectors, since those change how pages are chunked.

**Can I use my own chunker?**
Yes: set `granularity` to `page` to get each changed page as one Markdown document with a stable ID per URL.

**Does it work on JavaScript-heavy sites?**
Set `renderJavaScript` to `auto` and run with 1024 MB of memory. Only pages with too little static text use the browser and are charged the browser-render fee; once three pages on a site needed it, the whole site is rendered.

**What does a run cost when nothing changed?**
Only the page checks ($0.50 per 1,000 pages). No chunks are output, so there's nothing to embed.

### Input reference

| Field | Description | Default |
|---|---|---|
| `startUrls` | Where to start crawling, e.g. `https://docs.example.com/` | required (or `sitemapUrls`) |
| `sitemapUrls` | Optional sitemap.xml / sitemap index URLs | – |
| `useSitemaps` | Also load sitemaps from robots.txt or `/sitemap.xml` | `false` |
| `urlPrefixes` | Only crawl URLs starting with these | directory of each start URL |
| `includeUrlPatterns` | If set, URLs must also match one of these regexes | – |
| `excludeUrlPatterns` | Regexes for URLs to skip | – |
| `crawlLinks` | Follow in-scope links to find new pages | `true` |
| `maxCrawlDepth` | Links to follow from the start URLs (0 = start URLs only) | no limit |
| `maxPages` | Max pages checked per run | `200` (the form starts at 25) |
| `stateKey` | Name of the saved baseline. Use one per site / per index | derived from first start URL |
| `emitBaseline` | Output all chunks on the first run. Turn off only if your index already holds this Actor's chunks | `true` |
| `chunkSize` / `chunkOverlap` | Max characters per chunk and overlap within a section | `1500` / `150` |
| `granularity` | `chunk` (changed chunks) or `page` (changed pages as full Markdown, one stable ID per URL) | `chunk` |
| `contentSelector` | CSS selector for the main content | `main`, `article` or `body` |
| `removeSelectors` | Extra CSS selectors to strip (e.g. `.sidebar`, `.edit-page`) | – |
| `removeRepeatedText` | Learn lines repeated on most pages (menus, footers, banners, rotating sponsor lists) and remove them. Learned once, then frozen; the run after learning cleans up already-synced pages once | `false` |
| `renderJavaScript` | `never`, `auto` (browser only for pages with too little static text; a whole site switches to the browser after 3 such pages) or `always`. Use 1024 MB memory for a browser | `never` |
| `dynamicContentWaitSecs` / `waitForSelector` | Browser only: how long / for what to wait before extracting | `10` / – |
| `includeFiles` | Sync linked `pdf` and/or `docx` files on the same host | – |
| `useCanonicalUrls` | Sync the page's `rel=canonical` URL instead of duplicates | `true` |
| `keepUrlFragments` | Treat `#/route` URLs as separate pages (hash-routed apps) | `false` |
| `maxConcurrency` | Parallel requests | `5` |
| `proxyConfiguration` | Apify Proxy or your own proxies | none |
| `customHttpHeaders` / `cookies` | Sent only to the crawled hosts, stored encrypted | – |
| `forceRefetch` | Ignore ETag/Last-Modified and re-download everything | `false` |
| `dryRun` | Output the changes without saving the baseline | `false` |
| `respectRobotsTxt` | Skip pages disallowed by robots.txt | `true` |

### Tested on

Docusaurus, MkDocs / Material for MkDocs, Sphinx and Read the Docs, GitBook, Mintlify, VitePress, Astro Starlight, Hugo, Next.js docs, large developer-platform and API docs, MDN, react.dev, Intercom help centers and Wikipedia, including non-English (Japanese) docs.

Also tested: mdBook, Django, Kubernetes, Microsoft Learn, AWS docs, Cloudflare, Supabase, Tailwind, Vue, Laravel, PostgreSQL, GOV.UK, WordPress, Ghost, LangChain and Anthropic docs, hash-routed Docsify apps and JavaScript help centers (with `renderJavaScript`), and PDF/DOCX files.

### Limits

- Pages are fetched as plain HTML by default (fast and cheap). JavaScript-built sites need `renderJavaScript` and 1024 MB of memory (pages render 2 at a time; more memory renders up to 4 at a time). Rendered pages are re-rendered on every run, since their HTML shell can stay the same while the content changes.
- Scanned PDFs without a text layer produce no text (no OCR). Files larger than 30 MB and pages larger than 10 MB are skipped.
- Sync state is stored (compressed) in your account's `rag-sync-monitor-state` key-value store. One state name comfortably handles tens of thousands of pages; split very large sites with `urlPrefixes` and separate state names.
- Versioned or translated docs (e.g. `/docs/3.2.1/…`, `/docs/next/…`, `/zh-CN/…`) publish many copies of the same page. Index only the version you want with `excludeUrlPatterns`, e.g. `["/docs/\\d", "/docs/next/", "^https://[^/]+/[a-z]{2}(-[A-Z]{2})?/"]`.
- Only crawl sites you are allowed to crawl. The Actor respects robots.txt by default.

### Feedback and support

Found a site that doesn't extract cleanly, or need a feature (moving content between URLs without re-embedding, direct Pinecone/Qdrant writes, more file types)? Open an issue on the **Issues** tab.

If RAG Sync Monitor saves you embedding costs, a short review on this page helps other people find it.

# Actor input Schema

## `startUrls` (type: `array`):

Pages to start crawling from, e.g. the root of a docs site. Only pages under the URL prefixes are crawled.

## `sitemapUrls` (type: `array`):

Optional sitemap.xml or sitemap index URLs. Pages listed in them are added to the crawl.

## `useSitemaps` (type: `boolean`):

Also load sitemaps listed in each start host's robots.txt (or /sitemap.xml), so pages that aren't linked from navigation are found too.

## `urlPrefixes` (type: `array`):

Only URLs starting with one of these are crawled. Defaults to the directory of each start URL.

## `excludeUrlPatterns` (type: `array`):

Regular expressions; matching URLs are skipped, e.g. "/changelog/" or "\\?page=".

## `includeUrlPatterns` (type: `array`):

If set, a URL must also match one of these regular expressions to be synced, e.g. "/guides/" or "/api/v2/".

## `crawlLinks` (type: `boolean`):

Discover new pages by following in-scope links. Turn off to sync only start URLs and sitemap pages.

## `maxCrawlDepth` (type: `integer`):

How many links away from the start URLs to follow. 0 = only the start URLs (and sitemap pages). Empty = no limit.

## `maxPages` (type: `integer`):

Upper limit on pages checked in one run. The form starts at 25 so a first try is quick and cheap; raise it for a full sync (the default when called via API is 200). Unchanged pages are cheap to check thanks to conditional requests.

## `stateKey` (type: `string`):

Name of the saved baseline that runs are compared against. Defaults to the first start URL. Use a different name per site or per vector index. Change it to start a fresh baseline.

## `emitBaseline` (type: `boolean`):

On the first run for a state name, output every chunk as an upsert. Turn off if the site is already indexed and you only want future changes.

## `chunkSize` (type: `integer`):

Chunks follow headings and are packed by paragraph up to this size. Changing it re-chunks every page on the next run.

## `chunkOverlap` (type: `integer`):

Characters repeated from the previous chunk within the same section.

## `granularity` (type: `string`):

Chunks: changed chunks are output. Pages: each changed page is output once as full Markdown, with a stable ID per URL.

## `contentSelector` (type: `string`):

Optional CSS selector for the main content, e.g. "main .markdown". Defaults to main, article or body.

## `removeSelectors` (type: `array`):

Extra elements to strip before chunking, e.g. ".sidebar", ".edit-this-page". Nav, header, footer and scripts are always removed.

## `removeRepeatedText` (type: `boolean`):

Learns lines that appear on most pages of the site (menus, footers, banners, rotating sponsor lists) and removes them from chunks. Headings and code are never removed. When new lines are learned, the next run re-downloads every page once to apply the cleanup.

## `renderJavaScript` (type: `string`):

For sites that build their content in the browser (some help centers and single-page apps). Auto renders only pages with too little static text. Pages that need the browser are charged an extra 'page rendered' event. Use 1024 MB of run memory.

## `dynamicContentWaitSecs` (type: `integer`):

Browser only: how long to wait for the page to finish loading content.

## `waitForSelector` (type: `string`):

Browser only: wait until this element appears before extracting, e.g. "article".

## `includeFiles` (type: `array`):

Also sync linked PDF and Word files found on the same host. PDF chunks are grouped by page ("Page 3") for citations. Scanned PDFs without a text layer produce no text.

## `useCanonicalUrls` (type: `boolean`):

If a page declares another URL as canonical (rel=canonical), sync that URL instead, which removes mirrored and duplicate pages.

## `keepUrlFragments` (type: `boolean`):

For hash-routed apps (URLs like /#/guide). Plain #section anchors are always ignored. Usually combined with Render JavaScript.

## `maxConcurrency` (type: `integer`):

Parallel requests. Keep it low to be polite to the site.

## `proxyConfiguration` (type: `object`):

Use Apify Proxy (or your own) for sites that block cloud IP addresses.

## `customHttpHeaders` (type: `object`):

Extra headers sent with every request to the crawled hosts only, e.g. {"Authorization": "Bearer ..."} for docs behind a login. Stored encrypted.

## `cookies` (type: `array`):

Cookies sent to the crawled hosts only, e.g. \[{"name": "session", "value": "...", "domain": "docs.example.com"}]. Stored encrypted.

## `forceRefetch` (type: `boolean`):

Re-download every page even if the server says it is unchanged. Output is still only real chunk changes.

## `dryRun` (type: `boolean`):

Output what would change without updating the saved baseline, so the next run reports the same changes. Useful for testing a pipeline.

## `respectRobotsTxt` (type: `boolean`):

Skip pages disallowed by the site's robots.txt.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "useSitemaps": false,
  "crawlLinks": true,
  "maxPages": 25,
  "emitBaseline": true,
  "chunkSize": 1500,
  "chunkOverlap": 150,
  "granularity": "chunk",
  "removeRepeatedText": false,
  "renderJavaScript": "never",
  "dynamicContentWaitSecs": 10,
  "useCanonicalUrls": true,
  "keepUrlFragments": false,
  "maxConcurrency": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "forceRefetch": false,
  "dryRun": false,
  "respectRobotsTxt": true
}
```

# Actor output Schema

## `changes` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://docs.apify.com/academy"
        }
    ],
    "maxPages": 25,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("cipherwise/rag-sync-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [{ "url": "https://docs.apify.com/academy" }],
    "maxPages": 25,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("cipherwise/rag-sync-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://docs.apify.com/academy"
    }
  ],
  "maxPages": 25,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call cipherwise/rag-sync-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,cipherwise/rag-sync-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/t8sRNUMIBIdau70OC/builds/2sVAnFkBt4kHis9V4/openapi.json