# RAG Delta Sync (`danielmichalek/rag-delta-sync`) Actor

Compare crawler exports and prepare changed chunks with stable IDs for a RAG index. Keep unchanged text out of the embedding queue. Ready to refresh your knowledge base?

- **URL**: https://apify.com/danielmichalek/rag-delta-sync.md
- **Developed by:** [Daniel Michałek](https://apify.com/danielmichalek) (community)
- **Categories:** AI
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 71.4% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$20.00 / 1,000 completed delta reports

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RAG Delta Sync

Compare two crawler exports and prepare stable chunk IDs, changed-text upserts and guarded stale-chunk deletes for a RAG index. Unchanged text stays out of the embedding queue. No model API or vector database connection is required.

### What it does

Feed the Actor current and previous exports from Website Content Crawler or another crawler that returns a URL and Markdown or plain text. It normalizes line endings, splits text at paragraphs within a character budget, and identifies chunks by source URL and content hash. Reordered unchanged chunks keep their IDs. Title-only changes create metadata upserts with `embeddingRequired: false`.

The output is an explicit plan for an n8n workflow, LangChain application or custom ingestion worker. The Actor does not generate embeddings or modify an external index. Apply upserts first and then approved deletes. Review blocked deletion candidates separately.

### Input

Choose `currentDatasetId` or `currentPages`, and optionally `previousDatasetId` or `previousPages`. The inline fields are arrays of objects with `url` and one of `markdown`, `text` or `pageContent`; `title` is optional. If both Markdown and text exist, nonempty Markdown wins. Dataset selectors grant READ permission only to the selected dataset. A previous source must be an earlier crawler export, never this Actor's delta-report output.

Defaults are 100 pages per source, 1600 Unicode characters per chunk, 20000 UTF-16 code units per page and 3000 chunks or planned operations. Maximums are 200 pages and 5000 chunks or operations. Exceeding a limit fails before a partial report is emitted. Large pages are rejected rather than silently shortened. Keep `chunkChars` and the chunker version unchanged across comparisons; changing the ingestion chunking contract requires a deliberate index rebuild.

Deletes are blocked unless `snapshotComplete` is explicitly true. Confirm it only when both exports contain complete page content and complete site coverage. The default coverage gate requires at least 80 percent of the previous URLs to remain valid; unrelated new URLs cannot hide a loss. The default deletion gate blocks a plan deleting more than half of previous chunks. Any reported current source error also blocks deletions. A missing page is not proof it was removed from the website.

### Output

One dataset item is one complete comparison. `counts` describes pages, chunks, upserts, unchanged chunks, approved deletes, blocked deletes and skipped embedding requests. `upserts` contains `id`, `sourceUrl`, `title`, source-derived `text`, `contentHash`, `embeddingRequired` and `reason`. `deletes` contains only candidates that passed all gates; `blockedDeletes` contains the remaining candidates. `deletionGate` explains every blocking reason. IDs are SHA-256 values with a versioned namespace and duplicate-chunk occurrence index.

The key-value store includes the same machine-readable plan as `CHANGES`, a short readable `REPORT`, and the execution summary as `OUTPUT`. In n8n, split `upserts` into items, embed only records whose `embeddingRequired` is true, and upsert using `id` as the vector ID. For metadata-only updates, preserve the existing vector. Process `deletes` only after successful upserts. Never apply `blockedDeletes` automatically.

### Pricing and cost control

The intended Store price is 0.02 USD per completed comparison under event `delta-report`. One result record is charged once after its dataset write succeeds, regardless of the number of chunks inside the report. No AI inference, embedding provider, proxy or separate subscription is used. Set the Apify maximum charge to 0.02 USD to cap one report. Source crawler runs and your downstream embedding or database service have their own costs. Failed input validation produces no completed report.

### Limitations and fair use

This is a transformer for existing source exports. It does not visit websites, follow links, parse HTML into clean text, crawl JavaScript sites, or verify that a source truly contains the full website. Source content is not interpreted as instructions and is never sent to an AI model. Supply only content you are entitled to process. The Actor never requests full account permissions.

Whole paragraphs are grouped into character-budget chunks, not exact model-token chunks. Paragraph insertions can change neighboring chunk boundaries. Conflicting duplicate URLs, malformed sources, an all-empty current export, source read failures and invalid previous pages fail closed. Current failed pages are protected; use the reason codes to repair the crawl before destructive reconciliation.

### Example

```json
{
  "currentPages": [{"url": "https://example.com/docs", "text": "The return period is 30 days."}],
  "previousPages": [{"url": "https://example.com/docs", "text": "The return period is 14 days."}],
  "snapshotComplete": false
}
```

This produces one new-text upsert and one blocked stale-chunk candidate. After checking complete snapshots and reviewing the change, `snapshotComplete: true` and an appropriate `maxDeleteRatio` allow the stale chunk to be removed. For the next run, keep the current SOURCE crawler dataset as the previous baseline. The actor output itself is not the next source snapshot.

# Actor input Schema

## `currentDatasetId` (type: `string`):

Dataset ID containing the current crawl export. Rows need url and markdown, text or pageContent. Choose this OR currentPages.

## `previousDatasetId` (type: `string`):

Previous SOURCE crawl export using the same chunk settings. Do not point this at a delta-report output dataset.

## `currentPages` (type: `array`):

Alternative to currentDatasetId: objects with url and text or markdown. No website is fetched by this Actor.

## `previousPages` (type: `array`):

Alternative to previousDatasetId. Omit for an initial import.

## `snapshotComplete` (type: `boolean`):

Default false blocks all deletes. Enable only for complete crawl exports with complete page text. Coverage, source errors and delete ratio gates still apply.

## `minCoverageRatio` (type: `number`):

Fraction of previous URLs that must remain valid in the new crawl before any delete is approved. New unrelated pages cannot hide lost coverage.

## `maxDeleteRatio` (type: `number`):

Blocks deletions if the proposed deleted chunks exceed this fraction of previous chunks. Review large rewrites before increasing.

## `maxPages` (type: `integer`):

Fails before a partial comparison if either source contains more rows. Maximum 200.

## `chunkChars` (type: `integer`):

Paragraph-aware target size, not an exact token count. Keep this setting unchanged between comparisons.

## `maxPageChars` (type: `integer`):

Maximum UTF-16 code units per source page. Oversized pages fail; no silent truncation.

## `maxOperations` (type: `integer`):

Hard cap on either snapshot chunk count and the combined upsert/deletion candidate plan. Exceeding it fails without a partial report.

## `maxItems` (type: `integer`):

One completed comparison is one billable report regardless of chunk count.

## Actor input object example

```json
{
  "currentPages": [
    {
      "url": "https://example.com/docs/returns",
      "title": "Example returns policy",
      "text": "The return period is 30 days."
    }
  ],
  "snapshotComplete": false,
  "minCoverageRatio": 0.8,
  "maxDeleteRatio": 0.5,
  "maxPages": 100,
  "chunkChars": 1600,
  "maxPageChars": 20000,
  "maxOperations": 3000,
  "maxItems": 1
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `changes` (type: `string`):

No description

## `report` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "currentPages": [
        {
            "url": "https://example.com/docs/returns",
            "title": "Example returns policy",
            "text": "The return period is 30 days."
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("danielmichalek/rag-delta-sync").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "currentPages": [{
            "url": "https://example.com/docs/returns",
            "title": "Example returns policy",
            "text": "The return period is 30 days.",
        }] }

# Run the Actor and wait for it to finish
run = client.actor("danielmichalek/rag-delta-sync").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "currentPages": [
    {
      "url": "https://example.com/docs/returns",
      "title": "Example returns policy",
      "text": "The return period is 30 days."
    }
  ]
}' |
apify call danielmichalek/rag-delta-sync --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,danielmichalek/rag-delta-sync"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bGc1S4kGwXtt8qt17/builds/e8rr6sZgcN3fV5gPf/openapi.json