RAG Delta Sync

Compare two crawler exports and prepare stable chunk IDs, changed-text upserts and guarded stale-chunk deletes for a RAG index. Unchanged text stays out of the embedding queue. No model API or vector database connection is required.

What it does

Feed the Actor current and previous exports from Website Content Crawler or another crawler that returns a URL and Markdown or plain text. It normalizes line endings, splits text at paragraphs within a character budget, and identifies chunks by source URL and content hash. Reordered unchanged chunks keep their IDs. Title-only changes create metadata upserts with embeddingRequired: false.

The output is an explicit plan for an n8n workflow, LangChain application or custom ingestion worker. The Actor does not generate embeddings or modify an external index. Apply upserts first and then approved deletes. Review blocked deletion candidates separately.

Input

Choose currentDatasetId or currentPages, and optionally previousDatasetId or previousPages. The inline fields are arrays of objects with url and one of markdown, text or pageContent; title is optional. If both Markdown and text exist, nonempty Markdown wins. Dataset selectors grant READ permission only to the selected dataset. A previous source must be an earlier crawler export, never this Actor's delta-report output.

Defaults are 100 pages per source, 1600 Unicode characters per chunk, 20000 UTF-16 code units per page and 3000 chunks or planned operations. Maximums are 200 pages and 5000 chunks or operations. Exceeding a limit fails before a partial report is emitted. Large pages are rejected rather than silently shortened. Keep chunkChars and the chunker version unchanged across comparisons; changing the ingestion chunking contract requires a deliberate index rebuild.

Deletes are blocked unless snapshotComplete is explicitly true. Confirm it only when both exports contain complete page content and complete site coverage. The default coverage gate requires at least 80 percent of the previous URLs to remain valid; unrelated new URLs cannot hide a loss. The default deletion gate blocks a plan deleting more than half of previous chunks. Any reported current source error also blocks deletions. A missing page is not proof it was removed from the website.

Output

One dataset item is one complete comparison. counts describes pages, chunks, upserts, unchanged chunks, approved deletes, blocked deletes and skipped embedding requests. upserts contains id, sourceUrl, title, source-derived text, contentHash, embeddingRequired and reason. deletes contains only candidates that passed all gates; blockedDeletes contains the remaining candidates. deletionGate explains every blocking reason. IDs are SHA-256 values with a versioned namespace and duplicate-chunk occurrence index.

The key-value store includes the same machine-readable plan as CHANGES, a short readable REPORT, and the execution summary as OUTPUT. In n8n, split upserts into items, embed only records whose embeddingRequired is true, and upsert using id as the vector ID. For metadata-only updates, preserve the existing vector. Process deletes only after successful upserts. Never apply blockedDeletes automatically.

Pricing and cost control

The intended Store price is 0.02 USD per completed comparison under event delta-report. One result record is charged once after its dataset write succeeds, regardless of the number of chunks inside the report. No AI inference, embedding provider, proxy or separate subscription is used. Set the Apify maximum charge to 0.02 USD to cap one report. Source crawler runs and your downstream embedding or database service have their own costs. Failed input validation produces no completed report.

Limitations and fair use

This is a transformer for existing source exports. It does not visit websites, follow links, parse HTML into clean text, crawl JavaScript sites, or verify that a source truly contains the full website. Source content is not interpreted as instructions and is never sent to an AI model. Supply only content you are entitled to process. The Actor never requests full account permissions.

Whole paragraphs are grouped into character-budget chunks, not exact model-token chunks. Paragraph insertions can change neighboring chunk boundaries. Conflicting duplicate URLs, malformed sources, an all-empty current export, source read failures and invalid previous pages fail closed. Current failed pages are protected; use the reason codes to repair the crawl before destructive reconciliation.

Example

{
  "currentPages": [{"url": "https://example.com/docs", "text": "The return period is 30 days."}],
  "previousPages": [{"url": "https://example.com/docs", "text": "The return period is 14 days."}],
  "snapshotComplete": false
}

This produces one new-text upsert and one blocked stale-chunk candidate. After checking complete snapshots and reviewing the change, snapshotComplete: true and an appropriate maxDeleteRatio allow the stale chunk to be removed. For the next run, keep the current SOURCE crawler dataset as the previous baseline. The actor output itself is not the next source snapshot.