RAG Freshness - Incremental Vector Updates

Keep RAG knowledge bases fresh without re-embedding everything

RAG Freshness detects meaningful changes on public web pages and returns only the vector operations your retrieval pipeline needs. On the first run it creates a private baseline. On later runs it compares structure-aware Markdown chunks and produces stable-ID upserts and deletes.

Use it to keep documentation assistants, support bots, internal search, product catalogs, policy libraries, and other retrieval-augmented generation systems current while avoiding repeated embedding calls for unchanged content.

Importable n8n workflow

The public OmniAPI Apify examples repository includes a ready-to-import RAG Freshness n8n workflow, a step-by-step tutorial, and a verified repeat-run demonstration. The example prepares upserts, deletes, and embeddingsAvoided for a downstream vector database while keeping the commercial Actor implementation private.

What you get

Each successfully analyzed URL produces one dataset item with:

• actions.upsert: new or modified chunks with stable IDs, text, position, and hashes.
• actions.delete: chunk IDs that disappeared from the source.
• stats.embeddingsAvoided: unchanged chunks that do not need another embedding call.
• baselineCreated: whether this was the first stored snapshot for the URL.
• contentHash: a deterministic hash for auditing and idempotency.
• sourceId and processedAt: traceability fields for downstream automation.

The Actor is vector-database neutral. Connect the upserts and deletes to Pinecone, Qdrant, Weaviate, Supabase/pgvector, Elasticsearch, Milvus, Chroma, or your own storage layer.

Typical use cases

Incremental documentation updates

Schedule the Actor after a documentation release. Embed only actions.upsert, then delete the IDs in actions.delete. Unchanged sections retain their IDs, which helps prevent duplicate vectors and unnecessary churn.

RAG content-change monitoring

Run the same task daily or weekly and alert only when the returned actions are non-empty. This gives AI and knowledge-management teams a reviewable change feed without rebuilding the entire index.

Cost control for embedding pipelines

Track stats.embeddingsAvoided to estimate how many embedding requests the incremental workflow avoided. The Actor does not call an embedding provider, so you choose the model, provider, vector database, and retry policy.

Input example

{
  "urls": [{ "url": "https://example.com/docs" }],
  "maxChunkChars": 1800,
  "overlapChars": 120,
  "includeUnchanged": false,
  "dryRun": false,
  "snapshotStoreName": "omniapi-rag-freshness"
}

Run it again with the same URL and snapshotStoreName to compare against the saved baseline. Use a different store name for an independent environment, customer, or experiment.

Output example

{
  "success": true,
  "url": "https://example.com/docs",
  "baselineCreated": false,
  "stats": { "currentChunks": 14, "added": 1, "modified": 1, "removed": 1, "unchanged": 12, "embeddingsAvoided": 12 },
  "actions": {
    "upsert": [{ "id": "stable-chunk-id", "text": "Updated documentation section..." }],
    "delete": ["removed-chunk-id"]
  },
  "processedAt": "2026-08-13T12:00:00.000Z"
}

Fields may contain additional metadata. Treat the dataset schema as the source of truth when generating integrations.

Use with n8n, Make, Zapier, or the API

Call the Actor synchronously when a downstream workflow needs the dataset immediately, or start an asynchronous run and consume the dataset after completion. A common n8n flow is:

1. Schedule Trigger.
2. HTTP Request to the Apify synchronous dataset endpoint.
3. Split the returned items.
4. Embed each actions.upsert item.
5. Upsert vectors by stable ID.
6. Delete the IDs in actions.delete.

Keep your Apify token in a credential or secret environment variable, never directly in a shared workflow.

Pricing and cost control

The launch price is $0.02 per successfully analyzed page. Failed URL items are returned for diagnosis without charging the page-analyzed event. Your Apify platform usage is billed according to your own plan and the pricing shown before each run.

Start with a small URL list, inspect the output, and use Apify's maximum-charge controls before scheduling a large production job.

Limits and responsible use

• Public HTML pages only, up to 5 MB and 50 URLs per run.
• It does not bypass authentication, paywalls, CAPTCHA, robots restrictions, or anti-bot controls.
• Private-network and unsafe targets are blocked to reduce SSRF risk.
• It extracts page content; it does not execute a full browser-rendered application.
• Deterministic similarity should be evaluated on your content before allowing unattended destructive vector deletes.
• Website layouts and content can change. Monitor scheduled runs and review unexpected output.

Only process pages you are authorized to access and use. You remain responsible for source-site terms, privacy requirements, retention, and the downstream use of extracted content.

FAQ

Does it generate embeddings?

No. It returns the smallest set of chunk changes so you can use your preferred embedding model and vector database.

Where is the baseline stored?

In an Apify key-value store under the running user's account. Use snapshotStoreName to separate environments or workflows.

Will unchanged chunks preserve their IDs?

The matching strategy is designed to preserve stable IDs for unchanged or matched chunks. Test it with representative content before production use.

Can I use it for a single page?

Yes. One URL is a good way to validate the result before creating a scheduled task for a larger knowledge base.