# RAG Freshness - Incremental Vector Updates (`brocoswork/omniapi-rag-freshness`) Actor

Detect web content changes and return only vector upserts and deletes, avoiding unnecessary RAG re-embeddings.

- **URL**: https://apify.com/brocoswork/omniapi-rag-freshness.md
- **Developed by:** [Anxo Brocos](https://apify.com/brocoswork) (community)
- **Categories:** Developer tools, AI, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$20.00 / 1,000 page analyzeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### Keep RAG knowledge bases fresh without re-embedding everything

RAG Freshness detects meaningful changes on public web pages and returns only the vector operations your retrieval pipeline needs. On the first run it creates a private baseline. On later runs it compares structure-aware Markdown chunks and produces stable-ID upserts and deletes.

Use it to keep documentation assistants, support bots, internal search, product catalogs, policy libraries, and other retrieval-augmented generation systems current while avoiding repeated embedding calls for unchanged content.

### Importable n8n workflow

The public [OmniAPI Apify examples repository](https://github.com/BrocosHub/omniapi-apify-examples) includes a ready-to-import [RAG Freshness n8n workflow](https://github.com/BrocosHub/omniapi-apify-examples/blob/main/n8n/rag-freshness-vector-updates.json), a step-by-step tutorial, and a verified repeat-run demonstration. The example prepares `upserts`, `deletes`, and `embeddingsAvoided` for a downstream vector database while keeping the commercial Actor implementation private.

### What you get

Each successfully analyzed URL produces one dataset item with:

- `actions.upsert`: new or modified chunks with stable IDs, text, position, and hashes.
- `actions.delete`: chunk IDs that disappeared from the source.
- `stats.embeddingsAvoided`: unchanged chunks that do not need another embedding call.
- `baselineCreated`: whether this was the first stored snapshot for the URL.
- `contentHash`: a deterministic hash for auditing and idempotency.
- `sourceId` and `processedAt`: traceability fields for downstream automation.

The Actor is vector-database neutral. Connect the upserts and deletes to Pinecone, Qdrant, Weaviate, Supabase/pgvector, Elasticsearch, Milvus, Chroma, or your own storage layer.

### Typical use cases

#### Incremental documentation updates

Schedule the Actor after a documentation release. Embed only `actions.upsert`, then delete the IDs in `actions.delete`. Unchanged sections retain their IDs, which helps prevent duplicate vectors and unnecessary churn.

#### RAG content-change monitoring

Run the same task daily or weekly and alert only when the returned actions are non-empty. This gives AI and knowledge-management teams a reviewable change feed without rebuilding the entire index.

#### Cost control for embedding pipelines

Track `stats.embeddingsAvoided` to estimate how many embedding requests the incremental workflow avoided. The Actor does not call an embedding provider, so you choose the model, provider, vector database, and retry policy.

### Input example

```json
{
  "urls": [{ "url": "https://example.com/docs" }],
  "maxChunkChars": 1800,
  "overlapChars": 120,
  "includeUnchanged": false,
  "dryRun": false,
  "snapshotStoreName": "omniapi-rag-freshness"
}
```

Run it again with the same URL and `snapshotStoreName` to compare against the saved baseline. Use a different store name for an independent environment, customer, or experiment.

### Output example

```json
{
  "success": true,
  "url": "https://example.com/docs",
  "baselineCreated": false,
  "stats": { "currentChunks": 14, "added": 1, "modified": 1, "removed": 1, "unchanged": 12, "embeddingsAvoided": 12 },
  "actions": {
    "upsert": [{ "id": "stable-chunk-id", "text": "Updated documentation section..." }],
    "delete": ["removed-chunk-id"]
  },
  "processedAt": "2026-08-13T12:00:00.000Z"
}
```

Fields may contain additional metadata. Treat the dataset schema as the source of truth when generating integrations.

### Use with n8n, Make, Zapier, or the API

Call the Actor synchronously when a downstream workflow needs the dataset immediately, or start an asynchronous run and consume the dataset after completion. A common n8n flow is:

1. Schedule Trigger.
2. HTTP Request to the Apify synchronous dataset endpoint.
3. Split the returned items.
4. Embed each `actions.upsert` item.
5. Upsert vectors by stable ID.
6. Delete the IDs in `actions.delete`.

Keep your Apify token in a credential or secret environment variable, never directly in a shared workflow.

### Pricing and cost control

The launch price is **$0.02 per successfully analyzed page**. Failed URL items are returned for diagnosis without charging the `page-analyzed` event. Your Apify platform usage is billed according to your own plan and the pricing shown before each run.

Start with a small URL list, inspect the output, and use Apify's maximum-charge controls before scheduling a large production job.

### Limits and responsible use

- Public HTML pages only, up to 5 MB and 50 URLs per run.
- It does not bypass authentication, paywalls, CAPTCHA, robots restrictions, or anti-bot controls.
- Private-network and unsafe targets are blocked to reduce SSRF risk.
- It extracts page content; it does not execute a full browser-rendered application.
- Deterministic similarity should be evaluated on your content before allowing unattended destructive vector deletes.
- Website layouts and content can change. Monitor scheduled runs and review unexpected output.

Only process pages you are authorized to access and use. You remain responsible for source-site terms, privacy requirements, retention, and the downstream use of extracted content.

### FAQ

#### Does it generate embeddings?

No. It returns the smallest set of chunk changes so you can use your preferred embedding model and vector database.

#### Where is the baseline stored?

In an Apify key-value store under the running user's account. Use `snapshotStoreName` to separate environments or workflows.

#### Will unchanged chunks preserve their IDs?

The matching strategy is designed to preserve stable IDs for unchanged or matched chunks. Test it with representative content before production use.

#### Can I use it for a single page?

Yes. One URL is a good way to validate the result before creating a scheduled task for a larger knowledge base.

# Actor input Schema

## `urls` (type: `array`):

Public web pages to monitor. Reuse the same URLs and snapshot store on later runs.

## `maxChunkChars` (type: `integer`):

Larger chunks reduce vector count; smaller chunks localize changes more precisely.

## `overlapChars` (type: `integer`):

Characters repeated between adjacent oversized chunks to preserve local context.

## `includeUnchanged` (type: `boolean`):

Adds unchanged IDs to each response for debugging; normally leave this disabled.

## `dryRun` (type: `boolean`):

Compare without replacing the stored baseline.

## `snapshotStoreName` (type: `string`):

Keep this value stable between runs. The store belongs to the Actor user.

## Actor input object example

```json
{
  "urls": [
    {
      "url": "https://example.com"
    }
  ],
  "maxChunkChars": 1800,
  "overlapChars": 120,
  "includeUnchanged": false,
  "dryRun": false,
  "snapshotStoreName": "omniapi-rag-freshness"
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "urls": [
        {
            "url": "https://example.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("brocoswork/omniapi-rag-freshness").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "urls": [{ "url": "https://example.com" }] }

# Run the Actor and wait for it to finish
run = client.actor("brocoswork/omniapi-rag-freshness").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "urls": [
    {
      "url": "https://example.com"
    }
  ]
}' |
apify call brocoswork/omniapi-rag-freshness --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,brocoswork/omniapi-rag-freshness"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/mxsayXC5dgeK2XHp2/builds/ollmux2p6n6cURWh2/openapi.json