# RAG Snapshot Guard (`venture_lab/rag-snapshot-guard`) Actor

Compare two Website Content Crawler runs to see which pages to re-embed and which are safe to review for deletion, without modifying your search index.

- **URL**: https://apify.com/venture_lab/rag-snapshot-guard.md
- **Developed by:** [Venture Lab](https://apify.com/venture_lab) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.50 / 1,000 compared pages

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## RAG Snapshot Guard

Know exactly what to re-embed and what to remove before you update a RAG index. Point this Actor at two Website Content Crawler runs of the same site, and it returns the pages that were added or changed, the pages that could not be loaded, and the pages that are safe to consider for deletion.

It never deletes anything itself, and it never proposes a deletion because a crawl simply missed a page.

### How it works with Website Content Crawler

1. Crawl your docs site with [Website Content Crawler](https://apify.com/apify/website-content-crawler) and build your index from that run. This is your **baseline crawl**.
2. Later, crawl the same site again. This is your **current crawl**.
3. Run RAG Snapshot Guard with both datasets and your sitemap URL.
4. Re-embed the pages listed in `upserts`. Review `deletionCandidates` and `deletionProposals` before removing anything.

The result also contains a `ledger`. Pass it to the next comparison (same baseline crawl, newer current crawl) to turn a first absence into a confirmed deletion proposal.

### What you get

- **upserts**: pages that are new or whose text changed, with old and new content fingerprints. Read the page itself from your current crawl dataset.
- **unchanged**: pages whose text is identical to the baseline. Skip these and save embedding costs.
- **quarantined**: pages that failed, were blocked, came back empty, redirected off your site, or are listed in your sitemap but were not crawled. Any quarantined page blocks deletions for that run.
- **deletionCandidates**: pages from your baseline that are missing from both the sitemap and a complete current crawl.
- **deletionProposals**: candidates that were missing again in a second, later complete crawl. A proposal is a recommendation to review, never an automatic delete.
- **source**: what was read: page counts, sitemap counts, and plain-language notes about anything that limited the result.

### Why the sitemap matters

A crawler can miss pages for many reasons: timeouts, rate limits, link changes, crawl limits. If a missing page were treated as deleted, a bad crawl would empty your index.

With a sitemap, a page is a deletion candidate only when it is in neither the sitemap nor the current crawl, and only when every sitemap page was crawled successfully. Without a sitemap, upserts and quarantines still work, but no page is ever reported as removed.

Sitemap indexes and gzip-compressed sitemaps are supported.

### Input

- **Baseline crawl dataset** and **Current crawl dataset**: pick the two Website Content Crawler datasets.
- **Sitemap URL**: optional, but needed for deletion checks.
- **Allowed origins**: optional. Defaults to the sites in your baseline crawl. Pages outside these origins are quarantined.
- **Scope ID**: optional stable name for this index.
- **Ledger**: optional. The `ledger` from the previous result.

Use datasets with the default Website Content Crawler output. The Actor reads `url`, `crawl.loadedUrl`, `crawl.loadedTime`, `crawl.httpStatusCode`, `markdown` and `text`. Pages are identified by the URL their content was loaded from, so a same-site redirect is followed rather than reported as an error.

#### Other crawlers: JSON snapshots

If you use a different crawler, leave the dataset fields empty and provide two JSON snapshots instead. The input form contains a runnable sample. Each snapshot has `scopeId`, `allowedOrigins`, `runId`, `capturedAt` (UTC with milliseconds), `mode` (`full` or `delta`), `complete`, `expectedUrls` (your authoritative inventory) and `records` (`url`, `status` of `ok`, `failed` or `blocked`, `httpStatus`, `text`, optional `finalUrl`). When datasets are selected, the JSON snapshot fields are ignored.

### Pricing

**$0.0005 per compared page ($0.50 per 1,000 pages)**, subject to the price Apify shows before you run. A compared page is one distinct URL across your baseline and current crawls; a page present in both counts once.

| Site size | Cost per comparison | Weekly checks for a month |
|---|---|---|
| 100 pages | about $0.05 | about $0.20 |
| 500 pages | about $0.25 | about $1.00 |
| 2,000 pages | about $1.00 | about $4.00 |

Your Website Content Crawler runs are billed separately by that Actor.

If the run's maximum charge cannot cover every compared page, the run stops before producing output, and nothing is charged. Invalid input, unreadable datasets and unreachable sitemaps also fail without a charge.

### Output and automation

Each successful run writes one result row to the default dataset and the same result to the `OUTPUT` record of the default key-value store. `BILLING` records the metered page count. Failed runs write a safe `ERROR` record with a code and an explanation.

To automate, run Website Content Crawler on a schedule, then call this Actor with the new run's dataset ID and the previous result's `ledger`.

### Limits

- Up to 2,000 pages per crawl dataset (and 2,000 records per JSON snapshot).
- The ledger grows with each comparison and travels back in the input, which is limited to 2 MiB. That allows about 60 chained comparisons for a 100-page site, about 18 for 500 pages, and about 4 for 2,000 pages. When the limit is near, start again from a new baseline crawl.
- Keep your baseline crawl dataset: later comparisons re-read it. Unnamed Apify datasets expire after your plan's retention period, so name the dataset to keep it.
- The current crawl must be newer than the baseline crawl and than every crawl already in the ledger.

### Data handling

Page text stays in your crawl datasets. The result and ledger contain URLs, HTTP statuses and SHA-256 content fingerprints, not page text. The Actor reads only the two datasets you select (read-only access) and the sitemap you name.

### Support

Use the Actor's Issues tab. Include the run ID, what you expected and, if possible, a small redacted example. Do not share access tokens or private documents.

# Actor input Schema

## `baselineDatasetId` (type: `string`):

Dataset of the crawl your RAG index currently reflects. Its successfully loaded pages are the baseline.

## `currentDatasetId` (type: `string`):

Dataset of a newer crawl of the same site.

## `sitemapUrl` (type: `string`):

Your sitemap or sitemap index. It is the authoritative page inventory: a page is a deletion candidate only when it is missing from both the sitemap and the current crawl. Without it, removed pages are never reported.

## `allowedOrigins` (type: `array`):

Origins in scope, such as https://docs.example.com. Defaults to the origins of the baseline crawl.

## `scopeId` (type: `string`):

Stable name for this index. Defaults to an ID derived from the allowed origins.

## `baseline` (type: `object`):

Complete authoritative baseline with scopeId, allowedOrigins, runId, capturedAt, mode, complete, expectedUrls and records.

## `current` (type: `object`):

Current same-scope snapshot. Incomplete coverage prevents deletion proposals.

## `ledger` (type: `object`):

Carry forward the ledger from the previous result to confirm absences across runs. Works with both crawler datasets and JSON snapshots.

## Actor input object example

```json
{
  "sitemapUrl": "https://docs.example.com/sitemap.xml",
  "baseline": {
    "scopeId": "docs-v1",
    "allowedOrigins": [
      "https://example.test"
    ],
    "runId": "baseline",
    "capturedAt": "2026-10-31T00:00:00.000Z",
    "mode": "full",
    "complete": true,
    "expectedUrls": [
      "https://example.test/a",
      "https://example.test/b",
      "https://example.test/c"
    ],
    "records": [
      {
        "url": "https://example.test/a",
        "status": "ok",
        "httpStatus": 200,
        "text": "a"
      },
      {
        "url": "https://example.test/b",
        "status": "ok",
        "httpStatus": 200,
        "text": "b"
      },
      {
        "url": "https://example.test/c",
        "status": "ok",
        "httpStatus": 200,
        "text": "c"
      }
    ]
  },
  "current": {
    "scopeId": "docs-v1",
    "allowedOrigins": [
      "https://example.test"
    ],
    "runId": "current",
    "capturedAt": "2026-10-31T00:00:01.000Z",
    "mode": "full",
    "complete": true,
    "expectedUrls": [
      "https://example.test/a",
      "https://example.test/b"
    ],
    "records": [
      {
        "url": "https://example.test/a",
        "status": "ok",
        "httpStatus": 200,
        "text": "a"
      },
      {
        "url": "https://example.test/b",
        "status": "ok",
        "httpStatus": 200,
        "text": "b"
      }
    ]
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `error` (type: `string`):

No description

## `result` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "baseline": {
        "scopeId": "docs-v1",
        "allowedOrigins": [
            "https://example.test"
        ],
        "runId": "baseline",
        "capturedAt": "2026-10-31T00:00:00.000Z",
        "mode": "full",
        "complete": true,
        "expectedUrls": [
            "https://example.test/a",
            "https://example.test/b",
            "https://example.test/c"
        ],
        "records": [
            {
                "url": "https://example.test/a",
                "status": "ok",
                "httpStatus": 200,
                "text": "a"
            },
            {
                "url": "https://example.test/b",
                "status": "ok",
                "httpStatus": 200,
                "text": "b"
            },
            {
                "url": "https://example.test/c",
                "status": "ok",
                "httpStatus": 200,
                "text": "c"
            }
        ]
    },
    "current": {
        "scopeId": "docs-v1",
        "allowedOrigins": [
            "https://example.test"
        ],
        "runId": "current",
        "capturedAt": "2026-10-31T00:00:01.000Z",
        "mode": "full",
        "complete": true,
        "expectedUrls": [
            "https://example.test/a",
            "https://example.test/b"
        ],
        "records": [
            {
                "url": "https://example.test/a",
                "status": "ok",
                "httpStatus": 200,
                "text": "a"
            },
            {
                "url": "https://example.test/b",
                "status": "ok",
                "httpStatus": 200,
                "text": "b"
            }
        ]
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("venture_lab/rag-snapshot-guard").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "baseline": {
        "scopeId": "docs-v1",
        "allowedOrigins": ["https://example.test"],
        "runId": "baseline",
        "capturedAt": "2026-10-31T00:00:00.000Z",
        "mode": "full",
        "complete": True,
        "expectedUrls": [
            "https://example.test/a",
            "https://example.test/b",
            "https://example.test/c",
        ],
        "records": [
            {
                "url": "https://example.test/a",
                "status": "ok",
                "httpStatus": 200,
                "text": "a",
            },
            {
                "url": "https://example.test/b",
                "status": "ok",
                "httpStatus": 200,
                "text": "b",
            },
            {
                "url": "https://example.test/c",
                "status": "ok",
                "httpStatus": 200,
                "text": "c",
            },
        ],
    },
    "current": {
        "scopeId": "docs-v1",
        "allowedOrigins": ["https://example.test"],
        "runId": "current",
        "capturedAt": "2026-10-31T00:00:01.000Z",
        "mode": "full",
        "complete": True,
        "expectedUrls": [
            "https://example.test/a",
            "https://example.test/b",
        ],
        "records": [
            {
                "url": "https://example.test/a",
                "status": "ok",
                "httpStatus": 200,
                "text": "a",
            },
            {
                "url": "https://example.test/b",
                "status": "ok",
                "httpStatus": 200,
                "text": "b",
            },
        ],
    },
}

# Run the Actor and wait for it to finish
run = client.actor("venture_lab/rag-snapshot-guard").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "baseline": {
    "scopeId": "docs-v1",
    "allowedOrigins": [
      "https://example.test"
    ],
    "runId": "baseline",
    "capturedAt": "2026-10-31T00:00:00.000Z",
    "mode": "full",
    "complete": true,
    "expectedUrls": [
      "https://example.test/a",
      "https://example.test/b",
      "https://example.test/c"
    ],
    "records": [
      {
        "url": "https://example.test/a",
        "status": "ok",
        "httpStatus": 200,
        "text": "a"
      },
      {
        "url": "https://example.test/b",
        "status": "ok",
        "httpStatus": 200,
        "text": "b"
      },
      {
        "url": "https://example.test/c",
        "status": "ok",
        "httpStatus": 200,
        "text": "c"
      }
    ]
  },
  "current": {
    "scopeId": "docs-v1",
    "allowedOrigins": [
      "https://example.test"
    ],
    "runId": "current",
    "capturedAt": "2026-10-31T00:00:01.000Z",
    "mode": "full",
    "complete": true,
    "expectedUrls": [
      "https://example.test/a",
      "https://example.test/b"
    ],
    "records": [
      {
        "url": "https://example.test/a",
        "status": "ok",
        "httpStatus": 200,
        "text": "a"
      },
      {
        "url": "https://example.test/b",
        "status": "ok",
        "httpStatus": 200,
        "text": "b"
      }
    ]
  }
}' |
apify call venture_lab/rag-snapshot-guard --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,venture_lab/rag-snapshot-guard"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/we6FTRJhcM6ASDLRJ/builds/K5iAn7fgVCwdL1eHe/openapi.json