RAG Snapshot Guard

Know exactly what to re-embed and what to remove before you update a RAG index. Point this Actor at two Website Content Crawler runs of the same site, and it returns the pages that were added or changed, the pages that could not be loaded, and the pages that are safe to consider for deletion.

It never deletes anything itself, and it never proposes a deletion because a crawl simply missed a page.

How it works with Website Content Crawler

1. Crawl your docs site with Website Content Crawler and build your index from that run. This is your baseline crawl.
2. Later, crawl the same site again. This is your current crawl.
3. Run RAG Snapshot Guard with both datasets and your sitemap URL.
4. Re-embed the pages listed in upserts. Review deletionCandidates and deletionProposals before removing anything.

The result also contains a ledger. Pass it to the next comparison (same baseline crawl, newer current crawl) to turn a first absence into a confirmed deletion proposal.

What you get

• upserts: pages that are new or whose text changed, with old and new content fingerprints. Read the page itself from your current crawl dataset.
• unchanged: pages whose text is identical to the baseline. Skip these and save embedding costs.
• quarantined: pages that failed, were blocked, came back empty, redirected off your site, or are listed in your sitemap but were not crawled. Any quarantined page blocks deletions for that run.
• deletionCandidates: pages from your baseline that are missing from both the sitemap and a complete current crawl.
• deletionProposals: candidates that were missing again in a second, later complete crawl. A proposal is a recommendation to review, never an automatic delete.
• source: what was read: page counts, sitemap counts, and plain-language notes about anything that limited the result.

Why the sitemap matters

A crawler can miss pages for many reasons: timeouts, rate limits, link changes, crawl limits. If a missing page were treated as deleted, a bad crawl would empty your index.

With a sitemap, a page is a deletion candidate only when it is in neither the sitemap nor the current crawl, and only when every sitemap page was crawled successfully. Without a sitemap, upserts and quarantines still work, but no page is ever reported as removed.

Sitemap indexes and gzip-compressed sitemaps are supported.

Input

• Baseline crawl dataset and Current crawl dataset: pick the two Website Content Crawler datasets.
• Sitemap URL: optional, but needed for deletion checks.
• Allowed origins: optional. Defaults to the sites in your baseline crawl. Pages outside these origins are quarantined.
• Scope ID: optional stable name for this index.
• Ledger: optional. The ledger from the previous result.

Use datasets with the default Website Content Crawler output. The Actor reads url, crawl.loadedUrl, crawl.loadedTime, crawl.httpStatusCode, markdown and text. Pages are identified by the URL their content was loaded from, so a same-site redirect is followed rather than reported as an error.

Other crawlers: JSON snapshots

If you use a different crawler, leave the dataset fields empty and provide two JSON snapshots instead. The input form contains a runnable sample. Each snapshot has scopeId, allowedOrigins, runId, capturedAt (UTC with milliseconds), mode (full or delta), complete, expectedUrls (your authoritative inventory) and records (url, status of ok, failed or blocked, httpStatus, text, optional finalUrl). When datasets are selected, the JSON snapshot fields are ignored.

Pricing

$0.0005 per compared page ($0.50 per 1,000 pages), subject to the price Apify shows before you run. A compared page is one distinct URL across your baseline and current crawls; a page present in both counts once.

Site size	Cost per comparison	Weekly checks for a month
100 pages	about $0.05	about $0.20
500 pages	about $0.25	about $1.00
2,000 pages	about $1.00	about $4.00

Your Website Content Crawler runs are billed separately by that Actor.

If the run's maximum charge cannot cover every compared page, the run stops before producing output, and nothing is charged. Invalid input, unreadable datasets and unreachable sitemaps also fail without a charge.

Output and automation

Each successful run writes one result row to the default dataset and the same result to the OUTPUT record of the default key-value store. BILLING records the metered page count. Failed runs write a safe ERROR record with a code and an explanation.

To automate, run Website Content Crawler on a schedule, then call this Actor with the new run's dataset ID and the previous result's ledger.

Limits

• Up to 2,000 pages per crawl dataset (and 2,000 records per JSON snapshot).
• The ledger grows with each comparison and travels back in the input, which is limited to 2 MiB. That allows about 60 chained comparisons for a 100-page site, about 18 for 500 pages, and about 4 for 2,000 pages. When the limit is near, start again from a new baseline crawl.
• Keep your baseline crawl dataset: later comparisons re-read it. Unnamed Apify datasets expire after your plan's retention period, so name the dataset to keep it.
• The current crawl must be newer than the baseline crawl and than every crawl already in the ledger.

Data handling

Page text stays in your crawl datasets. The result and ledger contain URLs, HTTP statuses and SHA-256 content fingerprints, not page text. The Actor reads only the two datasets you select (read-only access) and the sitemap you name.

Support

Use the Actor's Issues tab. Include the run ID, what you expected and, if possible, a small redacted example. Do not share access tokens or private documents.