Pricing

$0.50 / 1,000 compared pages

Try for free

Go to Apify Store

RAG Snapshot Guard

Try for free

Compare two Website Content Crawler runs to see which pages to re-embed and which are safe to review for deletion, without modifying your search index.

Pricing

$0.50 / 1,000 compared pages

Rating

0.0

(0)

Developer

Venture Lab

Actor stats

Bookmarked

Total users

Monthly active users

5 days ago

Last modified

Categories

Security

Baseline crawl dataset

baselineDatasetId

Optional

Dataset of the crawl your RAG index currently reflects. Its successfully loaded pages are the baseline.

Type:string

Current crawl dataset

currentDatasetId

Optional

Dataset of a newer crawl of the same site.

Type:string

Sitemap URL (enables deletion checks)

sitemapUrl

Optional

Your sitemap or sitemap index. It is the authoritative page inventory: a page is a deletion candidate only when it is missing from both the sitemap and the current crawl. Without it, removed pages are never reported.

Type:string

Example:

https://docs.example.com/sitemap.xml

Allowed origins (optional)

allowedOrigins

Optional

Origins in scope, such as https://docs.example.com. Defaults to the origins of the baseline crawl.

Type:array

Scope ID (optional)

scopeId

Optional

Stable name for this index. Defaults to an ID derived from the allowed origins.

Type:string

Trusted baseline snapshot

baseline

Optional

Complete authoritative baseline with scopeId, allowedOrigins, runId, capturedAt, mode, complete, expectedUrls and records.

Type:object

Current snapshot

current

Optional

Current same-scope snapshot. Incomplete coverage prevents deletion proposals.

Type:object

Prior returned ledger (optional)

ledger

Optional

Carry forward the ledger from the previous result to confirm absences across runs. Works with both crawler datasets and JSON snapshots.

Type:object

Qdrant Integration

apify/qdrant-integration

Transfer data from Apify Actors to a Qdrant vector database.

Apify

5.0

(6)

PGVector Integration

apify/pgvector-integration

This integration transfers data from Apify Actors to a Postgres SQL database (with PGVector extension).

Apify

4.6

(5)

Website Content Vector Retriever

hamza.alwan/website-content-vector-retriever

Hamza Alwan

AI RAG Feeder V2

mickeywmoore/ai-rag-feeder-v2

Turn any website into AI-ready Markdown. Scrapes entire domains, removes ads/clutter, and formats text specifically for RAG pipelines and LLM training data.

Mickey Moore

RAG Freshness - Incremental Vector Updates

brocoswork/omniapi-rag-freshness

Detect web content changes and return only vector upserts and deletes, avoiding unnecessary RAG re-embeddings.

Anxo Brocos

AI/RAG Readiness & llms.txt Change Monitor

nacred_corner/llms-txt-generator

Audit public website AI/RAG readiness, generate llms.txt + llms-full.txt, and detect added or changed pages across repeat runs. API-ready, schedulable, pay per delivered page.

Cenk KURTOĞLU

Sitemap to Changed-Only RAG JSONL

orbiscribe/sitemap-to-rag-delta-dataset

Crawl sitemap.xml files and emit only added, changed, or deleted Markdown/JSONL chunks for cheaper RAG reindexing.

Orbiscribe Labs

Website RAG Knowledge Builder

glowing_glove/website-rag-knowledge-builder

Crawl public website pages and build clean RAG-ready knowledge records with page summaries, key facts, FAQs, links, and retrieval chunks.

Ushba Khan

RAG Markdown Cleaner

cooldev/rag-markdown-cleaner

Transform web pages into RAG-ready Markdown with smart chunking, metadata, code detection & quality scoring. Production-tested deduplication. Fully open-source (Apache 2.0)—review code, contribute, or self-host. Turn messy HTML into embedding-ready knowledge instantly.