RAG Snapshot Guard

Qdrant Integration

apify/qdrant-integration

Transfer data from Apify Actors to a Qdrant vector database.

PGVector Integration

apify/pgvector-integration

This integration transfers data from Apify Actors to a Postgres SQL database (with PGVector extension).

AI RAG Feeder V2

mickeywmoore/ai-rag-feeder-v2

Turn any website into AI-ready Markdown. Scrapes entire domains, removes ads/clutter, and formats text specifically for RAG pipelines and LLM training data.

RAG Freshness - Incremental Vector Updates

brocoswork/omniapi-rag-freshness

Detect web content changes and return only vector upserts and deletes, avoiding unnecessary RAG re-embeddings.

Audit public website AI/RAG readiness, generate llms.txt + llms-full.txt, and detect added or changed pages across repeat runs. API-ready, schedulable, pay per delivered page.

Crawl sitemap.xml files and emit only added, changed, or deleted Markdown/JSONL chunks for cheaper RAG reindexing.

Website RAG Knowledge Builder

glowing_glove/website-rag-knowledge-builder

Crawl public website pages and build clean RAG-ready knowledge records with page summaries, key facts, FAQs, links, and retrieval chunks.

RAG Markdown Cleaner

cooldev/rag-markdown-cleaner

Transform web pages into RAG-ready Markdown with smart chunking, metadata, code detection & quality scoring. Production-tested deduplication. Fully open-source (Apache 2.0)—review code, contribute, or self-host. Turn messy HTML into embedding-ready knowledge instantly.

Audit which AI crawlers can access a site via robots.txt, llms.txt, and sitemap signals for GPTBot, ClaudeBot, PerplexityBot, and more.