Crawls websites to extract clean, structured content for AI/LLM use, ideal for training datasets, knowledge bases, and RAG systems. Json output includes: * text: Normalized page content * links: Extracted sub-URLs

RAG Knowledge Loader

botflowtech/rag-knowledge-loader

RAG Knowledge Loader

RAG Spider - Web to Markdown Crawler for AI Training Data

lenient_grove/RAG-Spider

Enterprise-grade web crawler that converts messy websites into clean, chunked Markdown for AI systems. Uses Mozilla Readability for 95% cleaner extraction than competitors. Outputs RAG-ready data with metadata and token estimates. Perfect for building knowledge bases and training AI chatbots.

AI RAG Feeder V2

mickeywmoore/ai-rag-feeder-v2

Turn any website into AI-ready Markdown. Scrapes entire domains, removes ads/clutter, and formats text specifically for RAG pipelines and LLM training data.

RAG Pipeline Data Collector

scraper_guru/rag-pipeline-data-collector

AI-ready web content extraction for RAG systems, LLMs, and AI agents. Single-page or multi-page scraping with parallel processing.

RAG Markdown Cleaner

cooldev/rag-markdown-cleaner

Transform web pages into RAG-ready Markdown with smart chunking, metadata, code detection & quality scoring. Production-tested deduplication. Fully open-source (Apache 2.0)—review code, contribute, or self-host. Turn messy HTML into embedding-ready knowledge instantly.

Website To LLM Knowledge Pack

attainable_iota/website-to-llm-knowledge-pack

Crawl any website and turn it into an LLM-ready knowledge pack. This Actor extracts clean main text + metadata, follows links with depth/URL filters, and outputs per-page dataset items plus knowledge.jsonl, knowledge.md, and manifest.json for RAG/embeddings pipelines.

Gemini File Search Builder

yoloshii/gemini-file-search-builder

Build Gemini File Search RAG knowledge bases from any website with automatic citations.

📝 Markdown Maker: HTML to AI-Ready Text

shahidirfan/Markdown-Maker

Instantly convert complex HTML into clean, structured Markdown. This lightweight actor is optimized to render web content into a format that is easily readable for AI LLMs, reducing token usage and improving context. Perfect for RAG pipelines and preparing data for training.