Website to RAG Chunks - Re-crawl and Update Only What Changed

Website Content Extractor for RAG: Markdown, HTML, Text

nezha/website-content-crawler

Turn docs sites, help centers, blogs, and websites into clean markdown, text, or HTML for RAG, AI knowledge bases, and internal search. Crawl from start URLs or sitemaps and keep the crawl in scope.

Rag Knowledge Graph Builder

cspnair/rag-knowledge-graph-builder

Transform websites into RAG-ready datasets. Crawls pages, chunks content into semantic segments (500-1000 tokens), and generates hypothetical questions for each chunk. No API key needed with native mode. Output: pre-indexed JSON optimized for AI retrieval with 3x better accuracy than raw text.

Paste webpage and PDF URLs and get Markdown, JSONL chunks, PDF inventory, source warnings, and RAG-ready records.

AI RAG Feeder V2

mickeywmoore/ai-rag-feeder-v2

Turn any website into AI-ready Markdown. Scrapes entire domains, removes ads/clutter, and formats text specifically for RAG pipelines and LLM training data.

RAG Markdown Cleaner

cooldev/rag-markdown-cleaner

Transform web pages into RAG-ready Markdown with smart chunking, metadata, code detection & quality scoring. Production-tested deduplication. Fully open-source (Apache 2.0)—review code, contribute, or self-host. Turn messy HTML into embedding-ready knowledge instantly.

Stealth Website Crawler

nocturne/stealth-website-crawler

Crawl websites protected by Cloudflare, DataDome, and other anti-bot systems. Extract clean text or markdown for AI/LLM pipelines. Like Website Content Crawler, but for sites that block you.

Crawls Spanish-language websites and extracts clean markdown content optimized for LLM RAG pipelines. Handles Spanish character encoding, removes boilerplate, and outputs structured chunks ready for embedding.

Tech Docs to LLM-Ready Markdown

hedelka/tech-docs-scraper

Scrapes technical documentation sites (Docusaurus, GitBook, MkDocs, ReadTheDocs) and converts them to clean, structured Markdown for RAG pipelines, LLM training, and AI assistants. Automatically detects documentation framework and removes navigation elements.

E-commerce Product Scraper for AI & RAG

coolinbex/ecommerce-product-scraper-ai-rag

Scrape product catalogs into clean, chunked data for AI search, RAG, embeddings, and vector databases.

RagChunk: Semantic RAG Splitter & Markdown Cleaner

flaretool/rag-semantic-chunker

Extract, clean, and semantically split documentation URLs into token-safe chunks for Vector DBs. Built-in HTML boilerplate stripping saves up to 40% on LLM token costs.