RAG Web Corpus Builder

AI / RAG Web Crawler

groupoject/ai-rag-web-crawler

Crawl any website and extract clean, LLM-ready Markdown chunks to feed AI agents, chatbots, and RAG pipelines. One row per embeddable chunk.

Common Crawl Corpus Builder

zentrafoundry/common-crawl-targeted-corpus-builder

Build targeted text corpora from Common Crawl with provenance.

Score web pages for RAG corpus quality: text length, heading structure, boilerplate hints, token estimate, metadata, and warnings.

Knowledge Intelligence Engine — Website to Markdown for RAG

ryanclinton/website-content-to-markdown

Turn any website, documentation site or help centre into a retrieval-ready knowledge corpus for RAG and AI search. Clean Markdown plus chunks, change detection, deduplication, retrieval scoring, version awareness and a full corpus audit, in one run.

Convert patent and trademark documents into RAG-ready Markdown chunks. Clean JSON for IP counsel, brand teams and AI agents.

YouTube & Transcript Corpus Audit for RAG

taroyamada/youtube-channel-transcript-rag-intelligence

Audit authorized transcript text for RAG readiness, produce retrieval chunks and source-linked reports, and optionally try public YouTube captions without login or cookies.

YouTube Transcript Scraper

taroyamada/youtube-transcript-bulk-api

Extract YouTube captions, timestamps, SRT, VTT, and plain text from public videos in bulk without browser automation.

AI Web Search — Live SERP API for Agents (Tavily Alternative)

logiover/ai-web-search

Keyless live web search for AI agents and RAG. Send a query, get clean ranked results (title, URL, snippet) from DuckDuckGo + Bing, with optional page content as Markdown. A Tavily / Exa / Brave Search alternative. No API key.

google-search-autocomplete-scraper-youtube-suggestions

scraper-engine/google-search-autocomplete-api

Scrape Google Search autocomplete suggestions in real time. Collect keyword ideas, query variations, trending searches, and long-tail phrases at scale. Ideal for SEO research, content planning, PPC targeting, and market analysis. Export structured results in JSON, CSV, or Excel.

RAG Website Crawler: Clean Markdown Chunks for LLMs

themineworks/rag-crawler

Crawl any website into clean, pre-chunked Markdown with per-chunk token counts for RAG pipelines, vector DBs (Pinecone, Qdrant) and LLM context. MCP native for Claude & ChatGPT. SPA support via Playwright. Pay only for pages that crawl. A Firecrawl alternative.

AI Training Dataset Builder: Articles, Blogs & Web Pages

turboextract/ai-training-dataset-builder

Turn any list of URLs into clean, structured training data for AI models, RAG systems, and LLM fine-tuning. Built for ML engineers and AI teams.