Docs-to-RAG Pipeline Builder

AI-Ready Webpage Extractor

s3nafps/ai-ready-webpage-extractor

Convert public web pages into clean Markdown, text, links, tables, images, metadata, and JSON-LD for AI workflows.

AI / RAG Web Crawler

groupoject/ai-rag-web-crawler

Crawl any website and extract clean, LLM-ready Markdown chunks to feed AI agents, chatbots, and RAG pipelines. One row per embeddable chunk.

RAG Spider - Web to Markdown Crawler for AI Training Data

lenient_grove/RAG-Spider

Enterprise-grade web crawler that converts messy websites into clean, chunked Markdown for AI systems. Uses Mozilla Readability for 95% cleaner extraction than competitors. Outputs RAG-ready data with metadata and token estimates. Perfect for building knowledge bases and training AI chatbots.

LLM Data Pipeline Pro

sanztheo/llm-data-pipeline-pro

Transform websites into LLM training data. Scrape, validate, deduplicate, chunk for RAG, and export to OpenAI/Anthropic/Mistral formats. Built-in PII detection and GDPR compliance. Vector DB export to Pinecone & Qdrant.

Crawl any website and extract clean, LLM-ready Markdown for RAG pipelines, vector databases and AI agents. Boilerplate, navigation and cookie banners removed. No browser, no login.

Website to Text & Markdown — AI / RAG Content Crawler

inexhaustible_glass/rag-website-crawler

Scrape any website into clean text & Markdown with RAG-ready chunks and token counts for LLMs, vector databases (Pinecone, Qdrant) and AI chatbots. Also extracts linked PDF/Word/Excel. Anti-block, robots.txt-aware. Website-to-text for beginners, full RAG pipeline for pros. CPU only, no API key.

Crawl any website and extract clean Markdown optimized for LLM training, RAG pipelines, and AI knowledge bases - removes boilerplate and outputs structured JSON with URL, title, markdown, and metadata.

Sitemap Reindex Trigger for RAG and Agent Freshness

crawlware/sitemap-reindex-trigger

Tells your RAG pipeline or AI agent exactly which pages changed since the last check, so you re embed only what is new instead of the whole site.

Feed Reindex Trigger — Multi-Feed RSS/Atom for RAG & Agents

crawlware/feed-reindex-trigger

Multi-feed RSS/Atom monitor that dedupes by GUID and returns clean incremental Markdown since last run. Built for RAG pipelines and agents.

Doc-to-Markdown/JSON RAG Prep - Convert PDF & DOCX for RAG

bigjoecoding/doc-to-markdown-json-rag-prep

Convert PDF, DOCX, PPTX and webpages to clean Markdown and RAG-ready JSON chunks for your embedding pipeline. No LLM cost. $0.03 per document.