AI Training Data Collector — Clean Web Datasets for LLMs

Website Content Crawler for AI Datasets

boztek-ltd/website-content-crawler

Crawl any website and turn its content into AI-ready data: clean Markdown, token-aware RAG chunks, Q&A pairs, and OpenAI fine-tuning JSONL. Removes navigation and boilerplate, scores content quality, and deduplicates pages. Works with 🦜🔗 LangChain, LlamaIndex, Pinecone, and Qdrant.

Convert web pages to clean, LLM-friendly text. Perfect for RAG pipelines, AI chatbot training, and fine-tuning datasets. Removes ads,menus, and clutter automatically.

Crawl websites and convert to clean markdown for AI/RAG, LLM fine-tuning, and document pipelines.

Website to Markdown for LLMs and RAG

rodrgds/website-to-markdown

Convert webpages into clean markdown for LLMs, RAG pipelines, AI datasets, archives, and content extraction. Simple pay-per-page pricing.

Crawl any website and extract clean article text and Markdown, ready to feed into LLMs, ChatGPT, vector databases and RAG pipelines. Removes navigation, ads and boilerplate. Configurable crawl depth and page limits. Export to JSON, CSV, Excel.

Crawl any website and get clean markdown, text, or HTML per page, ready for RAG pipelines, chatbots, and LLM fine tuning. Plain HTTP, no browser, no API key. Pay per page.

Ai Training Data Enricher

fiery_dream/ai-training-data-enricher

Production-grade data enrichment and validation for LLM training datasets. Automatically clean, enrich, deduplicate, and validate your AI training data before fine-tuning.

RAG Web Browser

parseforge/rag-web-browser

Fetches web pages from search queries or direct URLs and returns clean content as markdown, text, and optional HTML. Designed to feed RAG pipelines and LLMs with structured data.

RAG-Ready Web Scraper & Smart Chunker for AI Knowledge Bases

adinfosys-labs/rag-ready-web-scraper-smart-chunker-for-ai-knowledge-bases

RAG-ready web scraper that collects, cleans, deduplicates, filters, and chunks web content into structured datasets for AI pipelines. Generates high-quality knowledge-base data optimized for LLMs, embeddings, and vector databases

LLM Markdown Crawler

sleek_waveform/llm-markdown-crawler

Crawl any website and extract clean, boilerplate-free Markdown optimized for LLMs, RAG pipelines, and AI training datasets. Uses Mozilla Readability to strip navigation and ads, then converts to clean Markdown. No browser required — fast and cheap.