RAG Post Processor - Text Cleaner & Chunker for LLM Pipelines

Download any PDF and chunk into semantically coherent segments ready for embedding/RAG. Configurable chunk size + overlap. Returns one row per chunk with page, char count, token estimate. Feed directly into OpenAI text-embedding-3 / Voyage / Cohere. $0.005 per PDF + $0.0002 per chunk.

Chunk tender text for RAG/embeddings with stable ids and metadata.

🧩 LLM Dataset Builder - Site to Embedding-Ready Chunks

that_red_bird/llm-dataset-builder

⚡ Turn any site into a RAG dataset in one run. ✅ Clean markdown (code blocks, tables, headings kept; nav and cookie banners stripped), split into semantic chunks on heading boundaries with token estimates, configurable overlap and a source URL on every chunk. ✅ Ready to embed, no post-processing.

Split Markdown or plain text into retrieval-ready chunks for RAG pipelines: cuts at headings, packs whole sentences up to a size limit with optional overlap, and tags every chunk with its heading breadcrumb. Handles Japanese sentence boundaries. No LLM cost.

Text Cleaner For RAG

junipr/text-cleaner-for-rag

Clean web text for RAG by removing boilerplate, normalizing whitespace, and producing chunk-ready text.

Clean and token-optimise HTML, JSON, scraped text, or URLs for LLM pipelines. Strip boilerplate, chunk by semantics, get token counts — feed your agents clean data, not nav bars.

🧩 RAG Dataset Builder - Crawl to Embedding-Ready Chunks

that_red_bird/rag-dataset-builder

🧩 Crawl a site and get back a FINISHED, embedding-ready RAG dataset — not just clean text. ✅ Semantic chunking on heading boundaries, never mid-code-block or mid-table. ✅ Token estimator, configurable overlap, near-duplicate removal, full provenance per chunk.

AI / RAG Web Crawler

groupoject/ai-rag-web-crawler

Crawl any website and extract clean, LLM-ready Markdown chunks to feed AI agents, chatbots, and RAG pipelines. One row per embeddable chunk.

Convert any URL into clean LLM-ready markdown. Strips ads, nav, footer. Preserves headings, lists, tables, code blocks. Returns token count. Perfect for RAG, fine-tuning, AI agents. 10x cheaper than Firecrawl.

Split long text or documents into properly sized, sentence-aware chunks with overlap for embeddings, vector databases, and RAG pipelines. Choose recursive, sentence-boundary, or fixed-token chunking. Fetch from URLs or paste text directly. Powered by Chonkie.