Pricing

Pay per event

🧩 LLM Dataset Builder - Site to Embedding-Ready Chunks

⚡ Turn any site into a RAG dataset in one run. ✅ Clean markdown (code blocks, tables, headings kept; nav and cookie banners stripped), split into semantic chunks on heading boundaries with token estimates, configurable overlap and a source URL on every chunk. ✅ Ready to embed, no post-processing.

Pricing

Pay per event

Rating

0.0

(0)

Developer

mohamed alaya

Actor stats

Bookmarked

Total users

Monthly active users

6 days ago

Last modified

LLM Dataset Builder

Turn any website into an embedding-ready dataset in one run — not just markdown, the finished artifact you can push straight into a vector store.

Why this exists

Plenty of tools give you "the page as markdown". Then you still have to strip the navigation, decide where to split, keep chunks from cutting mid-sentence, count tokens, add overlap, and attach a source URL to every piece so retrieval can cite it. That post-processing is the actual work. This does it.

What you get per chunk

text · estimatedTokens · characters · url (provenance — cite the source) · pageTitle · heading (the section the chunk came from) · chunkIndex / chunkCount · language · a stable id

How the chunking works

Splitting is heading-aware first. Sections are packed up to your target size and only hard-split when a single section is oversized — because splitting mid-sentence is what quietly ruins retrieval quality. Overlap is carried between chunks so a sentence spanning a boundary is still findable. Chunks below minChunkTokens are dropped: they are almost always nav scraps and they pollute results.

Tune with chunkTokens (512 suits most embedding models; 256 for tighter retrieval, 1024 for more context per hit), overlapTokens and minChunkTokens.

Content extraction

Headings, fenced code blocks with language tags, tables, lists, blockquotes and links are preserved. Navigation, headers, footers, sidebars, cookie banners, ad slots, share widgets and related-posts blocks are stripped. Set includeLinks: false for cleaner embedding text when URLs add noise.

Discovery

Sitemap first, including sitemap indexes; falls back to following same-origin links. Narrow the crawl with includePatterns / excludePatterns. Note that sitemaps are usually alphabetical, so when an include filter is set the whole sitemap is read before filtering — a section late in the alphabet would otherwise be silently missed.

Output modes

chunks (default) — one row per chunk, ready to embed
pages — one row per page with full markdown, if you want to chunk it yourself

Typical uses

Building RAG over your own docs · ingesting a vendor's documentation into a support bot · creating a searchable knowledge base · preparing fine-tuning or evaluation corpora · giving an internal agent grounded product knowledge.

Reliability

It reads public pages and your own sites — no login, no anti-bot, no third party that can cut you off. If no page yields usable content the run fails loudly rather than handing back an empty dataset.

🧩 RAG Dataset Builder - Crawl to Embedding-Ready Chunks

that_red_bird/rag-dataset-builder

🧩 Crawl a site and get back a FINISHED, embedding-ready RAG dataset — not just clean text. ✅ Semantic chunking on heading boundaries, never mid-code-block or mid-table. ✅ Token estimator, configurable overlap, near-duplicate removal, full provenance per chunk.

mohamed alaya

RAG Docs Extractor - Documentation to Chunks

ambitious_door/ragdocs-extractor

Turn any documentation site into clean, RAG-ready chunks in a single call. Semantic boundaries, preserved structure, per-chunk metadata.

C. K.

RAG Text Chunker — heading & sentence aware, Japanese ready

shoebill-dev27/rag-text-chunker

Split Markdown or plain text into retrieval-ready chunks for RAG pipelines: cuts at headings, packs whole sentences up to a size limit with optional overlap, and tags every chunk with its heading breadcrumb. Handles Japanese sentence boundaries. No LLM cost.

Shinobu Otani

Website to Markdown for LLMs

agentictools/website-to-markdown-llm

Crawl a site and export clean Markdown with token counts and chunks, ready for RAG.

Ken Agland

RAG Post Processor - Text Cleaner & Chunker for LLM Pipelines

jalicia/rag-post-processor

Clean and chunk scraped text into embedding-ready chunks, sized in tokens (tiktoken) rather than characters. Preserves headings, tables, code blocks and links. Every chunk carries its token count, heading path and content hash. Chain after any scraper. Billed per KB of text processed.

Jordan Wagner

PDF → RAG Chunks (Token-Aware, Vector-Ready)

gochujang/pdf-rag-chunker

Download any PDF and chunk into semantically coherent segments ready for embedding/RAG. Configurable chunk size + overlap. Returns one row per chunk with page, char count, token estimate. Feed directly into OpenAI text-embedding-3 / Voyage / Cohere. $0.005 per PDF + $0.0002 per chunk.

Hojun Lee

PDF to RAG-Ready Dataset

utilityforgelab/pdf-to-rag-ready-dataset

Convert public text-based PDFs into clean, token-aware RAG chunks with Markdown, page references, metadata, stable document IDs, and ready-to-use Apify Dataset output.

UtilityForgeLab

PDF to RAG Markdown Chunks for Embeddings

awesome_highboy/docforge

Convert PDFs into token-bounded Markdown chunks for RAG, embeddings, and vector databases (Pinecone, Chroma, Weaviate, Qdrant). Set maxTokens + overlap; get clean chunks with page number, token count, and SHA-256 content hash for dedup. JSON dataset ready for any LLM pipeline.

Adam

LLM-Ready Web Content Extractor

beyondxl/llm-ready-content-extractor

Turn any public web page or shallow site crawl into clean, LLM-ready Markdown + metadata + optional RAG chunks. Boilerplate/nav/ads removed. Honors robots.txt.