LLM Dataset Cleaner: Near-Duplicate Dedupe, Language, Tokens

Crawls Spanish-language websites and extracts clean markdown content optimized for LLM RAG pipelines. Handles Spanish character encoding, removes boilerplate, and outputs structured chunks ready for embedding.

Image to Text OCR: Markdown, Word Boxes & RAG Chunks

bongo-seakhoa/image-to-text-ocr-markdown

Extract text from PNG, JPG, TIFF and WebP images offline with Tesseract. Markdown, plain text, word boxes, confidence scores and RAG chunks. No API key.

Website to RAG Dataset

sebastian-actors/website-to-rag-dataset

Convert public websites, docs, blogs, and XML sitemaps into clean Markdown, structured metadata, and stable chunks for RAG pipelines and vector databases.

Dataset Deduplicator & Cleaner

eszetael_lab/dataset-deduplicator-cleaner

Returns your records with duplicates removed and values normalized — same fields as the input, nothing added or renamed. Exact, normalized or fuzzy matching on keys you choose. A data quality report lands under QUALITY_REPORT. Takes a dataset ID or inline JSON, so it chains after any scraper.

Fetch any URL and return clean markdown/text plus links. No headless browser for most pages. Free demo: https://vaytric.com/page-to-markdown/ — bulk URL lists here.

Crawl any website, extract clean text, split into chunks with quality scoring, and export to JSON, Pinecone, or Qdrant. Built for RAG pipelines and AI training data. Includes language detection, content type classification, and token counting.

Detect exact and near-duplicate texts in scraped or collected datasets: normalized hashing for exact matches plus character-shingle Jaccard similarity for near matches. Works on Japanese (no word tokenization needed). Returns per-text verdicts and duplicate clusters. No LLM cost.

Remove duplicate rows from an Apify Dataset or inline JSON. Deduplicate by selected fields, trim strings, and write the cleaned rows to a new Dataset.

Audio to Text & RAG Chunks: Offline Whisper, SRT, Timestamps

bongo-seakhoa/audio-to-text-rag-chunks

Transcribe MP3, WAV, FLAC and OGG audio offline with Whisper. Timestamped RAG chunks, SRT and VTT subtitles, language detection. No API key needed.

E-commerce Product Scraper for AI & RAG

coolinbex/ecommerce-product-scraper-ai-rag

Scrape product catalogs into clean, chunked data for AI search, RAG, embeddings, and vector databases.