🧽 Dataset Deduplicator - Clean Any Scraped Dataset

Detect exact and near-duplicate texts in scraped or collected datasets: normalized hashing for exact matches plus character-shingle Jaccard similarity for near matches. Works on Japanese (no word tokenization needed). Returns per-text verdicts and duplicate clusters. No LLM cost.

Dataset Deduplicator & Cleaner

eszetael_lab/dataset-deduplicator-cleaner

Returns your records with duplicates removed and values normalized — same fields as the input, nothing added or renamed. Exact, normalized or fuzzy matching on keys you choose. A data quality report lands under QUALITY_REPORT. Takes a dataset ID or inline JSON, so it chains after any scraper.

Dataset Deduper | Exact & Fuzzy Dedupe for Any Dataset

broomwagon/dataset-deduper

Collapse duplicates in any dataset, in two passes you control. Exact matching ignores case, punctuation, and word order; an optional fuzzy pass catches the near-duplicates normalization cannot. Every decision is explained in an audit trail, and a cross-run ledger remembers what you already got.

Convert any website sitemap into an LLM-ready dataset for RAG pipelines.

Transform dataset deduplicator v2 inputs into structured rows, clear errors, confidence signals, and automation-ready output.

Scraped Data Cleaner & Converter (No-Code CSV/JSON Tool) Rental

m3web/scraped-data-cleaner-rental

Clean and organize scraped .json or .csv data — no coding required. Remove duplicates, empty rows, unwanted columns, and sort by any field. Cleaned results are pushed to your Apify dataset. Perfect for marketers, researchers, and no-code workflows.

🧩 RAG Dataset Builder - Crawl to Embedding-Ready Chunks

that_red_bird/rag-dataset-builder

🧩 Crawl a site and get back a FINISHED, embedding-ready RAG dataset — not just clean text. ✅ Semantic chunking on heading boundaries, never mid-code-block or mid-table. ✅ Token estimator, configurable overlap, near-duplicate removal, full provenance per chunk.

Data.gov.uk Scraper - Cheap 🌐📊🇬🇧

scrapestorm/data-gov-uk-scraper---cheap

🔎 Easily collect dataset listings from data.gov.uk Provide one or multiple search URLs and extract dataset information such as 📄 Dataset Title 🏢 Published By 🕒 Last Updated 📝 Description 🔗 Dataset URL & more Perfect for open data research, government data monitoring & dataset discovery 📊🚀

Dataset Download

idiatech/apify-Dataset-Download

Download any dataset from the Apify platform automatically and in any format you want. Use this actor along with a Dataset toolbox automation tool.

Dataset Toolkit — Dedupe, Merge, Clean, Diff

glueworks/dataset-toolkit

Post-process any Apify dataset: remove duplicates, merge multiple datasets, clean and normalize fields, or diff two crawls. Outputs a new dataset plus a summary record.