Pricing

from $0.0025 / list processed

Try for free

Go to Apify Store

Line Deduplicator, Sorter & Shuffler

Try for free

Clean big text and list files: remove duplicate lines, sort alphabetically or numerically, reverse, shuffle and trim blanks — handles lists too large for a desktop editor.

Pricing

from $0.0025 / list processed

Rating

0.0

(0)

Developer

hiper soft

Actor stats

Bookmarked

Total users

Monthly active users

16 days ago

Last modified

What it does

Deduplicate lines (optionally case-insensitive).
Sort A→Z, Z→A, natural (numbers-aware), reverse, or shuffle.
Trim whitespace and remove blank lines.
Returns counts (in/out, duplicates, blanks) + the cleaned file.

Use cases

SEO / PPC — dedupe and sort huge keyword lists.
Data cleaning — tidy URL lists and exports before import.
Security / research — process large wordlists that choke local editors.

Input

Paste text, or point to a file URL.

{ "text": "banana\napple\nApple\ncherry\nbanana", "dedupe": true, "caseInsensitiveDedupe": false, "trim": true, "removeBlankLines": true, "sort": "asc" }

Field	Type	Description
`text`	string	Lines to process (one per line).
`fileUrl`	string	Optional link to a .txt/.csv file instead of pasting.
`dedupe`	boolean	Remove duplicate lines. Default `true`.
`caseInsensitiveDedupe`	boolean	Treat case-different lines as duplicates.
`trim`	boolean	Strip leading/trailing spaces. Default `true`.
`removeBlankLines`	boolean	Drop empty lines. Default `true`.
`sort`	string	`none`, `asc`, `desc`, `natural`, `reverse`, `shuffle`.

Output

Line Deduplicator, Sorter & Shuffler output — the results as a clean dataset

A summary row, plus the cleaned file at result.txt in your key-value store.

{ "linesIn": 5, "linesOut": 3, "duplicatesRemoved": 1, "blanksRemoved": 0, "resultUrl": "https://api.apify.com/v2/key-value-stores/.../records/result.txt" }

Output schema

Field	Type	Description
`linesIn`	integer	Lines in the input.
`linesOut`	integer	Lines after processing.
`duplicatesRemoved`	integer	Duplicate lines removed.
`blanksRemoved`	integer	Blank lines removed.
`resultUrl`	string (URL)	Link to the cleaned text file.

FAQ

How big a list can it handle? Very large — it streams the file into memory and processes in one pass.

Can I automate it? Yes — via integrations on the Apify platform and the Apify API.

Notes

Original clean-room implementation.

Dataset Cleaner & Deduplicator — Quality Engine

egeusta/dataset-quality-deduplication-engine

Clean datasets, remove duplicate rows, detect missing values, normalize emails and phones, and validate schemas before exporting to CRM or databases.

Ege Usta

Lead List Deduplicator & Normalizer

webdata_labs/lead-list-deduplicator

[💵 $0.05 / 1K] Clean messy B2B lead lists into CRM-ready company/contact records with duplicate clusters, confidence scores, match reasons, normalized domains, emails, and phones.

WebData Labs

JSON Dataset Cleaner and Deduplicator

rodrgds/dataset-cleaner

Clean JSON datasets, remove empty rows, deduplicate by any field, validate emails, and prepare scraper output for CRMs, analysis, or AI workflows.

Rodrigo Dias

Lead List Deduplicator & Merger

jurassic_jove/lead-deduplicator-merger

Merge and deduplicate lead lists from multiple Apify datasets, CSV files and inline JSON into one clean, outreach-ready list. Pure data processor — no scraping, no proxies, no external APIs.

Data Runner

Data Deduplicator

parsebird/dataset-deduplicator

Merge and deduplicate Apify datasets by any field combination. Remove duplicate rows while keeping the first or last occurrence. Supports case-insensitive matching and whitespace trimming.

ParseBird

Text Sentiment Analysis

maximedupre/text-sentiment-analysis

Classify one line or a full text block as positive, negative, or neutral. Get a confidence score for every line plus line counts and sentiment shares in a structured dataset.

Maxime Dupré

Semantic List Deduplicator

westerly_palisade/list-deduplicate-by-dravara

Merge meaning-equivalent list items without rewriting the originals. Returns the unique list, duplicate groups, reasons, and confidence as bounded JSON.

Dravara LLC

Line Email Scraper - Keyword Search & Domain Filter

parse-point/line-email-scraper-lightning-fast-and-accurate

💚 LINE Email Scraper extracts public account and business emails using keyword and location filters. 🎯 Domain restriction, decoding and duplicate skipping. 🇯🇵 Fast LINE lead generation for Japan & SEA outreach.