Remove Duplicates & Merge CSV, JSON and Datasets

Merge the results of several Actor runs, CSV files and JSON files into one clean table and remove duplicate rows by the fields you choose: url, email, website, phone, title + city, or anything nested. Keep the first row or the newest one. Download the result as CSV, Excel or JSON from the dataset, or send it on with any integration.

Typical uses:

• Combine daily runs of a scraper and keep one row per product, place or lead, with the newest data.
• Merge lead lists from different tools and drop the people or companies you already have (match on email or website).
• Clean a CSV with duplicate rows, or turn a JSON file into a table (JSON to CSV).

What it does

• Sources: Apify datasets (pick them in the form or paste dataset IDs), public CSV/JSON/JSON Lines file links (CSV with comma, semicolon or tab, UTF-8 or Windows/Excel encoding, up to 20 MB each) and rows pasted as JSON. Mix as many as you like. Pasted rows are read first, then the datasets and then the files, each in the order you list them.
• Remove duplicates by fields: rows with the same values in the chosen fields are duplicates. Use dots for nested fields (address.city). With no fields given, only rows identical in every field are removed.
• Loose matching (on by default): ACME = acme, 1 = "1" (CSV text and dataset numbers match), and https://www.acme.com/ = http://acme.com = www.acme.com. In URLs only the domain ignores letter case, so bit.ly/3AbC and bit.ly/3abc stay different. Turn it off for exact matching.
• Keep first or last: "last" keeps the newest version of each row, at the position where it first appeared.
• Rows without a value in the dedup fields are always kept, never merged with each other.
• Output only some fields, in your order, and optionally add a _source column that says which dataset or file each row came from.

The run's STATS record shows rows read, duplicates removed, rows written and any source that could not be read. A source that fails (a wrong dataset ID, a broken link) is skipped and named in the log. The others are still merged.

Example

Input (the pre-filled example):

{
  "rows": [
    {"company": "Acme", "website": "https://acme.com/"},
    {"company": "ACME ", "website": "http://www.acme.com"},
    {"company": "Globex", "website": "https://globex.com"}
  ],
  "dedupFields": ["website"]
}

Output: 2 rows, Acme and Globex. The second Acme is the same website written differently.

To merge two runs of any Actor, give their dataset IDs, for example:

{
  "datasetIds": ["aBcD1234efGH5678i", "JkLm9012noPQ3456r"],
  "dedupFields": ["url"],
  "keep": "last"
}

Pricing

Pay per event, no subscription:

• $0.0001 per row written ($0.10 per 1,000 rows). Removed duplicates are free.
• $0.001 per run start.

Example: merge 3 runs with 20,000 rows in total, 12,000 unique: $1.20. Other paid tools charge $0.50 to $2.30 per 1,000 rows. You can cap the cost with the maximum charge setting. The Actor stops writing at the cap and never charges more.

Good to know

• Each dataset needs to be one this run can read: your own datasets, or ones you selected in the form.
• Excel files: save as CSV first. Files over 20 MB: load them into an Apify dataset first.
• If none of the rows has a value in your dedup fields, the run says so in the log and status (field names are case-sensitive).
• Large merges with "keep last" hold all unique rows in memory until the end. Give the run more memory (Run options) for millions of rows.

More tools from Rowfeed

These are Actors whose results are worth merging:

• Website Email Extractor & Contact Finder – company emails and phones for the websites in your merged list.
• Google Sheets Export – send the merged table straight to a Google Sheet, without OAuth.
• Google Maps Scraper – places with phone, website, rating and address; merge searches across cities by place ID.
• Keyword Generator – long-tail keyword ideas; merge many seed runs and keep each keyword once.
• SERP API & Rank Checker – Google results for any keyword; merge daily runs into one rank history.
• Website Technology Checker – CMS, ecommerce and analytics tools any website uses.
• WHOIS Lookup & DNS Checker – domain age, expiry, DNS, SSL and an SPF/DMARC grade for lead lists.