Dataset Merge & Dedup Datasets — Filter, Join CSV & Excel

Merge Apify datasets, CSV, TSV, JSON, JSONL and Excel files, and Google Sheets into one clean table, in one run: clean fields, normalise URLs, extract domains, filter, remove duplicates, join (VLOOKUP) with a second table, sort, and export to CSV, Excel and JSON for free. Use it on its own or as the last step after any scraper.

$0.50 per 1,000 output rows. Rows removed by filters or as duplicates are free, saved duplicates are free, file exports are free.

Quick start

Press Start with the example input. It uses 11 fictional companies given inline (no account data needed), normalises the website URLs, extracts the domain, keeps companies with 10 or more employees, removes duplicates by domain, sorts by size and saves a CSV:

{
  "rows": [
    {"company": "Acme Robotics", "website": "https://www.acme-robotics.example/?utm_source=newsletter", "country": "DE", "employees": 120},
    {"company": "ACME Robotics GmbH", "website": "acme-robotics.example", "country": "DE", "employees": 120},
    "... 9 more rows ..."
  ],
  "normalizeUrlFields": ["website"],
  "extractDomainFrom": "website",
  "filters": [{"field": "employees", "op": "gte", "value": 10}],
  "dedupeFields": ["domain"],
  "sortBy": "employees",
  "sortDirection": "desc",
  "exportFormats": ["csv"]
}

The example run writes 6 rows from 11 (2 filtered out, 3 duplicates removed). First row:

{
  "company": "Granite Data Systems",
  "website": "https://granitedata.example?page=2",
  "country": "US",
  "employees": 1500,
  "industry": "Software",
  "domain": "granitedata.example"
}

Merge the output of two Actor runs and a Google Sheet, remove duplicate companies by domain, and keep the removed ones:

{
  "datasetIds": ["WkzbQMuFYuamGv3YF", "Xq3vLmN8pRtY2sKdE"],
  "fileUrls": ["https://docs.google.com/spreadsheets/d/<sheet-id>/edit#gid=0"],
  "renameFields": {"Company Name": "company", "URL": "website"},
  "extractDomainFrom": "website",
  "dedupeFields": ["domain"],
  "outputDuplicates": true,
  "addSourceColumn": true,
  "exportFormats": ["csv", "xlsx"]
}

Add columns from a second table, like VLOOKUP (here: technologies found by a tech-stack run, matched on domain):

{
  "datasetIds": ["<your company list>"],
  "extractDomainFrom": "website",
  "joinDatasetId": "<tech stack results>",
  "joinOn": "domain",
  "joinFields": ["technologies", "cms"],
  "joinType": "left"
}

What it does

Every step is optional. They run in this order:

1. Merge any number of sources: Apify datasets (picked in the input, by ID, or as username~dataset-name), public file links (CSV, TSV, JSON, JSONL / NDJSON, Excel .xlsx; Google Sheets, Google Drive and Dropbox share links are converted to download links) and rows given inline as JSON. addSourceColumn tags each row with where it came from (_source).
2. Clean fields: rename columns so different sources line up, keep only some fields (dots pull nested values into their own column, e.g. company.website), trim spaces, lowercase fields, normalise URLs (lowercase host, no utm_* or click-ID parameters, no fragment, no trailing slash) and extract the domain of a website field into a domain column (https://www.Example.com/about becomes example.com).
3. Filter with conditions {"field", "op", "value"}: eq, neq, contains, not_contains, regex, gt, gte, lt, lte, empty, not_empty, in. All conditions (AND) or any (OR). Text comparisons ignore case by default; gt/lte compare numbers when both sides are numbers ("1,200" counts as 1200) and text otherwise, so ISO dates work. List fields match when any element matches.
4. Remove duplicates by one or more fields, or by the whole row. Keep the first or the last row of each group, ignore case and extra spaces (on by default). Rows whose dedupe fields are all empty are never removed, so rows without a website do not collapse into one. With outputDuplicates, the removed rows go to a separate dataset of the run instead of being dropped.
5. Join (VLOOKUP) with a dataset, a file or inline rows on a key field. Left join keeps every row, inner join keeps only rows with a match. Bring all columns or only the ones you list; a column your rows already have is added with a prefix (right_ by default), never overwritten. The first matching row of the join table is used.
6. Sort by a field (numbers as numbers, empty values last) and limit with maxRows.
7. Output: rows go to the dataset. With exportFormats, the same rows are saved as merged.csv (UTF-8 with BOM, opens cleanly in Excel), merged.xlsx and merged.json in the key-value store, with download links in the OUTPUT record. Lists and objects become JSON text in CSV and Excel cells. Exports are safe to open in Excel: values that look like formulas (starting with =, +, -, @) are written as text, with a ' in front, so nothing in the data runs as a formula.

The OUTPUT record also lists rows read per source, rows filtered out, duplicates removed, join matches, rows written, the link to the removed duplicates and warnings (for example a filter field that no row has).

Use it after a scraper

The Actor reads any dataset, so it fits after any Actor run. Add it as an integration on another Actor or task ("Run Actor" after a successful run) with the input below, and it cleans each new result automatically:

{
  "datasetIds": ["{{resource.defaultDatasetId}}"],
  "dedupeFields": ["url"],
  "normalizeUrlFields": ["url"],
  "exportFormats": ["csv"]
}

Ideas with other Actors by the same developer:

• Career Site Jobs: merge several runs, remove jobs already seen by url, keep only roles matching regex on the title.
• Tech Stack Detector: join its results onto your lead list by domain to add CMS and ecommerce platform columns.
• UK Companies House: combine daily new-company lists, dedupe by companyNumber, filter by SIC code or town.
• Sitemap Extractor: merge the sitemaps of several sites, filter URLs with regex, compare two runs with an inner join.

Limited permissions

The Actor is built to run with limited permissions: it reads only the datasets you give it in the input (the dataset fields are declared as dataset inputs, so the platform grants the run read access to them) and writes only to its own run storages. If a dataset cannot be read, the run stops before writing anything and says which one, and how to pass it.

Input

Field	Default	What it does
datasetIds		Apify datasets to merge (ID, picker, or username~dataset-name)
fileUrls		Public CSV, TSV, JSON, JSONL, XLSX links; Google Sheets, Drive and Dropbox links
rows		Rows as a JSON array (the example rows are ignored when datasets or files are given, see below)
fileFormat, excelSheet	auto, first sheet	Force a file format; pick an Excel sheet
addSourceColumn	false	Add _source to each row
fields, renameFields		Keep only these fields; rename old -> new
trimStrings	true	Trim spaces in text values
lowercaseFields, normalizeUrlFields		Lowercase or normalise these fields
extractDomainFrom		Add a domain column from a website field
filters, filterMode	all	Filter conditions and how they combine
dedupe, dedupeFields	false	Remove duplicates by these fields (or the whole row)
keep	first	Keep the first or last row of each duplicate group
dedupeIgnoreCase	true	Ignore case and extra spaces when comparing
outputDuplicates	false	Save removed duplicates to a separate dataset (free)
joinDatasetId / joinFileUrl / joinRows		The table to look values up in
joinOn, joinRightOn		Key field of your rows, and of the join table
joinType	left	left keeps rows without a match, inner drops them
joinFields, joinPrefix	all, right_	Columns to bring over; prefix for clashing names
sortBy, sortDirection	asc	Sort field and direction
maxRows	0 (all)	Stop after this many output rows
exportFormats		csv, xlsx, json files in the key-value store (free)

What it costs

Run	Price
The example run (6 output rows)	$0.003
Two scraper runs of 5,000 rows each, 3,000 duplicates removed (7,000 output rows)	$3.50
A 50,000-row list filtered down to 8,000 rows, with CSV and Excel files	$4.00

You pay only for rows written to the output dataset. Rows read, filtered out or removed as duplicates, the duplicates dataset and the CSV / Excel / JSON files are free. maxRows caps the price; the run also stops cleanly at your spending limit and still writes the files and summary for the rows it wrote.

Limits and good to know

• Size: without sorting and without "keep last", rows stream through and only a small fingerprint per dedupe key stays in memory, so large datasets (millions of rows) are limited mainly by run time. Sorting and keeping the last duplicate hold all rows in memory: in a local test, 300,000 rows of 10 short fields took about 360 MB, so plan on roughly 500,000 such rows per GB of memory (default 1 GB; raise the memory for more). The join table is held in memory too.
• Files are downloaded to disk (up to 100 MB each) and read in full before anything is written, so a broken file stops the run before any row is charged. Text files can be UTF-8, UTF-16 or Windows-1252. Excel files that unpack to more than about 1 GB are refused. Excel files hold at most 1,048,576 rows; bigger outputs are complete in the dataset, CSV and JSON.
• Datasets are read in pages of 5,000 rows. The run stops about 3 minutes before its timeout and keeps everything written so far; raise the timeout for very large merges.
• Google Sheets must be shared as "Anyone with the link can view" (or published to the web). The tab in the link (gid) is the one read.
• Values from CSV and Google Sheets are text; filters and sorting still compare numbers as numbers.
• Only public file links are downloaded; private and internal network addresses are blocked. Links to (and redirects through) YouTube, TikTok, Instagram, Facebook, X/Twitter, LinkedIn, Reddit and the Amazon store are refused; files on Amazon S3 / AWS are fine.
• The regex filter gives up on a value after 0.05 seconds (a pattern with nested repeats like (a|aa)+ can take forever) and counts it as no match; the OUTPUT warnings say how often that happened.
• The example settings: the input form opens with example rows and example settings (URL clean-up and a domain column from website, a filter on employees, dedupe by domain, sort by employees, a CSV file). When you add your own datasets or file links and leave the example rows in place, the example rows and every example setting still at its example value are ignored, and the log says which. To use one of those settings on your own data, clear the example rows first.

FAQ

Can it merge datasets with different column names? Yes: rename the columns with renameFields so they line up. Columns that exist in only some sources stay, and are empty in the other rows.

Which duplicate is kept? The first one in source order (sources are read in the order: datasets, files, inline rows), or the last one with keep: "last". Put the source you trust most first.

Can an AI agent use it? Yes, through the Apify MCP server: give it rows inline or dataset IDs and ask for "dedupe by domain and save as CSV".

More from this developer

• Career Site Jobs: open jobs from company career sites (Greenhouse, Lever, Ashby, Workable, Recruitee, Workday).
• Tech Stack Detector: CMS, ecommerce platform, analytics and hosting of a list of websites.
• UK Companies House: UK company profiles and new-company lists from the official API.
• Sitemap Extractor: every URL a website lists in its sitemaps.
• Bulk HTTP Requests: call any API once for every row of a CSV, JSON list or Apify dataset.

Code examples (curl, Python, Node.js, n8n) for this and the other Actors: github.com/msalihk/apify-actor-examples.

Questions or a missing feature: open an issue on the Actor page.