Merge Datasets & Remove Duplicates: CSV, JSON, Apify

Merge datasets and deduplicate rows in one step: combine the output of several scraper runs or files into one clean table.

Merge Apify datasets and CSV, JSON or JSON Lines files into one clean dataset: remove duplicates by the fields you choose, rename columns, filter rows, and keep or drop fields. It streams large datasets page by page and keeps going when one source is broken, so a single bad input doesn't sink the whole run.

Quick start for AI agents and API users

• What it does: Merge Apify datasets and CSV/JSON/JSONL files into one dataset and remove duplicates.
• When to use it: To combine scraper outputs or exports.
• Example input (the form's prefill; it finishes in under a minute):

{
  "fileUrls": [
    "https://raw.githubusercontent.com/datasets/country-list/main/data.csv",
    "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
  ],
  "dedupFields": [
    "code"
  ],
  "renameFields": {
    "Code": "code",
    "Name": "name",
    "ISO3166-1-Alpha-2": "code",
    "official_name_en": "name"
  },
  "filters": [
    {
      "field": "code",
      "op": "startsWith",
      "value": "A"
    }
  ],
  "keepFields": [
    "code",
    "name"
  ],
  "addSourceField": true,
  "maxItems": 50
}

• Example output: one real dataset row from run h384xdQMlabUo2zJf (2026-10-23), with long fields shortened and empty fields left out. Fields not shown are listed in the output table below.

{
  "code": "AF",
  "name": "Afghanistan",
  "_source": "https://raw.githubusercontent.com/datasets/country-list/main/data.csv"
}

• Cost (pay per event, Free-plan prices): Row $0.001. Failed items are not charged.
• From an AI agent: with the Apify MCP server, call call-actor with actor phaux/dataset-merge-dedup, or load it as its own tool with https://mcp.apify.com?tools=phaux/dataset-merge-dedup. Read results with get-dataset-items.

What does Dataset Merge & Dedup do?

• Merges any number of datasets (picked in Console, or by ID via API) and file URLs, in the order you give them.
• Removes duplicates by one or more fields (dedupFields, nested dot paths like address.city work), or drops rows that are identical in every field. The first row wins; 1, 1.0 and "1" count as the same value, and case can be ignored.
• Renames fields first, so differently named columns from different sources line up ({"Company Name": "company", "company_name": "company"}).
• Filters rows with simple conditions: eq, neq, gt, gte, lt, lte, contains, notContains, startsWith, endsWith, regex, in, notIn, exists, notExists, empty, notEmpty.
• Keeps or drops fields (keepFields in a fixed column order, or omitFields), and can add a _source column.

Why use it?

• Reliable: clear input checks before anything runs, a per-source error report (SOURCE_ERRORS) instead of a crashed run, and a time guard that saves results before the timeout.
• Fast: reads 5,000 rows per request and writes in batches of 1,000; merging 9,600 rows from three datasets took about 4 seconds in testing.
• Limited permissions: the run can read only the datasets you pick, read-only.
• Pay only for rows kept: duplicates and filtered-out rows are free.

Use cases: combining runs of the same scraper (daily, per region or per keyword), de-duplicating leads or products across runs, aligning CSV exports from different tools, and cleaning data before sending it to a spreadsheet, CRM or database.

How to use it

1. Pick one or more datasets under Apify datasets and/or paste file URLs under CSV, JSON or JSONL file URLs. Empty input runs a small example with two public country-code CSV files.
2. Optionally map column names in Rename fields, set Deduplicate by fields, and add Filters.
3. Choose Keep only these fields for a tidy table.
4. Run it and download the merged dataset as CSV, Excel or JSON.

Input

{
  "fileUrls": [
    "https://raw.githubusercontent.com/datasets/country-list/main/data.csv",
    "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"
  ],
  "renameFields": {"Code": "code", "Name": "name", "ISO3166-1-Alpha-2": "code", "official_name_en": "name"},
  "dedupFields": ["code"],
  "filters": [{"field": "code", "op": "startsWith", "value": "A"}],
  "keepFields": ["code", "name"],
  "addSourceField": true,
  "maxItems": 50
}

Field	Type	Default	Description
datasetIds	array	—	Datasets to merge, in order. In Console pick them with the dataset picker; via API pass dataset IDs (or 'username~dataset-name'). Only these datasets can be read by the run (read-only access).
fileUrls	array	—	Public URLs of CSV (any delimiter), JSON (array, or object with an items/data list) or JSON Lines files. Each is downloaded once, up to maxFileSizeMb.
dedupFields	array	—	Rows with the same values in all these fields count as duplicates; the first one is kept. Dot paths work for nested fields (e.g. 'address.city'). Rows where all of these fields are empty are always kept. Leave empty for no field-based dedup.
dedupWholeItem	boolean	false	When no dedup fields are set: drop rows that are identical in every field.
caseInsensitive	boolean	false	Ignore letter case (and surrounding spaces) when comparing dedup keys and text filters.
renameFields	object	—	Map of old top-level field name to new name, e.g. {"Company Name": "company"}. Applied first. Map several source names to one new name to align differently named columns.
filters	array	—	Keep only rows matching these conditions, e.g. [{"field": "price", "op": "gt", "value": 10}, {"field": "email", "op": "notEmpty"}]. Ops: eq, neq, gt, gte, lt, lte, contains, notContains, startsWith, endsWith, regex, in, notIn, exists, notExists, empty, notEmpty. Numbers in text are compared as numbers.
filterMode	string	"all"	all = a row must match every filter; any = at least one. Allowed: all, any.
keepFields	array	—	Output only these fields, in this order (dot paths allowed; missing values become null). Leave empty to keep all fields.
omitFields	array	—	Top-level fields to remove (ignored when keepFields is set).
addSourceField	boolean	false	Add the dataset ID or file URL each row came from.
sourceFieldName	string	"_source"	Name of the source field.
maxItems	integer	0	Stop after this many rows are saved (0 = no limit). Also caps the cost of a run.
skipHiddenFields	boolean	false	Drop fields starting with # from dataset rows (Apify's hidden-field convention).
csvDelimiter	string	"auto"	auto detects , ; tab or |. Or give one character (use 'tab' for tab).
maxFileSizeMb	integer	100	Per file URL (1–500). Large JSON arrays need about 10x their size in memory; JSON Lines and CSV need less. Raise the run memory for big files.
failOnSourceError	boolean	false	Off: a missing dataset or broken file is reported in SOURCE_ERRORS and the other sources are still merged. On: the run fails at the first broken source.
timeBudgetSecs	integer	0	Stop reading after this many seconds and keep the rows saved so far (0 = only the run timeout guard).

Output

Each saved row keeps your own fields (after renaming, keepFields and omitFields). With addSourceField on, _source holds the dataset ID or file URL of the row. Example with the prefilled input:

{"code": "AF", "name": "Afghanistan", "_source": "https://raw.githubusercontent.com/datasets/country-list/main/data.csv"}

The key-value store also gets RUN_STATS (rows read, filtered out, duplicates and saved, per source) and SOURCE_ERRORS (datasets or files that could not be read, with the reason).

How much does it cost?

Pay per event. Apify platform usage is included.

Apify plan	Price per output row	Per 1,000 rows
Free	$0.001	$1.00
Bronze	$0.0009	$0.90
Silver	$0.0008	$0.80
Gold, Platinum, Diamond	$0.0007	$0.70

Plus an Actor start fee of $0.00005 per GB of memory (default 512 MB = one event). Duplicates and filtered-out rows are not charged. Use Max output rows to cap the cost of a run.

Use with AI agents, MCP and the API

• MCP: add https://mcp.apify.com?tools=phaux/dataset-merge-dedup to an MCP client and ask, for example, "merge these two datasets and remove duplicate URLs".
• API: POST https://api.apify.com/v2/acts/phaux~dataset-merge-dedup/run-sync-get-dataset-items with {"datasetIds": ["..."], "dedupFields": ["url"]}.
• After another Actor: call it from an integration or webhook with the other run's defaultDatasetId.

Data source and compliance

• Reads only your own data: the datasets you pick (read-only access granted per run) and file URLs you provide.
• File URLs are fetched once each, with robots.txt honoured for the user agent PhauxDatasetMerge, private and internal addresses refused, a size cap (maxFileSizeMb), and no further requests to a host after HTTP 429.
• No data is kept outside your run's storage.

Limits

• dedupFields keeps the first occurrence; it doesn't merge values from duplicate rows.
• JSON arrays are loaded whole, so a big JSON file needs about 10× its size in memory; CSV and JSON Lines need much less. Raise the run memory (up to 4 GB) for large files.
• Dedup keys are held in memory (about 100 bytes per unique row), so tens of millions of rows need more memory.
• Rename works on top-level field names; keepFields, dedupFields and filters accept dot paths for nested values.

FAQ

• "Dataset doesn't exist or isn't shared with this run"? Pick the dataset with the picker in Console, or pass its exact ID (or username~dataset-name) in datasetIds.
• Why weren't two similar rows deduplicated? The values differ, for example in letter case: switch on Case-insensitive matching. Leading and trailing spaces in dedupFields values are always ignored.
• Can I export CSV or Excel? Yes: export the default dataset from the run's Storage tab, or use the "Merged rows as CSV" output link.
• Something isn't working? Open an issue on the Issues tab with the run ID.