Merge & Deduplicate Datasets: Only New Items, CSV/Excel Export

Merge several Apify datasets and CSV, JSON, JSON Lines, Excel or Parquet files into one table, remove duplicates (exact, case/whitespace-insensitive or fuzzy), output only new items since your last run, join / VLOOKUP a second dataset or CSV/Excel file by key, filter, sort, pick and rename columns, and export to CSV, Excel (XLSX), JSON, JSONL or Parquet with download links. Large inputs are staged on disk, not in memory. $0.002 per 1,000 rows.

What people use it for

Use case	How
Combine the outputs of several scraper runs into one clean list	datasetIds (pick them) + dedupe on url, email or id
Scheduled scraper, only new items each day	onlyNew: true + dedupeFields: each run passes on only rows earlier runs did not deliver
CSV to JSON, JSON to CSV, dataset to Excel	fileUrls or datasetIds + exportFormats: ["xlsx", "csv"]
Clean a lead list before a CRM	dedupe: normalized on email, keep: most-complete, filter: "not email empty"
Fuzzy duplicates (typos, word order)	dedupe: fuzzy on company + city
Join / VLOOKUP: enrich rows from a second list (CRM export, price list)	joinFileUrl or joinDatasetId + joinOn: ["email"]
Rows not in another list (new leads vs. existing customers)	joinType: anti + joinOn

Quick start (real output)

The form's prefill reads a public 249-row CSV of countries, keeps Europe, deduplicates by currency, counts how many countries share each one, and exports CSV + Excel. It finishes in a few seconds and costs $0.013 (one 1,000-row block, two files and the $0.001 start fee):

{
  "fileUrls": ["https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"],
  "filter": "Continent = \"EU\"",
  "dedupe": "normalized", "dedupeFields": ["ISO4217-currency_alphabetic_code"],
  "duplicateCountField": "countries", "sortBy": ["-countries"],
  "fields": ["CLDR display name", "ISO4217-currency_alphabetic_code", "Capital", "countries"],
  "rename": {"CLDR display name": "country", "ISO4217-currency_alphabetic_code": "currency"},
  "exportFormats": ["csv", "xlsx"]
}

[
  {"country": "Åland Islands", "currency": "EUR", "Capital": "Mariehamn", "countries": 27},
  {"country": "Guernsey", "currency": "GBP", "Capital": "St Peter Port", "countries": 4},
  {"country": "Denmark", "currency": "DKK", "Capital": "Copenhagen", "countries": 2},
  ...
]

Status: 249 rows read, 197 filtered out, 32 duplicates removed, 20 rows out, 2 files exported. The SUMMARY record in the key-value store has the counts and the download links of output.csv and output.xlsx.

Only new items: add "onlyNew": true (without the filter) and run twice. The second run reports 249 rows read, 246 already seen in earlier runs and outputs just the 3 rows that have no currency (rows with empty match fields are never treated as seen).

Pricing in plain words

$0.002 per started 1,000 rows read (filtered, duplicate and already-seen rows count, they were processed) plus $0.005 per export file plus a $0.001 start fee per run (at the default memory). 100,000 rows deduplicated and exported to Excel = $0.206. Rows of a join's lookup source count as rows read. An input that cannot be read and an export that fails cost nothing. Only-new memory, joins, fuzzy matching and filtering cost nothing extra. The Pricing tab always has the current prices.

Why this one

• Only new items, at scale. The memory of delivered rows lives in a key-value store in your account, staged on disk while the run works, so it keeps up to 5 million rows without needing more memory. One memory per onlyNewKey.
• Finishes the job. Every input is read independently: a missing dataset or a broken file gets a clear error in the run summary while everything else is still processed.
• Dedupe that matches how data really looks. Exact, normalized (Ann@X.com = ann@x.com) or fuzzy (Acme Corporation Ltd = ACME Corporaton Ltd.), on any fields or the whole row. Keep the first, the last or the most complete row, and optionally count how many duplicates each row had.
• Join / VLOOKUP by key. Merge two datasets or files by key: left join (enrich every row), inner join (only matches) or anti join (rows not in the other list), with case-insensitive or fuzzy key matching and unmatched counts in the summary. The lookup is staged on disk, so it can have millions of rows.
• Filter without code. price >= 10 and country in ["CA", "US"] and not email empty. Parsed safely, never executed.
• Any format in, any format out. CSV/TSV (delimiter and encoding detected), JSON arrays, JSON Lines, XLSX, Parquet, gzip. Out: the run's dataset plus CSV, XLSX, JSON, JSONL and Parquet files.
• Pay per 1,000 rows, not per row.

Input

{
  "datasetIds": ["aBcDeFgHiJkLmNoPq", "nXyZ0123456789abc"],
  "fileUrls": ["https://example.com/leads.csv"],
  "sourceField": "_source",
  "flatten": "objects",
  "filter": "country in [\"CA\", \"US\"] and not email empty",
  "dedupe": "normalized",
  "dedupeFields": ["email"],
  "keep": "most-complete",
  "duplicateCountField": "duplicates",
  "sortBy": ["-rating", "name"],
  "fields": ["name", "email", "phone", "address.city", "rating", "duplicates"],
  "rename": {"address.city": "city"},
  "exportFormats": ["csv", "xlsx"]
}

Processing order: read the join lookup (if any) → read and merge (inline items, then datasets, then files) → flatten → add the source column → join → filter → dedupe → sort → limit → select / omit / rename → output. So filter, dedupe and sort use the original (flattened) field names and can use joined columns, and rename comes last.

Field	What it does
datasetIds	Apify dataset IDs (or username~dataset-name). Yours, shared with you, or public.
fileUrls	Public file URLs: CSV, TSV, JSON, JSONL/NDJSON, XLSX (first sheet), Parquet, optionally .gz.
items	Rows inline, as a JSON array (handy for agents).
fileFormat, csvDelimiter	Override detection.
sourceField	Column naming each row's input (inline, dataset:<id> or the URL).
flatten, flattenSeparator	objects: {"address": {"city": "X"}} → address.city. all: arrays too (tags.0).
filter	Rule language below.
dedupe, dedupeFields, fuzzyThreshold, keep, duplicateCountField	Duplicate removal.
joinDatasetId, joinFileUrl, joinItems	One lookup source to join with (see Join / VLOOKUP below).
joinOn, joinType, joinFields, joinMatch, joinPrefix	Key fields (email or email=Email Address), left / inner / anti, columns to bring in, normalized / exact / fuzzy keys, prefix for clashing names (joined_).
onlyNew, onlyNewKey	Leave out rows delivered by earlier runs with the same memory name (see below).
sortBy, limit	-price or price:desc for descending; :number / :text force a type.
fields, omitFields, rename	Output columns.
exportFormats, exportFileName	Files to write (output.csv, ...).
datasetOutput	rows (default), summary (one item with counts and links) or none.

Filter language

price >= 10 and (country = "CA" or country in ["US", "MX"])
title matches "(?i)^(senior|lead)" and not email empty
tags contains "vip" or `Company Name` startswith "acme"
address.city exists

• Comparisons: = != < <= > >=. With a number (price > 10) the field is compared numerically, also when it is text like "12.50". With text (date >= "2024-01-01") it's compared as text, so ISO dates work.
• contains (substring, or list membership), startswith, endswith: ignore case.
• matches: Python regular expression; add (?i) to ignore case.
• in [..], exists (present and not null), empty (missing, null, blank text, empty list/object).
• Combine with and, or, not (or &&, ||, !) and parentheses. Keywords are case-insensitive.
• A condition on a missing field is false, except !=, which is true.
• Field names with spaces or that are keywords go in backticks. Nested fields: address.city, items.0.sku.

Join / VLOOKUP (merge two datasets by key)

Give one lookup source (joinDatasetId, joinFileUrl or joinItems) and the key in joinOn. Each row is matched with the first lookup row that has the same key, like VLOOKUP, so a join never adds rows.

{
  "datasetIds": ["aBcDeFgHiJkLmNoPq"],
  "joinFileUrl": "https://example.com/crm-export.csv",
  "joinOn": ["email=Email Address"],
  "joinType": "left",
  "joinFields": ["Plan", "Owner"]
}

• joinOn: email when both sides call it that, email=Email Address when the lookup names it differently. Several entries are combined into one key (["first=First Name", "last=Last Name"]).
• joinType: left keeps every row and adds the lookup's columns to matched ones; inner keeps only matched rows; anti keeps only rows with no match (e.g. scraped leads that are not yet in your CRM).
• joinFields: lookup columns to add (default: all but the key). A column the row already has is added as joinPrefix + name (joined_name), so nothing is overwritten.
• joinMatch: normalized (default; ignores case, extra spaces, Unicode width), exact, or fuzzy (RapidFuzz with fuzzyThreshold, the same matcher as fuzzy dedupe, e.g. Acme Corporaton Ltd = ACME Corporation Ltd.). Rows whose key fields are all empty never match.
• The lookup file's format is detected (fileFormat and csvDelimiter are for the main files). It is read first and staged on disk, so it can be large. If it cannot be read completely, the main inputs are not read (JOIN_SOURCE_FAILED) and nothing more is charged.
• SUMMARY has join: the lookup's status, lookupRowsRead, lookupDuplicateKeys, lookupRowsWithoutKey, rowsMatched, rowsUnmatched, rowsRemovedByJoin and lookupRowsNeverMatched.

Only new rows

With onlyNew: true, rows that an earlier run with the same onlyNewKey (default default) delivered are left out, and this run's delivered rows are remembered. Rows are matched on dedupeFields (or the whole row, without sourceField): exactly when dedupe is none or exact, ignoring case and spaces when it is normalized or fuzzy (fuzzy matching is not applied across runs). Rows removed by filter or cut by limit are not remembered, so a later run still delivers them. Rows whose match fields are all empty are always new.

The memory is the key-value store toolkit-only-new-<onlyNewKey> in your account (64-bit keys, about 8 MB per million rows; up to 5 million rows, the oldest are forgotten first). Delete it, or use a new onlyNewKey, to start over. SUMMARY has onlyNew: rowsAlreadySeen, rememberedBefore, newKeysRemembered, remembered. If the memory cannot be read, the run fails before reading or charging anything.

Dedupe modes

• exact: values must be identical (1 and "1" differ).
• normalized: ignores letter case, repeated or surrounding whitespace and Unicode width (1 = "1").
• fuzzy: normalized text compared with RapidFuzz token sort ratio; a row joins the first earlier group scoring at least fuzzyThreshold (default 90). Catches typos, punctuation and word order. To stay fast on large data, rows are only compared with groups whose text starts with the same character, so a typo in the very first letter is not caught.

With several dedupeFields, their values are combined into one key. Without dedupeFields, the whole row is compared (ignoring the source column). Rows whose dedupe fields are all empty are never removed.

Output

• Dataset: the final rows (or one summary item with datasetOutput: "summary").
• Key-value store:
  • SUMMARY: counts and per-input status, for example:

    {"rowsRead": 12000, "rowsFiltered": 1500, "duplicatesRemoved": 830, "rowsCutByLimit": 0, "rowsOut": 9670,
     "sources": [{"source": "dataset:aBcDeFgHiJkLmNoPq", "kind": "dataset", "status": "ok", "rowsRead": 10000},
                 {"source": "https://example.com/leads.csv", "kind": "url", "status": "error", "rowsRead": 0,
                  "error": {"code": "HTTP_ERROR", "message": "The server answered HTTP 403 ..."}}],
     "exports": [{"format": "csv", "key": "output.csv", "status": "ok", "rows": 9670, "bytes": 1203311,
                  "url": "https://api.apify.com/v2/key-value-stores/.../records/output.csv"}],
     "stoppedReason": null, "charged": {"rows-processed-1000": 12, "file-exported": 1}}
    

  • The export files. CSV, XLSX and Parquet have every column seen, in first-seen order; nested values are written as JSON text. Parquet column types are inferred (integer, float, boolean, otherwise text).

Input statuses: ok, partial (rows before the error are kept), error, skipped (spending limit). Error codes: DATASET_NOT_FOUND, ACCESS_DENIED, DATASET_READ_FAILED, HTTP_ERROR, DOWNLOAD_FAILED, BLOCKED_ADDRESS, INVALID_URL, FILE_TOO_LARGE, PARSE_ERROR, UNSUPPORTED_FORMAT, SPENDING_LIMIT_REACHED, JOIN_SOURCE_FAILED; exports: EXPORT_FAILED, TOO_MANY_COLUMNS. The run fails only if the input itself is invalid (e.g. no sources); if every input fails, the run still succeeds and its status message says why. NaN/Infinity numbers are output as null, and a row over Apify's 9 MB item limit is skipped from the dataset (counted in rowsNotPushed) but kept in export files. Rows not written to the dataset because the run's maximum charge was reached are counted there too (SPENDING_LIMIT_REACHED).

Pricing (pay per event)

Event	Price	When
rows-processed-1000	$0.002	Each started block of 1,000 input rows read (rows later filtered out or removed as duplicates count)
file-exported	$0.005	Each export file written
apify-actor-start	$0.001 per started GB of run memory (from 2026-10-30)	Once per run, charged by Apify. The default 256 MB run counts as 1; Apify covers the first 5 seconds of compute in return

Example: dedupe 50,000 rows from two datasets and export a CSV and an XLSX at the default 256 MB = 50 × $0.002 + 2 × $0.005 + $0.001 start = $0.111. Inputs that fail to load and failed exports are not charged. With a maximum charge per run, the Actor checks the budget before each block of 1,000 rows (keeping enough back for the exports you asked for), stops reading when it runs out and still delivers what it read, with stoppedReason: "SPENDING_LIMIT_REACHED".

Limits

• Rows are staged on the run's local disk, so the row count is limited by disk, not memory. Measured: about 15,000 to 25,000 rows/s; 1,000,000 rows with dedupe, sort and CSV + XLSX + Parquet exports in about 70 s with ~300 MB peak memory at 1 GB.
• Memory: the default is 256 MB, which suits the typical job (up to a few hundred thousand rows; measured: 860,000 rows with normalized dedupe, sort and a CSV export finished at 256 MB in about 10 minutes, at 512 MB in 5 minutes, for the same compute). For big jobs raise Run options > Memory: 1 GB for fuzzy dedupe or XLSX export of large inputs, 2 to 4 GB for multi-million-row runs (disk scales with memory). Compute per row costs about the same at any size; more memory mostly makes it faster. If memory gets nearly full, reading stops before the run could crash, the rows read so far are delivered and the summary says stoppedReason: "MEMORY_LOW". Below 1 GB the fuzzy group cap is reduced in proportion (minimum 50,000).
• Files: up to 1 GB each (after download). JSON files whose top level is an object (not an array) are loaded whole and limited to 1/16 of the run's memory (16 MB at 256 MB, up to 100 MB); use JSON arrays or JSON Lines for bigger files.
• Fuzzy dedupe keeps up to 300,000 groups in memory (at 1 GB or more) and compares each row with up to 20,000 groups that start with the same character; beyond that, later rows are matched only against known groups (the summary warns). Fuzzy runs at about 10,000 rows/s on typical data. For millions of rows prefer normalized.
• XLSX holds at most 1,048,575 rows and 16,384 columns (the file stops there; other formats have everything). CSV, XLSX and Parquet exports allow up to 10,000 columns. Cells longer than 32,767 characters are cut in XLSX.
• Only the first sheet of an XLSX workbook is read; old .xls is not supported. Datasets are read with hidden (#-prefixed) fields and empty items skipped.
• At most 100 inputs per run. File URLs must be public (private and reserved network addresses are refused).

Use from an AI agent (MCP)

Through the Apify MCP server, an agent can pass rows as items or point at datasets from other Actor runs, and set datasetOutput: "summary" to get counts and download links instead of every row.

Privacy

Your data is processed only inside your run and stored only in your run's storages.

Open-source licenses

Built on permissively licensed software: Apify SDK (Apache-2.0), HTTPX (BSD-3-Clause), RapidFuzz (MIT), Apache Arrow / pyarrow (Apache-2.0), openpyxl (MIT) and SQLite (public domain).