Merge Datasets & Dedupe: Combine, Filter, Export CSV & Excel

Merge datasets from any Apify run, plus CSV, JSON, JSONL and Excel files and pasted text, into one clean dataset. This dataset merge Actor can dedupe rows by key fields or by a full-row hash, apply dataset filter rules (equals, contains, regex, greater/less than, between, exists, empty, AND/OR), flatten nested fields, pick and rename columns, sort, limit, and write CSV, Excel (.xlsx), JSON and JSONL export files with public links. Use it to combine datasets from several scraper runs, to merge CSV exports from different tools, or to clean up a dataset before you send it somewhere else. Nothing to install and no API key: it reads other datasets with your own run's access.

What can you merge

• Apify datasets from any Actor. Pick datasets in the input form, or paste dataset IDs, username~name names or dataset links. Your own runs and other creators' runs all work, as long as you can read them.
• CSV, TSV, JSON, JSONL and Excel (.xlsx) files from a direct link. The type is detected from the extension or the content. CSV delimiters , ; tab |, UTF-8 and BOM are handled.
• Pasted text. Put CSV, JSON or JSONL straight into the input (inlineSources). It works offline and is handy for tests and for AI agents that already hold the data.
• Sources have different columns? The result has all of them. A row simply has no value for a column its source did not have.

How to merge datasets

1. Add your sources: Datasets to merge, Dataset links or IDs, File URLs and/or Pasted data. They are read in that order.
2. Choose how to dedupe: by key fields (email, url, firstName + lastName...), or identical rows only.
3. Optionally add filter rules, columns to keep, renames and a sort order.
4. Pick export files (CSV, XLSX, JSON, JSONL) if you want downloadable files. Run it.
5. Read the rows from the run's dataset, and the file links and counts from the SUMMARY record.

Dedupe datasets

• By key fields: rows with the same value in all dedupeKeys are duplicates. Keys are trimmed, and 42 and "42" count as the same value. Turn on ignore upper/lower case so ANNA@x.com equals anna@x.com. Dots reach nested fields (author.id).
• First or last wins: keep the first row seen (earlier sources win) or the last one. Keep-first streams through the data and stores only a 12-byte hash per row. Keep-last holds the surviving rows in memory.
• Full-row hash: only rows that are identical in every field are removed (key order does not matter, 1 equals 1.0).
• Rows with no key: if all key fields of a row are empty or missing, the row is compared by its full content instead, so such rows are never all collapsed into one. They are only removed when they are exact copies of each other.
• This is exact matching. It does not detect near-duplicates such as "Jon Smith" and "John Smith".

Dataset filter rules

Each rule is {"field": "...", "op": "...", "value": ...}. Rules are combined with AND (default) or OR. Filters run before dedupe, on the original field names (or the flattened names when Flatten is on).

Operator	Keeps a row when...
equals, notEquals	the value equals (or does not equal) value. Numbers match numbers (5 = "5"), text ignores case unless Case-sensitive filters is on. For a list field, any element may match.
contains, notContains, startsWith, endsWith	the text contains / starts with / ends with value (or any item of a list of values).
regex	the text matches the regular expression (case-insensitive by default).
gt, gte, lt, lte	the number or ISO date compares as stated. "$1,299.50" and "2024-05-31" are understood. Text that is not a number or date never matches.
between	the number or date is inside [min, max] (inclusive). Use null to leave one side open.
in	the value equals any item of the list.
exists, notExists	the field is present with a non-null value / absent or null.
empty, notEmpty	the field is missing, null, blank text, [] or {} / is not.

Input example

{
  "datasetIds": ["aBcD1234eFgH5678i", "kLmN9012oPqR3456s"],
  "fileUrls": ["https://example.com/exports/leads.csv"],
  "inlineSources": [{"name": "manual", "format": "csv", "text": "email,name\nzoe@example.com,Zoe\n"}],
  "dedupeMode": "keys",
  "dedupeKeys": ["email"],
  "dedupeCaseInsensitive": true,
  "filters": [
    {"field": "plan", "op": "notEquals", "value": "free"},
    {"field": "signup", "op": "between", "value": ["2024-01-01", "2024-12-31"]}
  ],
  "flatten": true,
  "fields": ["email", "name", "company.name"],
  "renameFields": {"company.name": "company"},
  "sortBy": ["signup:desc"],
  "exports": ["csv", "xlsx"]
}

The default input of the Actor needs no datasets: it merges two small pasted CSV lists with overlapping emails, so you can see dedupe, a filter, a sort and both export files in one quick run.

Output

One dataset row per merged row, with the columns of your sources (or only the ones you chose):

{
  "email": "elena@example.com", "name": "Elena Rossi", "country": "IT", "plan": "pro",
  "signup": "2024-04-12", "company": "Initech", "source": "newsletter-list"
}

The SUMMARY record in the key-value store has the run report:

{
  "rowsIn": 8, "rowsFiltered": 3, "duplicatesRemoved": 2, "rowsOut": 3,
  "sources": [{"name": "crm-export", "type": "inline", "rowsRead": 4, "status": "ok"}],
  "errors": [],
  "exports": [{"format": "csv", "key": "merged.csv", "url": "https://api.apify.com/v2/key-value-stores/.../records/merged.csv", "rows": 3, "bytes": 242}],
  "stoppedBySpendingLimit": false, "inputCapReached": false
}

• sources lists every source with its row count and a status: ok, truncated (a cap was reached), skipped or error. A failing source never stops the others: its reason is in errors and the run goes on.
• Export files are saved in the key-value store as merged.csv, merged.xlsx, merged.json and merged.jsonl (change the base name with Export file name). They contain exactly the rows that were delivered.
• CSV files are UTF-8 with a BOM so Excel opens them correctly. Nested values and lists are written as JSON text in CSV and Excel cells.

Merge CSV and Excel files

Give direct links to .csv, .tsv, .json, .jsonl or .xlsx files. Excel files are read from the first sheet (or the sheet you name), the first non-empty row is the header, dates become ISO text and formulas are read as the values Excel saved. CSV numbers and booleans are converted when that is unambiguous; 007, +1555 and very long digit strings stay text. Links to localhost, internal names and private networks are blocked, including through redirects.

Merge datasets API

Call this Actor from the Apify API, Python, JavaScript, Make, Zapier or n8n, for example right after a scraper run. Example with the Apify Python client:

from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("data-done-right/merge-datasets").call(run_input={
    "datasetIds": ["aBcD1234eFgH5678i", "kLmN9012oPqR3456s"],   # default datasets of two runs
    "dedupeKeys": ["url"],
    "exports": ["csv"],
})
rows = client.dataset(run["defaultDatasetId"]).list_items().items

In the Apify Console you can also add it as an integration of a scraper (Integrations, "Run Actor") and set the dataset to {{resource.defaultDatasetId}}.

Use it in Claude, ChatGPT and Cursor

Connect Apify's MCP server (mcp.apify.com) and your AI assistant can combine data for you. For example: "merge these three dataset IDs, dedupe by url, keep only rows where price is under 50, and give me an Excel file". The inputs are datasetIds (or datasetUrls), dedupeKeys, filters and exports.

Use cases

• Combine scraper runs: merge today's and yesterday's runs, or the same scraper run in several countries, and drop the repeats.
• Dataset cleanup: filter out rows with empty fields, keep only the columns you need, rename them and sort the result.
• Mixed sources: join a CRM export (CSV), a partner's Excel sheet and a scraper dataset into one deduplicated list.
• Reports and handoffs: produce a CSV or Excel file with a public link for someone who does not use Apify.

How much does it cost

Pay per event: $0.0003 per output row (row), and $0.005 per export file (export-file) when you ask for exports. You are charged for the rows that end up in the result, once each. Rows that were read, filtered out or removed as duplicates are free, and so are the SUMMARY record and sources that fail. 10,000 merged rows cost about $3.00, plus $0.005 for each export file. Paid Apify plans get 10% (Silver) to 20% (Gold and up) off. If you set a maximum spend per run, the Actor stops cleanly when it is reached and keeps the rows already delivered. Max rows to read and Max output rows cap what a run can cost.

FAQ

How do I merge datasets from two scraper runs?

Pick both datasets in Datasets to merge (or paste their IDs), set the key that identifies a row in Dedupe key fields and run. The Actor reads both through your run's access and writes one deduplicated dataset.

Can I merge a dataset with a CSV or Excel file?

Yes. Add the dataset and the file link in the same run. Columns are united: a row gets a value only for the columns its source had.

Which duplicate is kept?

The first one seen by default, in source order (datasets, then dataset links, then files, then pasted text). Choose Last seen to keep the latest instead.

Can it read datasets from other creators' Actors?

It can read any dataset your Apify account is allowed to read, including runs of other creators' Actors that you started. For datasets that are not yours, pick them in the Datasets to merge field: that is what gives a limited-permission run read access. A dataset given only as a link or ID in Dataset links or IDs can fail on a limited-permission run. The failure is reported for that source and the run continues.

What happens when one source fails?

It is listed in errors with the reason (not found, no access, bad file, blocked address, too large) and the other sources are still merged. If every source fails, or all sources are empty, the run fails and nothing is charged.

Is this an Apify dataset merger or a SQL join?

It is a merger (rows are stacked and deduplicated). It does not join two datasets on a key to combine their columns.

How big can the input be?

By default it reads up to 100,000 rows in total; raise Max rows to read up to 2,000,000. Keep-first dedupe streams and keeps only hashes. Sorting and keep-last dedupe hold the surviving rows in memory (the Actor has 1 GB by default), so use a smaller cap or more memory for very wide rows. Files are limited to 50 MB by default (500 MB max), and a single JSON array file is loaded whole and limited to 100 MB (use JSONL or CSV for more).

Why did my CSV export put an apostrophe in front of some cells?

A text cell starting with =, +, - or @ can run as a formula when opened in Excel or Sheets. The CSV export prefixes such text with an apostrophe (plain numbers are left alone). Switch off Protect CSV from spreadsheet formulas if you need the raw text. Excel (.xlsx) exports always store such text as plain text, and the dataset, JSON and JSONL are never changed.

Does it work with Google Sheets?

Use a published CSV link or export the sheet as CSV or XLSX. A link to the normal Google Sheets page returns a web page, which the Actor rejects with a clear message.

Limits

• Exact duplicates only (after trimming, and optional case folding). No fuzzy matching, no joins.
• Old .xls files are not supported (save as .xlsx or CSV). CSV is read as UTF-8.
• Excel exports are limited to 1,048,575 rows per sheet and 32,767 characters per cell (longer text is cut). When a result is too big for Excel, the other export formats still work.
• Regular expressions run on your own data and are limited to 1,000 characters.
• Filters, dedupe and sort use the original (or flattened) field names. Selecting and renaming columns is the last step.
• If you run it with limited permissions, datasets you do not own must be picked in Datasets to merge.

Related tools

• Google Sheets Import & Export: Apify Data, No Login: send the merged dataset to a Google Sheet.
• Slack Notifications: Send Data to Slack, Discord & Email: send the new rows to your team.
• HTTP Request Runner: Bulk API Calls, Retries, Proxies: fetch API data to merge.