Merge, Deduplicate & Transform Datasets

Merge, Deduplicate & Transform Datasets combines datasets from your actor runs into one, in the order you list them, and removes duplicate rows. Rows match on the fields you choose: exactly, ignoring spaces and letter case, as web addresses without tracking parameters, or by similar text. Filters, field selection, renaming and flattening run in the same pass. Pick the datasets in Datasets, enter a field such as url in Deduplicate by fields and click Start. It reads datasets only, never key-value stores or request queues. It costs $0.30 per 1,000 rows on the Free plan ($0.15 on Business), and the $5 of monthly credit in Apify's Free plan covers about 16,000 rows.

What does it return?

Each output row is a row from your datasets, unchanged apart from the field options you set, in the order of Datasets. With Add the source dataset ID on, every row also gets sourceDatasetId, the ID of the dataset it came from, such as kx6fsPfmMfEw8IxVY.

The SUMMARY record in the run's key-value store, linked in the Output tab, describes what happened:

Field	Description
datasets	One entry per dataset with datasetId, rowsRead, filteredOut, duplicatesRemoved, rowsWithoutKey and outputRows. A removed duplicate is counted in the dataset it came from.
inputRows, outputRows	Rows read from all datasets and rows written to the output.
filteredOut, duplicatesRemoved	Totals of rows left out by Keep only rows where and of duplicates removed (or found, with Output set to the duplicates).
rowsWithoutKey	Rows where every Deduplicate by fields value was empty. They are always kept.
distinctKeys	Number of distinct rows (groups of duplicates) seen.
complete, stoppedBecause	true when every row was read; otherwise the reason, such as Maximum output rows, the maximum cost per run or the run timeout.
restarted	Present when the run restarted on another server, with the number of rows that were already written before.

How to merge and deduplicate datasets

1. In Datasets, pick datasets from your storage or enter dataset IDs or names such as username~my-dataset, in the order you want them merged. Paste IDs: Apify refuses a dataset link in this field before the run starts.
2. In Deduplicate by fields, enter the fields that make two rows the same, such as url, or companyName and title together. Nested fields use dot notation (company.name), and * compares whole rows. Leave it empty to merge without removing anything.
3. Set Match values to compare text exactly, ignoring extra spaces, or ignoring spaces and letter case, and turn on Treat URL variants as the same for addresses. For names or titles written in different ways, enter the field in Similar text field and set Similarity threshold.
4. Choose Keep first or last. Optionally add conditions in Keep only rows where, such as price > 100, and pick, remove, rename or flatten fields.
5. Click Start. The rows appear in the Output tab, where you can download them as CSV, Excel, JSON or HTML. Over the API, an input looks like this:

{
  "datasetIds": ["kx6fsPfmMfEw8IxVY", "2Udbt7i7aspbxBKEV"],
  "dedupeFields": ["url"],
  "normalizeUrls": true,
  "keep": "last",
  "filters": ["salaryMin > 100000", "remote = true"],
  "omitFields": ["description"]
}

Example output

One row from a run on 11 October 2026 that merged two job datasets of 9,905 and 9,970 rows, kept remote jobs with salaryMin above 100,000, removed repeated URLs, kept seven fields, renamed companyName to company and added the source dataset:

{
  "title": "Product Manager, Community Support",
  "company": "Airbnb",
  "salaryMin": 200000,
  "salaryMax": 240000,
  "salaryCurrency": "USD",
  "locations": [
    "Remote"
  ],
  "url": "https://careers.airbnb.com/positions/8189782?gh_jid=8189782",
  "sourceDatasetId": "kx6fsPfmMfEw8IxVY"
}

The SUMMARY of a run on the same two datasets deduplicated by url alone, shortened:

{
  "datasets": [
    { "datasetId": "kx6fsPfmMfEw8IxVY", "rowsRead": 9905, "duplicatesRemoved": 8861, "outputRows": 1044 },
    { "datasetId": "2Udbt7i7aspbxBKEV", "rowsRead": 9970, "duplicatesRemoved": 8818, "outputRows": 1152 }
  ],
  "inputRows": 19875,
  "duplicatesRemoved": 17679,
  "outputRows": 2196,
  "complete": true
}

How much does it cost to merge and deduplicate datasets?

	Free plan	Starter	Scale	Business
1,000 rows	$0.30	$0.24	$0.195	$0.15

You pay only for the events in the table. There is no start fee, and compute time and proxies are included.

You pay for the rows written to the output dataset. In a test on 11 October 2026, 25 datasets with 99,375 job rows deduplicated by URL gave 2,196 rows, which costs $0.6588 on the Free plan and $0.3294 on Business. Removed duplicates, filtered rows and the rows of the built-in sample are not charged.

How rows are matched

• With Match values set to exactly, text is compared as it is. Ignoring extra spaces trims the ends and treats runs of spaces as one, and ignoring spaces and letter case also makes ACME Inc equal acme inc. Numbers match their text form, so 42 and "42" are the same.
• With Treat URL variants as the same on, https://www.example.com/jobs/123/?utm_source=x#apply and http://example.com/jobs/123 match: the scheme, www., a trailing slash, the #fragment, the order of query parameters and tracking parameters (utm_*, gclid, fbclid, msclkid and 45 others) are ignored. Parameters that pick a product or page, such as ?id=5, still count, and the path stays case-sensitive. More URL parameters to ignore adds your own, such as ref.
• A field in Similar text field is compared by similarity, from 0.5 to 1, ignoring letter case, accents and punctuation. The score is the share of three-letter sequences two texts have in common, so Senior Software Engineer and Senior Software Engineer (Remote) score 0.87, while iPhone 15 Pro 128GB and iPhone 15 Pro 256GB score 0.80. With Deduplicate by fields filled in, those fields must match exactly and only the similar field may differ: same company, similar job title.
• With Keep set to first, the first copy in the order of Datasets stays, and with last the last one stays, so listing an older run before a newer one keeps the newest version of each row. Either way the output follows the original order.

Limits

• Speed, measured on 11 October 2026 with 1 GB of memory on job rows of about 760 bytes: 99,375 rows in 25 datasets were deduplicated by URL in 6.7 seconds for the whole run (5.1 seconds of work after the start), and in 7.6 seconds with Keep set to last, which reads every dataset twice. Writing is the slow part, at about 4,500 rows per second: merging all 99,375 rows without removing any took 24.5 seconds.
• Memory: deduplication keeps a short key for every distinct row, and similarity matching keeps the text of every kept row. The 99,375-row run above used at most 153 MB with 256 MB of memory. A run that comes close to its memory or its timeout stops, writes SUMMARY and ends as succeeded with the rows written so far; raise Memory or Timeout in Run options to finish the rest.
• Similarity can merge rows that differ in one short word. In our test data, Junior Interior Designer and Senior Interior Designer scored 0.86, and two titles ending in Web and iOS scored 0.90. Set Output to the duplicates first to see what a threshold removes. Texts are compared on their first 500 characters.
• Rows whose Deduplicate by fields values are all empty, or whose similar text is empty, are kept and counted in rowsWithoutKey. In one test, 4,775 job rows without companyName stayed when matching by company and similar title.
• Flattening turns nested objects into columns such as company.name; lists stay lists.
• A dataset that changed in the last 15 seconds is read once its row count settles, because the Apify API shows new rows with a delay of a few seconds.
• A dataset of another Apify account can be read only when its owner shares it with you or makes it readable by anyone with the ID.

Using it in an integration

To clean the dataset of every run of another actor or task, open that actor or task, go to Integrations, add this actor and enter {{resource.defaultDatasetId}} in Datasets (it is listed under Available variables). You can put a fixed dataset before it, such as last week's merged results, and set Keep to last to keep the newest version of each row. The variable is required: Apify grants a run with limited permissions access to the datasets in Datasets only, so an integration that passes the dataset just in its default payload fails with a message that explains this fix.

FAQ

Is it legal to merge and deduplicate datasets?

The actor reads only the datasets you pick in your Apify account, or datasets someone shared with you, and writes the result to the dataset of its own run. It visits no website and collects nothing new. Your right to keep and combine the data depends on where it came from, so you are responsible for using it in line with the terms of those sources and the law that applies to you, including rules on personal data.

Can the actor read my other datasets?

No. It runs with limited permissions, so Apify gives each run read access to the datasets in Datasets and to the run's own storage, nothing more. A dataset ID passed in any other field is refused, and the run status says which field to use.

Can I see which rows were removed?

Set Output to the duplicates. The run then writes the rows that the same settings would remove, which is a quick way to check a similarity threshold or a URL rule before the real run. SUMMARY also counts removed rows per dataset.

What happens if the run is moved to another server?

Apify sometimes moves a running actor to another server. The run then starts again, computes the same result and skips the rows already in its dataset. In a test, a run restarted in the middle of a 99,375-row merge ended with exactly 99,375 rows in the original order.

Related scrapers

• Career Site Jobs Search
• Remote Jobs Scraper
• Career Site Jobs Scraper

Support

This actor is built and maintained by deriverge s.r.o., a software company based in the Czech Republic. If a run fails or a field you need is missing, please open an issue in the Issues tab or write to us at info@deriverge.com. We respond in English and Czech. Runs can be scheduled in Apify Console or started from the API tab, which has examples for Python, JavaScript and cURL and works with Make, Zapier, n8n and the Apify MCP server. If the actor saves you time, a short review helps other people find it.