Dataset Merge, Dedup & Transform – Clean Data

What does Dataset Merge, Dedup & Transform do?

Dataset Merge, Dedup & Transform combines the results of several Apify runs into one clean dataset. It merges datasets, removes duplicates by one or more fields, filters items with simple conditions, and picks, removes, renames or flattens columns, all in one run.

• 🧩 Merge any number of datasets, plus items pasted as JSON.
• 🧹 Deduplicate by one or more fields (url, id, name + city…), ignoring letter case and extra spaces, or remove exact duplicates.
• 🔀 Keep the first, the last or a merged version of each duplicate (empty fields filled from the other copies).
• 🔎 Filter with conditions such as price < 20, title contains "shoe", date >= 2026-01-01, email is empty.
• ✂️ Transform: keep only some fields, drop others, rename them, flatten nested objects for CSV and Google Sheets.
• 🔒 Limited permissions: the Actor only gets READ access to the datasets you pick. It cannot see anything else in your account.

Why use Dataset Merge, Dedup & Transform?

• 🗂️ Combine scraper runs. You ran the same scraper on 10 search terms or 5 cities: get one deduplicated list.
• 📈 Incremental scraping. Merge yesterday's and today's results and keep the latest version of each item (keep: last).
• 🧾 Prepare exports. Rename and flatten fields before sending data to Google Sheets, Airtable, a CRM or a database.
• 🤖 Pipelines and AI agents. Call it after any Actor (webhook, Make, Zapier, n8n, MCP) to clean the output before the next step.

How to merge and deduplicate datasets

1. Click Try for free.
2. In Datasets to merge, pick the datasets of your previous runs (or paste items in Items (JSON)).
3. In Deduplicate by fields, enter the field that identifies an item, for example url.
4. Optionally add Filters, choose fields to keep, remove or rename, and turn on Flatten nested objects.
5. Click Start and download the clean dataset as CSV, Excel or JSON.

Input

The default input deduplicates a few sample items, so you can see the result in seconds:

{
    "datasetIds": ["<your dataset ID>"],
    "dedupFields": ["url"],
    "caseInsensitive": true,
    "keep": "first",
    "filters": [{ "field": "price", "op": "lt", "value": 53 }],
    "omitFields": ["stock.available"],
    "renameFields": { "title": "name" },
    "flattenNested": true
}

Filter operators

eq, neq, contains, notContains, startsWith, endsWith, regex, gt, gte, lt, lte (numbers or dates), exists, notExists, empty, notEmpty, in, notIn (value is an array). Text comparisons ignore case unless you add "caseInsensitive": false to the condition. All conditions must match.

Order of operations

1. Read inline items, then each dataset in the order given.
2. Apply filters (on the original fields).
3. Deduplicate (on the original fields).
4. Pick / remove fields, flatten, then rename.

Output

Your items, cleaned. For example, with flattenNested and renameFields: {"stock.count": "stock_count"}:

[
    {
        "url": "https://books.toscrape.com/catalogue/a-light-in-the-attic_1000/index.html",
        "title": "A Light in the Attic",
        "price": 51.77,
        "stock.available": true,
        "stock_count": 22
    }
]

The SUMMARY record in the key-value store lists, for each source, the items read, filtered out and removed as duplicates, and the total written.

How much does it cost?

The Actor is free to use for now: there is no Actor fee, you only pay the Apify platform usage, which is very small (no browser, no proxy, no web requests). Deduplication with keep: first streams the data with little memory. keep: last and keep: merge hold the kept items in memory: for several million items, raise the run memory.

Integrations

Chain it after any Actor with a webhook or an Actor-to-Actor integration, or call it from Make, Zapier, n8n, the API or an AI agent through the Apify MCP server. Then send the clean dataset to Google Sheets, Airtable, Slack or your database.

FAQ

How do I remove duplicates from an Apify dataset?

Pick the dataset, enter the field that identifies an item (for example url) in Deduplicate by fields, and start the run. Every item after the first one with the same value is removed.

Can I deduplicate on several fields?

Yes. Items are duplicates only if all the listed fields are equal, for example name and address.city. Items whose key fields are all empty are always kept.

Why does the Actor need my datasets?

It reads only the datasets you pick in the input, with READ permission. It runs with Apify's limited permissions, so it cannot access any other data in your account and it never changes your datasets.

Can I write to an existing dataset?

The result is written to the run's default dataset, which you can export or pass to the next Actor.

Something does not work?

Open an issue in the Issues tab with your input (without private data) and the SUMMARY record.

Related actors

More Actors by the same author (innovative_cookie on Apify):

• Sitemap URL Extractor: get every URL of a site from its XML sitemaps.
• Broken Link Checker: find 404s and dead links on any website.
• Website Image Extractor: list every image of a page with size, format and alt text.
• Bulk Domain Availability Checker: check whether domain names are free via RDAP.