Dataset Transformer - Filter, Dedupe, Merge & Export CSV

Clean up the results of any Apify scraper in one step: filter rows, remove duplicates, merge several runs, keep or rename columns, flatten nested JSON, sort, and export a ready-to-use CSV. No code, no spreadsheet formulas, and no downloading 50 MB of JSON to fix it by hand.

Point it at one or more datasets (or run IDs) and it writes the cleaned rows to a new dataset plus an OUTPUT.csv file you can open in Excel or Google Sheets.

What you get

• Filters on any field, including nested fields (address.city) and lists (posts.likes): equals, not equals, contains, does not contain, regex, greater than / less than (numbers and dates), exists, does not exist, in list, not in list. Combine with AND or OR.
• Smart number handling: values scraped as text such as "1,234", "$19.99", "12%", "10.5K" or "2.1M" compare as numbers.
• Deduplication by one or more fields (phone, url, username) or by the whole row. Loose matching ignores case and spacing and compares phone numbers by digits, so (555) 123-4567 and 555.123.4567 count as the same lead.
• Merge any number of datasets in the order you list them, optionally tagging every row with the dataset it came from.
• Column control: keep only the fields you want (in your order), drop fields, rename fields, and flatten nested objects into address.city-style columns.
• Sort by any field, ascending or descending, then skip and limit rows.
• Exports: transformed rows in the run's dataset, OUTPUT.csv (always flat, RFC 4180 quoting, optional Excel BOM and ; or tab delimiter) and optional OUTPUT.json.
• Run summary (SUMMARY record): rows read, rows filtered out, duplicates removed, rows output and download links.

Use cases

• Lead list cleanup: dedupe Google Maps or directory scrapes by phone number or website before importing to a CRM.
• Influencer and audience research: keep only Instagram, TikTok or X profiles above a follower threshold, verified accounts, or bios matching a keyword.
• Combine scheduled runs: merge daily or weekly runs into one deduplicated master list.
• Spreadsheet-ready exports: flatten nested JSON and rename columns so the CSV drops straight into Google Sheets, Excel, Airtable or a mail-merge tool.
• Competitor and price monitoring: filter product or listing scrapes by price, rating, category or date and sort the result.

How to use

1. Paste one or more dataset IDs (or run IDs, dataset names, or dataset URLs) into Datasets to transform. You can find them in the Storage tab of any finished run.
2. Optionally add Filters, Remove duplicates by, Keep only these fields, Rename fields, Sort by and Max rows.
3. Run the actor. Clean rows appear in the Output tab; OUTPUT.csv and SUMMARY are in the Key-value store tab.

Chain it after any scraper

To clean every run automatically, open your scraper, go to Integrations, add Apify Actor, choose Dataset Transformer, and set Datasets to transform to {{resource.defaultDatasetId}}. Every time the scraper finishes, the transformer runs on its results.

Examples

Dedupe Google Maps leads by phone and export a CSV

{
  "datasetIds": ["<your Google Maps run's dataset ID>"],
  "filters": [{ "field": "phone", "operator": "exists" }],
  "dedupeBy": ["phone"],
  "fields": ["title", "phone", "website", "address", "totalScore", "reviewsCount"],
  "renameFields": { "title": "businessName" },
  "sortBy": "reviewsCount",
  "sortOrder": "desc",
  "flatten": true
}

Sorting by reviewsCount descending before deduplication means that when the same phone appears twice, the listing with the most reviews is the one kept.

Instagram profiles with more than 10,000 followers

{
  "datasetIds": ["<your Instagram profile scraper dataset ID>"],
  "filters": [
    { "field": "followersCount", "operator": "gt", "value": 10000 },
    { "field": "biography", "operator": "regex", "value": "coach|trainer|fitness" }
  ],
  "filterLogic": "AND",
  "dedupeBy": ["username"],
  "fields": ["username", "fullName", "followersCount", "biography", "externalUrl"]
}

followersCount values such as "12.5K" or "1,200,000" are compared as numbers.

Merge three runs into one CSV

{
  "datasetIds": ["<run 1 dataset ID>", "<run 2 dataset ID>", "<run 3 dataset ID>"],
  "dedupeBy": ["url"],
  "addSourceField": true,
  "exportCsv": true,
  "csvBom": true
}

The output keeps the first occurrence of each URL in the order the datasets are listed, and _sourceDataset tells you which run each row came from.

More filter examples

Goal	Rule
City is Austin (any case)	{"field": "address.city", "operator": "equals", "value": "austin"}
Has a website	{"field": "website", "operator": "exists"}
Email missing	{"field": "email", "operator": "notExists"}
Rating 4.5 or higher	{"field": "totalScore", "operator": "gte", "value": 4.5}
Posted in 2025 or later	{"field": "createdAt", "operator": "gte", "value": "2025-01-01"}
Country is US, CA or UK	{"field": "country", "operator": "in", "value": ["US", "CA", "UK"]}
Title does not mention "closed"	{"field": "title", "operator": "notContains", "value": "closed"}
Any post has over 1,000 likes	{"field": "latestPosts.likes", "operator": "gt", "value": 1000}
Exact-case match	{"field": "sku", "operator": "equals", "value": "AB-12", "caseSensitive": true}

Output format

The dataset contains your transformed rows, for example:

{
  "businessName": "Blue Bottle Coffee",
  "phone": "+1 512-555-0134",
  "website": "https://bluebottlecoffee.com",
  "address.street": "123 Main St",
  "address.city": "Austin",
  "totalScore": 4.6,
  "reviewsCount": 1834
}

The SUMMARY record in the key-value store:

{
  "sources": [{ "input": "aBcD1234", "datasetId": "aBcD1234", "rowsRead": 5000 }],
  "rowsRead": 5000,
  "rowsFiltered": 1210,
  "duplicatesRemoved": 344,
  "rowsSkippedByOffset": 0,
  "rowsOutput": 3446,
  "stoppedReason": null,
  "csvUrl": "https://api.apify.com/v2/key-value-stores/.../records/OUTPUT.csv",
  "jsonUrl": null
}

stoppedReason is maxItems, maxChargeReached or timeout when the run stopped early, and null otherwise.

How the steps are applied

1. Datasets are read in the order listed (hidden #fields are skipped).
2. Filters run on the original rows, so you can filter on fields you later drop or rename.
3. If Sort by is set, rows are sorted; duplicates are then removed keeping the first row in sorted order.
4. Skip first N rows and Max rows are applied to the final result.
5. Fields are selected, removed, renamed and (optionally) flattened.

Flattening and lists. With the default Join mode, a list of plain values becomes one cell ("Cafe, Coffee shop") and a list of objects becomes JSON text. One column per item gives images.0.url, images.1.url and so on. The CSV file is always flat, even when Flatten nested objects is off for the dataset.

Limits and performance

• Without sorting, rows stream through in pages, so memory use stays flat for any dataset size. Deduplication keeps a short fingerprint per unique row (about 60 bytes), so 256 MB handles a few million unique keys.
• Sorting keeps a small index per row (about 250 bytes) and writes the rows to temporary disk, so 256 MB comfortably sorts several hundred thousand rows. For multi-million-row sorts, raise the run memory to 1 GB or more.
• Up to 50 datasets per run.
• If a run gets close to its timeout, it stops reading and still saves everything processed so far, including the CSV.

FAQ

Can I use a dataset from another account? Yes, if that account has shared it with you or its storage access is public. Use the dataset ID or username~dataset-name.

Does it change my original dataset? No. The source datasets are only read; results go to this run's own dataset and key-value store.

How do I open the CSV in Excel with accents and emoji intact? Turn on Excel-friendly CSV (UTF-8 BOM), and pick the semicolon delimiter if your Excel uses a decimal comma.

What do I pay for? One result event per row written to the output. Rows you filter out or remove as duplicates are not charged, and the run stops cleanly if it reaches your maximum cost per run.