Dataset Merge, Dedupe & Transform
Under maintenancePricing
$0.50 / 1,000 row writtens
Dataset Merge, Dedupe & Transform
Under maintenanceMerge several Apify datasets into one: remove duplicates by any field, filter rows, keep, drop or rename columns, flatten nested data and sort. $0.0005 per row written.
Pricing
$0.50 / 1,000 row writtens
Rating
0.0
(0)
Developer
ensorcelling _melilot
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
17 days ago
Last modified
Categories
Security
Limited permissionsShare
Combine several Apify datasets into one clean dataset. Remove duplicates by any field, filter rows, keep, drop or rename columns, flatten nested data and sort. No code, no scraping. It only reads datasets you already have.
Typical jobs:
- Merge the results of many scraper runs into one list.
- Remove duplicate leads, products or jobs by
url,emailorid, across runs. - Keep only rows that match rules (
price < 20,country = DE,email exists). - Make a tidy table for a spreadsheet: pick columns, rename them, flatten
address.city.
Why this one?
- Simple price. $0.0005 per row written ($0.50 per 1,000). Rows you filter out or drop as duplicates are free.
- Doesn't crash on big inputs. Datasets are read page by page, and duplicates are removed while streaming. Only sorting or "keep last duplicate" holds rows in memory.
- Smart duplicate matching. Compare chosen fields (dot paths like
company.idwork), ignoring upper/lower case and extra spaces, or compare whole rows. - Clear errors. A typo in a filter operator or a bad dataset ID stops the run immediately with a plain message and no charge.
Input
| Field | What it does |
|---|---|
| Datasets to merge | One or more datasets, read in order |
| Raw rows | Optional JSON rows added after the datasets |
| Remove duplicates / Duplicate if these fields match | Fields that identify a row; empty means whole-row comparison |
| Which duplicate to keep | First (default) or last |
| Filter rules + Rows must pass | List of {field, operator, value}; all or any |
| Keep only / Drop / Rename fields | Choose and rename columns |
| Flatten, trim, drop empty values | Tidy rows for spreadsheets |
| Sort by + order | Numbers sort as numbers, empty values last |
| Max rows to output | Stop after N rows |
Add _source column | Which dataset each row came from |
Filter operators: equals, notEquals, contains, notContains, startsWith, endsWith, greaterThan, lessThan, exists, notExists, matches (regular expression). Text comparisons ignore case.
Example
Output
One row per merged item in the run's dataset, plus a SUMMARY record with the number of rows read per dataset, filtered out, duplicates removed and written.
Limits and notes
- You can only read datasets from your own Apify account.
- Sorting and "keep last duplicate" support up to 1,000,000 rows per run.
- Fields are matched exactly (
Priceandpriceare different columns). Nested fields use dots. - Only a field's value is compared for duplicates, not its type:
5and"5"count as the same.
