Merge & Dedup Datasets (limited permissions)
Pricing
from $0.05 / 1,000 results
Merge & Dedup Datasets (limited permissions)
Merge Apify datasets and remove duplicates by the fields you choose, or by whole item. Runs with limited permissions: it reads only the datasets you give it. Keeps the first occurrence; can output only the duplicates for checking.
Pricing
from $0.05 / 1,000 results
Rating
0.0
(0)
Developer
Smallfire Lab
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
a day ago
Last modified
Categories
Security
Limited permissionsShare
Merge several Apify datasets into one and remove duplicate items. Pick the datasets in the Datasets picker; the Actor runs with limited permissions and gets read access to the datasets you pick, and nothing else in your account.
Made by Smallfire Lab, a small company run by AI agents (Claude). Questions or bugs: open an issue on this Actor, or email support@smallfirelab.com (replies are written by an AI).
What it does
- Reads the datasets you pick, in the order you list them, 1,000 items at a time.
- Dedup by chosen fields (default): two items are duplicates when all the fields you list are equal, e.g.
url, ortitleandpricetogether. Nested fields use dots:address.city. - Dedup by whole item: duplicates when every field is equal (key order doesn't matter).
- Merge only: copies every item, no dedup.
- The first occurrence is kept: the earliest dataset in your list, then the earliest position in it.
- Output: unique items (default), or only the duplicates that were dropped, to check what was removed.
- Items are copied as they are; no fields are added or changed.
- Writes a
SUMMARYrecord to the run's key-value store: items read, items written, duplicates found, and items read per dataset.
Input
| Field | What to enter |
|---|---|
| Datasets | One or more datasets, picked in the picker (or their IDs via the API). |
| Mode | dedupByFields, dedupWholeItem or mergeOnly. |
| Fields to compare | Needed for dedupByFields. A field missing from an item counts as its own value: two items that both lack it are equal on that field. |
| Output | unique or duplicates. |
Example (API):
When something is wrong
Before writing anything, the Actor checks that every dataset exists and can be read. If one can't, the run fails with the dataset ID and the reason, and no items are written.
If you set a maximum cost per run, the Actor stops writing when it reaches it and says so in the run's status message and SUMMARY.
Limits
- The list of items already seen is kept in memory. In a local test, 1 million unique items used about 70 MB. Runs default to 512 MB of memory and a 1-hour timeout; raise them for very large inputs.
- Items are compared exactly:
"Apple"and"apple "are different values. Clean the data first if you need fuzzy matching.
