Merge & Deduplicate Datasets (works with limited permissions) avatar

Merge & Deduplicate Datasets (works with limited permissions)

Pricing

Pay per usage

Go to Apify Store
Merge & Deduplicate Datasets (works with limited permissions)

Merge & Deduplicate Datasets (works with limited permissions)

Merge several Apify datasets (and public JSON files) into one and remove duplicates by any fields, with case/whitespace/URL normalization, keep first or last, only-new mode and field projection. Requests read-only access only to the datasets you pick.

Pricing

Pay per usage

Rating

0.0

(0)

Developer

Open Registry Data

Open Registry Data

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

25 days ago

Last modified

Categories

Features

Limited permissions

Share

Merge several Apify datasets (for example the outputs of many scraper runs) and public JSON files into one clean dataset, and remove duplicates by the fields you choose. Works under Apify's limited permissions and on accounts with restricted resource access: the dataset picker asks for read-only access to just the datasets you select, so runs don't fail with permission errors.

No scraping happens. The Actor only reads the datasets / JSON URLs you give it and writes the result to its own default dataset.

Features

  • Merge any number of datasets (picker or IDs via API) plus optional public JSON / JSON Lines URLs.
  • Dedup by fields: one field (url) or a combination (name + address), nested paths (location.city, images.0), objects/arrays compared by value. Or dedup by whole item, or merge only.
  • Normalization before comparing: ignore letter case, extra whitespace, and URL noise (http/https, www., trailing slash, #fragment, utm_*/fbclid/gclid).
  • Keep first or last occurrence (list datasets oldest → newest and keep last to get the freshest copy).
  • Only new items: exclude anything whose key already exists in other datasets (e.g. last week's output) for incremental pipelines.
  • Output unique items, only the duplicates (audit), or just a duplicate-count summary.
  • Field projection: output only chosen fields, or drop fields; optional __source field with the origin dataset.
  • SUMMARY record: items loaded per source, unique, duplicates, excluded, output, errors.
  • Clear per-source errors: an unreadable or missing dataset is skipped with a message, the rest still merges.

Input example

{
"datasetIds": ["<datasetIdRun1>", "<datasetIdRun2>", "<datasetIdRun3>"],
"dedupFields": ["url"],
"normalizeUrls": true,
"keep": "last",
"excludeItemsInDatasetIds": ["<lastWeeksMergedDatasetId>"],
"outputFields": ["url", "title", "price"]
}

Performance (local test, 28 Oct 2026)

  • 92,820 items → 4,641 unique (dedup on 3 fields) in 3.0 s.
  • 12,345 items read through the paginated dataset API (with one simulated HTTP 429 retry, a 403 and a 404 source skipped) → 10,000 unique in 3.5 s.
  • 17 local test runs: 15 succeeded, 2 invalid inputs (empty, all sources missing) failed by design with a clear message. 10 outputs (merge, keep-last, only-new, duplicates-only, summary, projection + case-folding, whole-item, nested-path, 92k scale, URL normalization + nulls) were cross-checked against an independent reference implementation and all matched. See samples/.

Limits (honest list)

  • Dedup keys for every unique item are kept in memory (~100 bytes each): fine for tens of millions of items at 4 GB; keep: last also keeps the unique items themselves in memory.
  • Output always goes to this run's default dataset (no writing into your other datasets, so no write permission is requested). Use Apify integrations or the API to copy it onward.
  • Named datasets: pass them with the picker, or as username~dataset-name via API.
  • No JavaScript transform functions (keeps runs safe and predictable); use field projection or post-process downstream.

Pricing

Free to use for now: you only pay Apify's normal platform usage for the run. Pay-per-event pricing may be added later, with Apify's standard advance notice.

By Open Registry Data.