# Changelog of Merge, Deduplicate & Join Datasets (`quietworkslab/merge-dedup-datasets`) Actor

- **URL**: https://apify.com/quietworkslab/merge-dedup-datasets/changelog.md
- **Full Actor documentation**: https://apify.com/quietworkslab/merge-dedup-datasets.md

## Changelog

### 0.1 (not published yet)

- Build 0.1.3 (found on Apify with build 0.1.2):
  - Right after the last push of a scraper, the item count the API reports can be behind (a run started a few seconds after 30 items were pushed saw 0 and read nothing). At the start, the Actor now reads on after the reported count (pages of up to 1,000 items, at most 100 requests) and freezes the real count, for the datasets to merge and for the lookup dataset; the log says when the count was behind. A restarted run keeps the counts of its first attempt.
  - `counts.lookup.chargedAs` is the number of events really charged for the lookup dataset (0 in the sample).
  - With items of several MB the first-item sample is small: the "no key value at all" stop now needs at least 5 sampled items, otherwise it is a warning.
- Build 0.1.2 (after the review and the checks on Apify):
  - Clearer order: "keep first" keeps the item from the dataset read first, "keep last" the item from the dataset read last. To keep the newest version with "keep last", put the newest data last; `{{resource.defaultDatasetId}}` also works inside `datasetIds`.
  - "Keep last" and the duplicate count stop at the start, with nothing charged, when the maximum cost per run does not cover all items (they read everything before the first row is written). The message says what maximum to set.
  - The lookup dataset is charged at 1 `item-processed` event per started 20 lookup items (was 1 per lookup item).
  - `fields` and `renameFields` apply after the join, so lookup columns (with their prefix) can be listed, ordered and renamed. Without dedupe key fields, only the fields to keep are compared.
  - Checks before charging stop only when none of the first items has a key value (or none of the first lookup items has a lookup key), or for an inner join with an empty lookup dataset. A key that only some items have, or a join whose first items find nothing, now gives a warning and the run continues.
  - After reading each dataset up to its count at the start, one request checks whether items were added; if so, the log and `OUTPUT` (`itemsAddedAfterStart`) say how many were not read.
  - Big items: the first page and the pages are sized by the items' size (from 1 item per page), and the memory check counts the largest item. Tested locally with items of 300 KB and 3 MB at the heap of a 256 MB run.
  - In the built-in sample, the key matching options apply (default: spaces only; all four show the width and kana matching).
  - Messages name the fields with their Console titles. A dataset ID that is not text is an input problem; temporary API errors are no longer reported as input problems.
  - Smaller fixes: case folding is stable for the capital sharp s; keys with lone surrogates no longer share a hash; a field named `__proto__` is kept; a restarted run's summary shows the real `chargedItems`; an abort or the timeout after all items were processed finishes `COMPLETE`; overwritten `_sourceDataset`/`_duplicateCount` fields are counted and logged; the matched join key is not counted as a clash; at most 9 entries in `datasetIds`.
- First version: merge up to 10 datasets in order, remove duplicates by key fields (dotted paths) or whole items, keep the first or the last item of each key.
- Optional join with one lookup dataset (like VLOOKUP): first lookup item per key, left or inner join, a prefix for the added fields, a choice of lookup fields (missing ones are written as `null`).
- Key matching options for duplicates and the join, used only for comparing: trim spaces, ignore case, full-width/half-width folding (with kana voiced sound marks and NFC), katakana = hiragana. The output keeps the original values.
- Output options: fields to keep (in order), field renames, `_sourceDataset` and `_duplicateCount`.
- Pay-per-event: one `item-processed` event per item read from the datasets to merge, charged per page right after it is saved; reading twice for "keep last" or duplicate counts does not charge twice. Stops cleanly at the maximum cost per run. An interrupted run continues without charging an item twice. Platform migrations, aborts and the run timeout (30 s before it) are handled by the Actor itself, so the summary is always written.
- Built-in sample data (30 made-up products in two datasets and a maker list) for a run without input.
- After the second review (2026-11-01): a failed request in the check for items added during the run only logs a warning; a warning names the lookup columns that "Fields to keep" leaves out; the README says that duplicates are found before the join.
- Build 0.1.4 (2026-11-01): a second dataset view "Built-in sample" (the sample's columns, for the example task pages; "Merged items" still shows every field) and README links to the example tasks and the other Actors of Quiet Works Lab.