# Changelog of Merge & Deduplicate Datasets - Remove Duplicates (`luxe_tasks/dataset-dedupe`) Actor

- **URL**: https://apify.com/luxe_tasks/dataset-dedupe/changelog.md
- **Full Actor documentation**: https://apify.com/luxe_tasks/dataset-dedupe.md

## Changelog

### 0.1.3 (first public release)

- If a run stops on an error (while reading a dataset, while saving rows, or an unexpected error) after some charges could not be confirmed, the stop message now also says how many rows may have been charged.
- Fix: with keep "last", reaching the 2,000,000-row read limit is now reported (it was silent before).
- When a run reaches the 2,000,000-row read limit, the status message now says "Partial result possible", and the Done line and the log warning say "any later rows": in some cases a run that read exactly 2,000,000 rows is flagged even though no rows were left unread.
- Fix: if a run was stopped and restarted before it had saved its first progress record, rows it had already written are now skipped instead of written a second time; they are not charged again.
- Fix: rows already in the run's dataset when it starts are no longer mistaken for a resumed run.
- Correction to the r10 restart fix above ("stopped and restarted before it had saved its first progress record"): it treated every row already in the run's dataset as written by this Actor, so rows left there by another Actor were skipped as if resumed. This Actor now saves the dataset's starting row count before it writes any row, and on a restart skips only the rows it wrote itself.
- If this Actor cannot tell which rows in the run's dataset are its own (its saved progress record has no starting row count, the starting count is larger than the dataset, the progress record or the dataset's row count cannot be read, confirmed or saved, or the input changed after it had written rows, or after a restart the rows in the dataset do not line up with its saved progress record), it now stops without writing or charging any more rows, with a message saying why (the per-run start fee still applies).
- Fix: right after rows are added to a dataset, Apify can briefly report a lower row count. This Actor now confirms the run's dataset row count by reading it again before relying on it; if the count cannot be confirmed within about 30 seconds, it stops without writing or charging any more rows, with a message saying why (the per-run start fee still applies).
- Fix: after a restart, this Actor now also checks the content of the rows it saved before resuming; if they don't match, it stops without writing or charging any more rows, with a message saying why (the per-run start fee still applies).
- Change: runs handed over from another Actor (metamorph) now stop at the start without reading, writing or charging any rows, because such runs use the other Actor's pricing. Start this Actor directly instead.
- Correction to 0.1.0: the page and batch sizes listed there are set by this Actor for memory safety, not Apify limits.

### 0.1.2 – 2026-10-31 (pre-release, never published)

- Dataset check errors now end with "No rows were read or charged (the per-run start fee still applies)." HTTP 401 says the dataset "could not be authorised"; HTTP 403 still says it "is not shared with this run".
- An error partway through a run (reading a dataset or saving rows) now stops with a message saying how many rows were written and charged before the stop, and `RUN_SUMMARY` is still saved.
- README: pricing bullets define an output row; the logs bullet lists dataset IDs and field names; new "Stays in your Apify account" bullet; enforced limits are no longer called assumed.
- Sample data uses neutral "Sample …" product names (same 6 rows in, 5 out).
- Correction to 0.1.0: logs and RUN_SUMMARY also contain dataset IDs and the field names you chose, never your field values.

### 0.1.1 – 2026-10-31 (pre-release, never published)

- Dataset errors now show the HTTP status code and Apify's error type, for example `HTTP 403 insufficient-permissions`. "Not found" (HTTP 404) has its own message, separate from "not shared with this run" (HTTP 401/403). Messages never contain your token, request headers or row data.
- Wording: the dataset field takes dataset IDs; pick datasets with the picker or paste their IDs.
- Pricing text now gives the start fee per GB of memory in the same sentence as the per-row price.

### 0.1.0-r3 – 2026-10-31 (pre-release, never published)

- Dataset schema: removed the Output-tab view that listed no fields (Apify's schema needs at least one field per view); the Output tab now shows all fields.
- Large rows: dataset pages start at 50 rows and shrink to about 16 MB of JSON per page for wide rows; write batches also flush at about 16 MB (all ASSUMED sizes).
- `RUN_SUMMARY` uses the number of rows Apify actually charged when a charge covers fewer rows than requested.
- Wording: sample text in the input form mentions the per-run start fee; README adds Known limitations and a Support section.

### 0.1.0 – 2026-10-31 (pre-release, never published)

- First version: merge Apify datasets (read-only dataset picker) and inline JSON items; dedupe by whole row or key fields (dot paths),
  keep first/last, trim and case options, missing-key policy; pick/omit/rename/cast fields; CSV/Excel formula escaping on by default;
  optional source fields; max output rows.
- Pay-per-event `output-row`, charged after the rows are saved; respects the run's maximum total charge; sample runs charge no rows (the per-run start fee still applies).
- Restart-safe: rows already written are skipped and not charged again.
- Logs and `RUN_SUMMARY` contain counts only.
- r2 (same day, before any review): per-source `rowsRead` in `RUN_SUMMARY` now counts the last row read before a budget or
  max-rows stop (was 1 short; total `rowsRead` was already right).
- Not in this version (deferred): CSV/XLSX/JSON file URL intake, filters/expressions.
- Local memory test (Node 24.22.1, development machine, not Apify): 2,000,000 distinct keys read in about 13 s with peak RSS about 491 MiB (515 MB)
  (`keep: last`); 1,000,000 rows with 50% duplicates about 3 s, about 270 MiB (`keep: first`).