# Changelog of Dataset Deduplicator & Cleaner (`eszetael_lab/dataset-deduplicator-cleaner`) Actor

- **URL**: https://apify.com/eszetael\_lab/dataset-deduplicator-cleaner/changelog.md
- **Full Actor documentation**: https://apify.com/eszetael\_lab/dataset-deduplicator-cleaner.md

## Changelog

### 0.1 (2026-07-23) — G4 hardening (adversarial gate, round 5)

- **No double-charge**: charge is never retried after an ambiguous error (`Actor.charge` isn't
  idempotent, so a lost response after a server-side charge would bill twice) — the single unit is
  dropped instead. Exposure is at most one under-charged record, never an overcharge.
- **Field boundaries preserved in dedup**: each key/field is normalized separately and joined with a
  separator, so `{first: "Ann", last: "Marie"}` and `{first: "Annma", last: "rie"}` are no longer
  merged.
- **Bounded signatures**: very large list/dict values are summarized by length plus a head+tail sample
  instead of being fully serialized — a 2-million-element field no longer makes a run hang, while
  values that differ at the end stay distinct.
- **Loading distinguishes a real end from a backend glitch**: a missing/malformed page is retried and,
  if it persists, returns the partial data with a loud truncation warning (or fails if nothing loaded)
  rather than silently stopping.
- Buyer-facing messages and logs are all in English.

### 0.1 (2026-07-23) — G4 hardening (adversarial gate, round 4)

- **Quality report is linear** (single pass, computed only when requested) — heterogeneous data
  (unique field per row) no longer turns the report into a multi-minute step.
- **Phone cleaning is shape-gated** like email/URL: values that aren't phone-shaped (e.g.
  `"Order #12345678 pending"`) are left untouched instead of being reduced to digits.
- **Billing is per-record with an ambiguity-safe policy**: a record is charged right after it's
  delivered (small abort/migration exposure), and an ambiguous charge error drops that single unit
  rather than re-sending it — so a lost response never double-charges.
- **Robust dataset loading**: pages are retried on transient errors, a short page no longer ends the
  load, and an under-reported `total` can't cut it short (termination is empty-page / cap only).
- **Push circuit-breaker**: repeated push failures abort the run instead of burning hours on backoff.
- **Memory**: minimum memory raised to 512 MB and intermediate lists freed after use to avoid OOM
  near the 50k cap.
- Type coercion preserves leading zeros (ZIP/IDs); collapse guard also covers small inputs.

### 0.1 (2026-07-23) — G4 hardening (adversarial gate, round 3)

- **Fuzzy no longer chains near-duplicates** (data-loss fix): switched from single-linkage union-find to
  leader/canopy clustering — a record joins a cluster only if it matches the cluster's representative,
  not any member. A gradient of strings (A…A → B…B) no longer collapses records that are 0% similar.
- **Fuzzy cost budget is now quadratic** (`pairs × len²`) plus a pair-count cap and a 128-char comparison
  cap — near-duplicate inputs that previously ran for minutes now degrade to normalized instantly.
- **Billing no longer under-charges on transient errors**: units stay queued and retry on the next flush
  instead of being dropped; charges are counted only after a confirmed success. Charge batching (50)
  cuts platform round-trips.
- **Pagination** treats a `total` of 0 as "unknown" (Crawlee's default) rather than end-of-data.
- **Collapse guard**: if more than 70% of the input is removed as duplicates, the run says so in its
  status and quality report (heavy collapse is never silent).
- Field-type detection is safe on non-string keys; SEO title trimmed to ≤50 chars; docs aligned.

### 0.1 (2026-07-23) — G4 hardening (adversarial gate, round 2)

- **Fuzzy cost-aware budget**: comparison strings are capped at 256 chars and the budget now measures
  *work* (pairs × length), not just pair count — long-string inputs (e.g. 500 rows × 600 chars) that
  previously ground for ~55s now degrade instantly to normalized instead of risking a run timeout.
- **Full pagination fix**: no longer stops on a short-but-non-empty page (Apify caps responses ~9MB for
  large records) — only an empty page / dataset total / the cap end the load. Removes silent data loss.
- **Empty key values**: `None`/`""` in a dedup key now yield an empty signature → the record stays
  unique (previously `None` serialized to `"null"` and could merge distinct records).
- **Billing free-flip narrowed**: only an unambiguous "not pay-per-event" signal flips to free; a
  transient charging error stays fail-closed (no silent free delivery of a paid run).
- **Output schema** (`.actor/output_schema.json`, wired via `output`): declares the clean dataset and
  the `QUALITY_REPORT` KVS output with link templates, so AI agents (MCP) can discover and chain it.
- **Prefill fixed** so the default run actually removes a duplicate (demonstrates the product).
- Publish checklist (`PUBLISH.md`) documents the required `result-item` pay-per-event registration.

### 0.1 (2026-07-23) — G4 hardening (adversarial gate, round 1)

Fixes from the L4 adversarial review before first publish:

- **Data-loss guard (critical)**: a record with an empty signature — a missing/renamed `dedupKeys`
  field, or a value that is only punctuation/emoji/CJK — is now treated as **unique** and never
  merged with another empty. Previously a whole dataset could silently collapse to one row.
- **dedupKeys validation**: if none of the given keys exist in any record the run fails loudly;
  low key coverage is warned in the status and report.
- **Fuzzy is bounded**: hard comparison budget with a `quick_ratio` pre-filter and value-based
  blocking; on very large or low-diversity input fuzzy auto-degrades to `normalized` and reports it,
  instead of grinding for hours (a shared-prefix column no longer defeats blocking).
- **Full pagination** of source datasets, with an explicit truncation warning if the 50,000 cap is hit
  (no more silently processing only the first page).
- **Billing fail-closed**: unknown pricing shape defaults to charging (never silent free delivery of a
  paid run); genuine free actors are detected on the first no-charge signal.
- **Push isolation**: per-record push retries; charges only follow a confirmed push.
- **Robustness**: field-type detection matches on word boundaries (`tel` no longer matches `hotel`);
  signature building is recursion-safe; clean+dedup is wrapped so bad input fails cleanly.
- Default run now ships a `prefill` example so the first click returns a result instead of an error.
- 17 unit tests (up from 12), including regressions for every fix above.

### 0.1 (2026-07-23) — initial build

- First version: Dataset Deduplicator & Cleaner — a non-scraper utility that plugs into any dataset.
- **Dedup**: exact, normalized (case/whitespace/punctuation-insensitive) and fuzzy (near-duplicate,
  difflib similarity with a configurable threshold, union-find clustering, blocking above 2000 rows
  for scale). Keep-strategy: most-complete / first / last.
- **Cleaning**: per-type field cleaning auto-detected by field name — emails (validate + lowercase),
  phones (E.164-ish), URLs (normalize), plus whitespace collapse, control-char strip, HTML strip,
  case, empty→null, drop-empty, and optional type coercion (numbers/booleans).
- **Quality report**: input/output counts, duplicates removed, per-field fill/null rates, values
  cleaned — saved to the key-value store as `QUALITY_REPORT`.
- Zero-config by default (normalized dedup + sensible cleaning) and fully configurable.
- Hardened skeleton reused from our reference actors: per-result PPE with fail-closed billing, free
  mode (no charge), FAIL-on-zero. 12 unit tests (cleaners, dedup, quality).