No double-charge: charge is never retried after an ambiguous error (Actor.charge isn't
idempotent, so a lost response after a server-side charge would bill twice) — the single unit is
dropped instead. Exposure is at most one under-charged record, never an overcharge.
Field boundaries preserved in dedup: each key/field is normalized separately and joined with a
separator, so {first: "Ann", last: "Marie"} and {first: "Annma", last: "rie"} are no longer
merged.
Bounded signatures: very large list/dict values are summarized by length plus a head+tail sample
instead of being fully serialized — a 2-million-element field no longer makes a run hang, while
values that differ at the end stay distinct.
Loading distinguishes a real end from a backend glitch: a missing/malformed page is retried and,
if it persists, returns the partial data with a loud truncation warning (or fails if nothing loaded)
rather than silently stopping.
Buyer-facing messages and logs are all in English.
Quality report is linear (single pass, computed only when requested) — heterogeneous data
(unique field per row) no longer turns the report into a multi-minute step.
Phone cleaning is shape-gated like email/URL: values that aren't phone-shaped (e.g.
"Order #12345678 pending") are left untouched instead of being reduced to digits.
Billing is per-record with an ambiguity-safe policy: a record is charged right after it's
delivered (small abort/migration exposure), and an ambiguous charge error drops that single unit
rather than re-sending it — so a lost response never double-charges.
Robust dataset loading: pages are retried on transient errors, a short page no longer ends the
load, and an under-reported total can't cut it short (termination is empty-page / cap only).
Push circuit-breaker: repeated push failures abort the run instead of burning hours on backoff.
Memory: minimum memory raised to 512 MB and intermediate lists freed after use to avoid OOM
near the 50k cap.
Type coercion preserves leading zeros (ZIP/IDs); collapse guard also covers small inputs.
Fuzzy no longer chains near-duplicates (data-loss fix): switched from single-linkage union-find to
leader/canopy clustering — a record joins a cluster only if it matches the cluster's representative,
not any member. A gradient of strings (A…A → B…B) no longer collapses records that are 0% similar.
Fuzzy cost budget is now quadratic (pairs × len²) plus a pair-count cap and a 128-char comparison
cap — near-duplicate inputs that previously ran for minutes now degrade to normalized instantly.
Billing no longer under-charges on transient errors: units stay queued and retry on the next flush
instead of being dropped; charges are counted only after a confirmed success. Charge batching (50)
cuts platform round-trips.
Pagination treats a total of 0 as "unknown" (Crawlee's default) rather than end-of-data.
Collapse guard: if more than 70% of the input is removed as duplicates, the run says so in its
status and quality report (heavy collapse is never silent).
Field-type detection is safe on non-string keys; SEO title trimmed to ≤50 chars; docs aligned.
Fuzzy cost-aware budget: comparison strings are capped at 256 chars and the budget now measures
work (pairs × length), not just pair count — long-string inputs (e.g. 500 rows × 600 chars) that
previously ground for ~55s now degrade instantly to normalized instead of risking a run timeout.
Full pagination fix: no longer stops on a short-but-non-empty page (Apify caps responses ~9MB for
large records) — only an empty page / dataset total / the cap end the load. Removes silent data loss.
Empty key values: None/"" in a dedup key now yield an empty signature → the record stays
unique (previously None serialized to "null" and could merge distinct records).
Billing free-flip narrowed: only an unambiguous "not pay-per-event" signal flips to free; a
transient charging error stays fail-closed (no silent free delivery of a paid run).
Output schema (.actor/output_schema.json, wired via output): declares the clean dataset and
the QUALITY_REPORT KVS output with link templates, so AI agents (MCP) can discover and chain it.
Prefill fixed so the default run actually removes a duplicate (demonstrates the product).
Publish checklist (PUBLISH.md) documents the required result-item pay-per-event registration.
Fixes from the L4 adversarial review before first publish:
Data-loss guard (critical): a record with an empty signature — a missing/renamed dedupKeys
field, or a value that is only punctuation/emoji/CJK — is now treated as unique and never
merged with another empty. Previously a whole dataset could silently collapse to one row.
dedupKeys validation: if none of the given keys exist in any record the run fails loudly;
low key coverage is warned in the status and report.
Fuzzy is bounded: hard comparison budget with a quick_ratio pre-filter and value-based
blocking; on very large or low-diversity input fuzzy auto-degrades to normalized and reports it,
instead of grinding for hours (a shared-prefix column no longer defeats blocking).
Full pagination of source datasets, with an explicit truncation warning if the 50,000 cap is hit
(no more silently processing only the first page).
Billing fail-closed: unknown pricing shape defaults to charging (never silent free delivery of a
paid run); genuine free actors are detected on the first no-charge signal.
Push isolation: per-record push retries; charges only follow a confirmed push.
Robustness: field-type detection matches on word boundaries (tel no longer matches hotel);
signature building is recursion-safe; clean+dedup is wrapped so bad input fails cleanly.
Default run now ships a prefill example so the first click returns a result instead of an error.
17 unit tests (up from 12), including regressions for every fix above.
0.1 (2026-07-23) — initial build
First version: Dataset Deduplicator & Cleaner — a non-scraper utility that plugs into any dataset.
Dedup: exact, normalized (case/whitespace/punctuation-insensitive) and fuzzy (near-duplicate,
difflib similarity with a configurable threshold, union-find clustering, blocking above 2000 rows
for scale). Keep-strategy: most-complete / first / last.
Cleaning: per-type field cleaning auto-detected by field name — emails (validate + lowercase),
phones (E.164-ish), URLs (normalize), plus whitespace collapse, control-char strip, HTML strip,
case, empty→null, drop-empty, and optional type coercion (numbers/booleans).
Quality report: input/output counts, duplicates removed, per-field fill/null rates, values
cleaned — saved to the key-value store as QUALITY_REPORT.
Zero-config by default (normalized dedup + sensible cleaning) and fully configurable.
Hardened skeleton reused from our reference actors: per-result PPE with fail-closed billing, free
mode (no charge), FAIL-on-zero. 12 unit tests (cleaners, dedup, quality).
