# Changelog of Dataset Transformer: Filter, Dedupe, JSON to CSV (`humble-echidna/dataset-transform`) Actor

- **URL**: https://apify.com/humble-echidna/dataset-transform/changelog.md
- **Full Actor documentation**: https://apify.com/humble-echidna/dataset-transform.md

## Changelog

Versions follow MAJOR.MINOR.PATCH (`src/version.py`); Apify shows MAJOR.MINOR from `.actor/actor.json`.
Every run logs its version and records it in the `RUN_STATS` key-value record.

### 1.3.2 (2026-10-16)

- A run that returns nothing (every input bad, private, blocked or unreachable, or an empty input) now ends
  SUCCEEDED with an empty dataset and the reason as its status message and in the log, instead of FAILED. Only an
  unexpected error still fails the run. `RUN_STATS` keeps the reason as before.
- No change to output fields or price.

### 1.3.1 (2026-10-07)

- An uploaded file is read with the run's own Apify token, so it works when your account keeps storage private (the
  default; 1.3.0 fetched the upload's link anonymously, which Apify refuses then). A gzip-compressed upload is
  decompressed. If Apify still refuses, the run says how to share that key-value store.

### 1.3.0 (2026-10-07)

- **Upload a file:** the file field (**Or: file URL or upload**) now has an upload button, so a CSV, JSON or Excel
  file on your computer works without hosting it somewhere first. A link still works as before.
- **Google Sheets links:** a sheet's `/edit` or share link is read through Google's export (the tab in the link, or
  the one named in `sheetName`); a published sheet's link too. A sheet that isn't shared fails with how to share it.
- A path on your own computer (`file:///C:/...`, `C:\...`) or an Apify dataset id in the file field now fails
  straight away with what to do instead, rather than "not a web address" / "no host". (All three came from real runs
  on 2026-10-06.) Same price; no change for datasets, pasted rows or direct links.

### 1.2.0 (2026-10-05)

- **Paste rows instead of a link:** the new **Or: paste rows (JSON array)** field (`items`) takes the rows directly,
  e.g. `[{"name": "Ada"}, {"name": "Bo"}]`, so an AI agent or a script holding JSON gets it filtered, deduplicated or
  converted to CSV without uploading a file first. A picked dataset still wins; pasted rows win over an integration's
  dataset and the file URL. An empty list counts as nothing pasted, so a blank box in the form leaves the file URL
  working. Same price.
- README: the "Run it automatically" steps no longer say an empty **Apify dataset** cleans the example file or that
  the integration's `payload` is ignored (both untrue since 1.1.3); the first screen points scraper users to the
  worked example; where to download `OUTPUT.csv` from the API.
- Store description: names deduplicate, pasted JSON, XLSX, CSV to JSON and Excel to JSON.

### 1.1.3 (2026-10-04)

- Started by an Actor-to-Actor integration or a webhook with **Apify dataset** left empty, the run now reads the
  finished run's dataset (`payload.resource.defaultDatasetId`, then `resource.defaultDatasetId`) instead of the
  pre-filled example file. A filled-in dataset still wins. No input-schema, output or price change.
- README: the price line no longer compares us with other actors.

### 1.1.2 (2026-10-04)

- No change in what it does, its input or its price.
- README: a worked example, "Example: scraper results to a clean CSV": a Google Maps-style scraper's 15 rows (two
  searches, so places repeat) filtered, deduplicated by `placeId`, sorted and trimmed to 12 renamed columns, with the
  before and after tables, the input and the CSV from a local run.
- README: "Run it automatically after your scraper finishes": the same input as an Actor-to-Actor integration
  (`{{resource.defaultDatasetId}}`), and as a webhook on `ACTOR.RUN.SUCCEEDED` created through the API.

### 1.1.1 (2026-10-03)

- No change in what it does. The file and dataset reader, the row operations and the file export now live in a
  shared package (`mms_rows`) that the new Dataset Diff and Join Datasets actors read with too, so a fix
  to reading CSV, JSON or Excel reaches all three. The "that is this run's own output dataset" error now says "pick
  the dataset with your rows".
- README: Related actors links Dataset Diff and Join Datasets.

### 1.1.0 (2026-10-02)

- **Reads Excel files:** a link to an `.xlsx` (or `.xlsm`) workbook in **Or: file URL** is detected from the file
  itself and read like any other source: one row per spreadsheet row, keyed by the first non-empty row. Numbers and
  booleans keep their type, dates become ISO 8601 text, formulas give the value Excel last saved, empty rows are
  skipped. **Excel: sheet name** picks a sheet other than the first, and **Excel: header row** the row with the column
  names when a title sits above the table. Excel to JSON, or to CSV with **Also save as a file**, in one step.
- Numbers are written to Excel's own 15 significant digits (`15.7`, not the `15.700000000000001` some tools save).
- Two new ready-to-run examples: Excel to JSON or CSV, and a table that starts below a title.
- Old `.xls` and password-protected workbooks fail with what to do instead (save as .xlsx or CSV). Workbooks that
  would unzip past 1 GB, or 150 MB of text, are refused before they're read.
- No change to price, or to how datasets, CSV and JSON files are read.

### 1.0.8 (2026-10-01)

- **Waits out longer `Retry-After` answers:** when a site or API asks it to wait before retrying, it now waits up
  to 90 seconds (was 30), once per site, with at most 3 minutes of such long waits per run; a longer wait, or a
  site that asks again after it was waited out, is reported with the reason instead of stalling the run.
- No change to input, output or price.

### 1.0.7 (2026-10-01)

- **Remove duplicates by** accepts comma-separated fields on one line (`name, city`), as agents often send them;
  before, that line was read as one field name nobody had, so no duplicates were removed. A field whose name has a
  comma goes in double quotes.
- A row with a broken character (half an emoji, left by a scraper that cut text mid-character) no longer fails the
  run: the character becomes �, the row is kept, and the log and `RUN_STATS` (`rowsTextRepaired`) count it.
- Filters and duplicate removal treat text that looks the same as the same ("é" as one character or as "e" plus an
  accent). Your values are written unchanged.
- Files saved as UTF-16 (Excel "Unicode Text", Windows PowerShell) are read correctly, and a JSON or JSON Lines file
  in Windows-1252 is read like a CSV in it, instead of failing or garbling accented letters.

### 1.0.6 (2026-09-29)

- Three more example views (JSON API to CSV, CSV to JSON, top rows) for the new ready-to-run examples. Your own rows are unchanged.

### 1.0.5 (2026-09-29)

- README: step-by-step chaining after any actor through its Integrations tab (`{{resource.defaultDatasetId}}`).

### 1.0.4 (2026-09-28)

- 1 ready-to-run example published on Apify Store and linked from the README.

### 1.0.3 (2026-09-28)

- Dataset schema with an "Example: exchange rates" view for the pre-filled example. Your own rows are unchanged and not validated against it: they keep whatever fields you choose.

### 1.0.2 (2026-09-28)

- Cheaper on paid Apify plans: $0.90 per 1,000 rows on Starter, $0.80 on Scale and $0.70 on Business ($1.00 on the Free plan).

### 1.0.1 (2026-09-25)

- Input field descriptions rewritten for AI agents (Apify's MCP server shows agents only the description, not the
  form): each now states its default, its allowed range, the value of each option and how it combines with other
  fields. No change to field names, types, defaults or behaviour.

### 1.0.0 (2026-09-24)

First release.

- Reads one source per run: one of the user's own Apify datasets (picked in the input, read page by page with the
  run's own token, read-only), or a CSV/TSV, JSON (an array, or a list inside an object via `itemsPath`) or JSON
  Lines file from a URL the user supplies (up to 200 MB, gzip or not; format, delimiter and encoding detected).
- Transforms, in this order: flatten nested objects and arrays into dotted fields (optional); filters (`==`, `!=`,
  `>`, `>=`, `<`, `<=`, `contains`, `not contains`, `starts with`, `ends with`, `is empty`, `is not empty`; all or
  any); remove duplicates by one or more fields (or whole rows with `*`), keeping the first; sort by one or more
  fields, each ascending or descending; keep and rename fields. Dotted paths reach nested values everywhere.
- Writes the rows to the default dataset, and optionally also as one CSV or JSONL file in the key-value store
  (`OUTPUT.csv` / `OUTPUT.jsonl`, up to 100 MB, no extra charge).
- Charged per output row through Apify's standard `apify-default-dataset-item` event ($1.00 per 1,000 rows) plus
  the $0.00005 start fee. Rows that are empty after the transformation, or larger than 5 MB, are skipped and never
  charged.
- Memory-bounded: files stream to disk and are parsed a row at a time; duplicates are remembered as 64-bit
  fingerprints (about 110 bytes of memory per distinct row, measured); sorting keeps only the top rows when a row
  limit applies, and otherwise holds at most 100 MB of rows (the run says so and writes nothing if the input is
  larger). At most 1,000,000 rows are read per run.
  Measured locally on a 1,000,000-row, 179 MB CSV: peak memory 96-116 MB when streaming, filtering, deduping and
  keeping the top 1,000; 303 MB for a sort that reaches the 100 MB cap. Default memory 1024 MB, minimum 512 MB.
- "Max rows per run" input and the maximum cost per run are honoured: without sorting, reading stops as soon as
  the limit is covered; with sorting, the limit selects the top rows.
- Clear failures: a dataset the run can't read, the run's own output dataset, a missing file, a web page instead
  of a file, a JSON object without `itemsPath` (the error names the lists it found), or an invalid filter all fail
  the run with the reason before anything is charged. A source that breaks midway keeps the rows already written
  and reports the problem. Fields named in the input that no row has are reported.

Sources (ACTOR-CHECKLIST.md, section 0 and 6):

- The actor has no source of its own: it reads what the user names. Apify datasets through Apify's own API
  (https://docs.apify.com/api/v2/dataset-items-get) with the run's token, which reaches only datasets the user can
  access; the input field declares READ-only access for limited-permission runs
  (https://docs.apify.com/platform/actors/development/actor-definition/input-schema/specification/v1, read
  2026-09-24). Files at user-supplied URLs are fetched through mms_common (robots.txt, private-network guard,
  ports 80/443, size cap), the PLAN.md legal-hold "tools that only process URLs or files the user supplies" case.
- The default input is the US Treasury Fiscal Data API, `v1/accounting/od/rates_of_exchange` as CSV
  (https://fiscaldata.treasury.gov/api-documentation/, read 2026-09-24): "Our API is open, meaning that it does not
  require a user account or registration for a token", and "The data is offered free, without restriction, and
  available to copy, adapt, redistribute, or otherwise use for non-commercial or commercial purposes." Its
  robots.txt allows the path. The date is fixed (2025-12-31), so the default output doesn't change.