# Changelog of Dataset Cleaner: Filter, Dedupe & Rename Rows \[$1/1K] (`automa-flow/dataset-filter-transform`) Actor

- **URL**: https://apify.com/automa-flow/dataset-filter-transform/changelog.md
- **Full Actor documentation**: https://apify.com/automa-flow/dataset-filter-transform.md

## Changelog

### 2026-10-14: Integration runs start, and skipped rows say why (0.1.8)

#### Fixes

- A run started from a scraper's Integrations tab no longer fails to start. The integration adds a `payload` object to the input, which the input schema refused; it is now accepted and not read. If the integration input has no `datasetId`, the run fails and says to set `"datasetId": "{{resource.defaultDatasetId}}"`, instead of cleaning sample rows saved in the Task.
- `RUN_SUMMARY.skipCodes` says why rows were skipped: `EMPTY_ROW` when the transforms left no field, `NON_OBJECT` when a row is not a JSON object. A successful run names the emptied rows in its next step.
- A filter or `dedupeBy` on a field an earlier `rename`, `drop` or `keep` removed is reported together with the other rule problems before any row is read, so one retry fixes them all. A rename chain names every step and the last name.
- Every `RUN_SUMMARY.inputErrors` entry has the same keys: `rule`, `op`, `reason`, `key`, `accepts` and `message`.

#### Improvements

- When a number or boolean filter reads a field that holds `"12.50"` or `"true"` as text, `inputWarning` names the `cast` to add before the filter.
- A template placeholder that no row has is named in `inputWarning`, like any other missing field.
- A bare date as the upper bound of `lte` or `between`, such as `"2026-10-31"`, gets a note in `inputNotes`: it means midnight UTC, and the note shows how to include the whole day.
- The status line no longer starts with "datasetId is set, so the inline data was ignored" on every Console run with a picked Dataset. `RUN_SUMMARY.inputWarning` still says it.
- The README has a recipe for cleaning prices such as `"$1,299.00"` before a numeric filter, and lists the 100,000-character text limit.

### 2026-10-13: Empty keeps and filters after a rename (0.1.7)

#### Fixes

- A `keep` that names fields the rows do not have no longer writes an empty object for each row or charges the kept-row price. Those rows are not written. When the first 50 or more rows all become empty, the run fails before any row is charged and names the fields.
- A filter or `dedupeBy` that reads a field an earlier `rename`, `drop` or `keep` removed fails before any row is read. The message names the field to use, for example `state` after `status` was renamed to `state`.

#### Documentation

- The input schema describes date comparisons on `gt`, `gte`, `lt`, `lte` and `between`, including `2026-10-01` and `now-7d`.
- The crawler recipe keeps `url` and `markdown`. A title stored at `metadata.title` is not read: only top-level fields are.
- To clean each scraper run, save a Task without `datasetId`. On the scraper, open Integrations, connect that Task, and pass `datasetId` from `{{resource.defaultDatasetId}}`. A schedule does not fill that placeholder.
- The limits section no longer says this Actor cannot deduplicate.

### 2026-10-06: Remove duplicates and filter by date (0.1.6)

#### Features

- `dedupeBy` removes duplicate rows: list the fields that identify a row, such as `["email"]` or `["url"]`, and only the first row for each value is written. Duplicates are charged as read, not kept, and counted in `rowsDuplicate`.
- `gt`, `gte`, `lt`, `lte` and `between` compare ISO 8601 dates, and take `"now-7d"`, `"now-24h"` or `"now-30m"` so a scheduled Task keeps filtering a moving window. If the field holds no ISO date in the first rows, such as `Oct 5, 2026`, `inputWarning` says so instead of a silent empty result.

#### Improvements

- The README has recipes for cleaning crawler output for an LLM, preparing leads for a CRM, keeping the last 7 days, and checking rules on 20 rows first.

### 2026-10-06: Large rows and short timeouts no longer fail the run (0.1.5)

#### Fixes

- A source of large rows, such as crawler output with `html` or `markdown`, no longer runs out of memory. Pages are now sized from the rows already read; before this, 1,000 rows of about 150 KB stopped the run at the default 512 MB.
- A run that is about to reach its timeout stops 30 seconds early with `stoppedReason: "TIME_LIMIT"` and a `PARTIAL` summary, so the rows it wrote and the summary are kept.
- When every row of a large source fails the same rule, the message shows the first value it read, for example `"$12.99"`, so you can see what to clean.

#### Improvements

- When `keep` or `drop` comes first, the fields they remove are not downloaded at all.

### 2026-10-05: Case-insensitive filters and input errors fixed in one retry (0.1.4)

#### Improvements

- Text filters take `"caseSensitive": false` to match any letter case: `equals`, `notEquals`, `contains`, `notContains`, `startsWith`, `endsWith`, `in`, `notIn` and `regex`. `"ignoreCase": true` is read the same way.
- `trim`, `lowercase` and `uppercase` take `to` to write the result to a new field and keep the original.
- More rules run as meant, with a note in `inputNotes`: plain text in `pattern` for a text comparison, a JSON list written as text for `in`, `notIn` and `between`, and `to` as the value of `default`. Descriptive keys such as `description` or `id` are ignored.

#### Fixes

- A run that fails on its input lists every rule problem in `RUN_SUMMARY.inputErrors`, with the rule, the reason, the key and the keys that op accepts, so all of them can be fixed in one retry.
- `equals` without `value` now suggests `notEmpty` or `empty` when the aim is to check whether a field is set.

### 2026-10-05: Rules written by agents and integrations run as meant (0.1.3)

#### Improvements

- A rule written in a form with only one meaning now runs instead of failing, and `RUN_SUMMARY.inputNotes` shows how it was read: a number written as text for `gt`, `gte`, `lt`, `lte` and `between`, `{"min", "max"}` for `between`, `values` for `in` and `notIn`, `value` as the `regex` pattern, `"value": null` on `empty`, `notEmpty`, `true` and `false`, `int`, `float`, `bool` or `text` as a cast target, `from` for `rename`, `field` for `keep`, and a missing `replacement`, which removes the matches.
- `cast` takes `integer` as a target. A value with a fraction, such as 12.5, is a failed cast for that row rather than a rounded number.
- `trim`, `lowercase` and `uppercase` accept `fields` to change several fields with one rule.
- `regexReplace`, `regexExtract` and the `regex` filter read a number or boolean as its written text, so a phone number stored as a number can be cleaned.
- The input form now lists the keys each `op` takes, and a filter without `field` is rejected before the run starts, so it is not charged.
- A pattern written as a JavaScript literal, such as `/abc/i`, and a template with `{single}` braces are flagged in `inputNotes`, because both are read as plain text.

#### Fixes

- An input problem names the rule, the key and the fix, for example `filters[0]: unknown key "caseSensitive"; equals takes "op", "field", "value"`, instead of `Extra inputs are not permitted`.
- A source of fewer than 50 rows in which every row fails the same rule finishes as `PARTIAL` with the failure codes instead of failing the run. The status names the most common failure.
- A blank string in a `number` or `boolean` cast becomes null instead of dropping the row.

### 2026-10-03: Filter and reshape a Dataset without code (0.1.2)

#### Features

Paste rows or pick an Apify Dataset, then apply declarative transforms and filters. The run writes the kept rows and a `RUN_SUMMARY` with counts, transform failures and the next step.

You pay $0.00002 for each input row that is read and $0.00098 for each row that is written. A run that keeps every row is $1 per 1,000 rows. Filtered rows are not charged the kept-row price. The run also has the $0.00005 start. Set `maxTotalChargeUsd` before an API or agent run.

The sample input keeps one open company, trims its name, lowercases the email and renames `company` to `companyName`. Use the same rules from a webhook when the next scraper finishes.

#### Fixes

- A pattern with several groups, such as `(a)|(b)`, keeps the group that matched instead of an empty first group.
- A replacement the pattern cannot apply drops each row it touches with `REGEX_FAILED`, instead of failing the run.
- A regular expression that takes longer than 50 milliseconds drops that row with `REGEX_TIMEOUT`. The run continues, and stops after 200 such rows.
- Null and missing fields are left as they are. A cast or text change fails only when the value has the wrong type.
- If you set `datasetId` and the sample rows are still filled in, the Dataset is used and the pasted rows are ignored.
- If every row in the first batch fails the same rule, the run fails before you are charged. A rule field that none of those rows have is reported in `inputWarning` and the run continues.
- A schedule that writes to the same `outputDatasetName` every time keeps working when that dataset already holds rows from earlier runs.
- A run that the platform restarts in the middle of a batch checks what that batch already charged and wrote, so the same rows are not charged twice.
- Casting an integer with more than 15 digits drops that row instead of writing a value other tools cannot read.
- The Actor refuses to write into the Dataset it is reading.