Dataset Cleaner: Filter, Dedupe & Rename Rows [$1/1K]

Clean a scraper export before it goes to a CRM, a sheet or the next Actor: filter rows by value or date, remove duplicates, and rename, trim or cast fields. Paste a few rows, or point datasetId at the Dataset an upstream run just wrote. The default input keeps one sample row: Northwind, open, ada@example.com.

Transforms run first, in order. Filters then decide which of those rows are written. The output rows are your fields, not a wrapper. Counts, failures and the next step are in the Key-Value Store record RUN_SUMMARY.

A run that keeps every row costs $1 per 1,000 rows: $0.02 for rows read and $0.98 for rows written. A filter that keeps nothing pays only the $0.02 read price. There is also a $0.00005 start charge. Platform usage is included in these prices. Set maxTotalChargeUsd on API and agent runs. A small test fits in $0.01.

Try the sample

Leave the prefilled rows, the status = open filter, and the trim / lowercase / rename steps. Run it. The Dataset should contain:

{"companyName": "Northwind", "status": "open", "email": "ada@example.com"}

Closed Co is filtered out. It is still counted in RUN_SUMMARY.rowsRead and rowsFilteredOut.

For a real export, pick the source Dataset. If the sample rows are still filled in, datasetId wins and those rows are ignored. RUN_SUMMARY.inputWarning says so. maxItems is 100 in the demo. Delete it to process the whole Dataset. If at least 50 rows are read and every one fails the same rule, the run stops before you are charged. The same stop happens when those rows all become empty objects, which is what a keep of the wrong field names does. A smaller source with a bad cell finishes as PARTIAL with the failure codes. When fewer than 50 rows come out of keep, drop or rename with no fields left, those rows are not written and are not charged as kept; RUN_SUMMARY.skipCodes.EMPTY_ROW counts them. If a rule names a field that none of those rows have, the run continues and RUN_SUMMARY.inputWarning names the field, so a typo or a leftover sample rule is easy to spot. The same warning names the cast to add when a number or boolean filter reads a field that holds "12.50" or "true" as text.

Run it again

The useful second run is the next export, not a second copy of the same rows. Save a Task with the rules and leave datasetId empty:

{
  "filterMode": "AND",
  "filters": [{"op": "equals", "field": "status", "value": "open"}],
  "transforms": [
    {"op": "trim", "field": "company"},
    {"op": "lowercase", "field": "email"},
    {"op": "rename", "field": "company", "to": "companyName"}
  ]
}

On the scraper, open Integrations, connect this Task, and pass datasetId from {{resource.defaultDatasetId}}. The integration fills that placeholder when the scraper finishes. It also adds a payload object to the input, which this Actor accepts and does not read. If the integration input has no datasetId, the run fails and says what to set, instead of cleaning sample rows saved in the Task. A saved Task stores the text as written, so the placeholder fails the 17-character dataset id check on every start. A schedule starts from the Task input as saved, so it keeps reading one Dataset. Same rules, a new Dataset each time.

Recipes

Copy one into the input, or into a Task you call from a webhook. Each is a complete input apart from datasetId.

Clean crawler output for an LLM or RAG pipeline

Keeps the URL and Markdown of each page, drops pages without text and writes each URL once. Because keep comes first, the heavy html and text fields are never downloaded. Only top-level fields are read. Website Content Crawler stores the page title in metadata.title, and that nested value is not available here.

{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "keep",
      "fields": [
        "url",
        "markdown"
      ]
    }
  ],
  "filters": [
    {
      "op": "notEmpty",
      "field": "markdown"
    }
  ],
  "dedupeBy": [
    "url"
  ]
}

Prepare scraped leads for a CRM

Trims the company name, lowercases the email, keeps only leads with an email and writes each email once.

{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "trim",
      "field": "title"
    },
    {
      "op": "rename",
      "field": "title",
      "to": "company"
    },
    {
      "op": "lowercase",
      "field": "email"
    }
  ],
  "filters": [
    {
      "op": "notEmpty",
      "field": "email"
    }
  ],
  "dedupeBy": [
    "email"
  ]
}

Keep only the last 7 days on a schedule

now-7d is measured from each run's start, so a saved Task keeps filtering the last week. Dates must be ISO 8601, such as 2026-10-01 or 2026-10-01T08:00:00Z. If no value of that field in the first rows is such a date, for example Oct 5, 2026 or a Unix timestamp, RUN_SUMMARY.inputWarning says so, so an empty result is not mistaken for no new rows.

{
  "datasetId": "YOUR_DATASET_ID",
  "filters": [
    {
      "op": "gte",
      "field": "postedAt",
      "value": "now-7d"
    }
  ]
}

Clean prices before filtering by amount

Turns "$1,299.00" into 1299 and keeps prices above 10. A cell with no digits, such as "N/A", becomes null and is filtered out instead of failing the row. For prices written with a decimal comma, such as 12,50, replace the comma with a dot first.

{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "regexReplace",
      "field": "price",
      "pattern": "[^0-9.]",
      "replacement": ""
    },
    {
      "op": "cast",
      "field": "price",
      "to": "number"
    }
  ],
  "filters": [
    {
      "op": "gt",
      "field": "price",
      "value": 10
    }
  ]
}

Check the rules on 20 rows first

maxItems stops after 20 kept rows, so a run on a large Dataset costs about two cents. Read RUN_SUMMARY before running it on everything.

{
  "datasetId": "YOUR_DATASET_ID",
  "filters": [
    {
      "op": "equals",
      "field": "status",
      "value": "open",
      "caseSensitive": false
    }
  ],
  "maxItems": 20
}

What you can do

Transform	What it writes
trim, lowercase, uppercase	Changes a string in field, or in each name in fields. With one field and to, writes the result to that new field and keeps the original. Null, a missing field, and other types stay as they are.
rename	Moves field to to.
drop / keep	Removes fields, or writes only the named ones.
cast	to is string, number, integer or boolean. integer fails a value with a fraction instead of rounding it. Null and a missing field stay as they are. A blank string becomes null for number and boolean. A value of the wrong type drops the row. Integers with more than 15 digits are a failed cast.
default	Sets field only when it is missing, null or "". 0 and false stay.
regexReplace	Replaces matches in the field's text. A number or boolean is read as written and the result is text. An empty or missing replacement removes the matches. Null and a missing field stay as they are.
regexExtract	Writes the first capturing group that matched, or the whole match when there are no groups. A number is read as written. A miss writes null and keeps the row.
template	Writes field from {{other}} placeholders.

Filter	Matches when
equals / notEquals	Values are equal. true is not 1. A missing field is not equal to null. Text matches letter case unless "caseSensitive": false.
contains / notContains	A string contains the text, or a list contains the value.
startsWith / endsWith	The field is a string with that prefix or suffix.
regex	The pattern is found in the field's text. A number is matched as written; lists and objects never match.
in / notIn	value is a list.
gt / gte / lt / lte / between	Both sides are JSON numbers, or ISO 8601 dates when value is a date such as "2026-10-01". "now", "now-7d", "now-24h" and "now-30m" count from the run start. A date without a time is midnight UTC and a time without a zone is UTC, so lte "2026-10-31" stops at the start of that day; inputNotes says so. Cast a numeric string first.
empty / notEmpty	Missing, null, "", [] or {}. 0 and false are not empty.
true / false	The field is a boolean, not the string "true". Cast a text field to boolean first.

filterMode is AND or OR. There is no nested expression tree. Top-level fields only: metadata.title is the field name metadata.title, not the title inside metadata. Filters and dedupeBy read the field names after the transforms. A name that an earlier rename, drop or keep removed fails the run before a row is read, and the message names the field to use, for example use state after status was renamed to state.

dedupeBy lists the fields that identify a row, such as ["email"] or ["name", "city"]. After the transforms and filters, only the first row for each value is written. Later rows are counted in rowsDuplicate and charged as read, not kept. Values compare exactly, so lowercase an email first. Rows whose key fields are all empty are kept. One run checks up to 1,000,000 distinct keys; past that it stops as DEDUPE_LIMIT with the rows written so far.

Rule format

Each rule is an object with op, field and the keys its op takes: value for comparisons, pattern for regex, and to, fields, replacement or template for transforms. The input form lists them per op.

Text comparisons (equals, notEquals, contains, notContains, startsWith, endsWith, in, notIn, regex) match letter case exactly. Add "caseSensitive": false to match any case.

A spelling with only one possible meaning is read in the documented form and noted in RUN_SUMMARY.inputNotes: a number written as text for gt, gte, lt, lte or between, {"min": 1, "max": 5} or "[1, 5]" for between, values or a JSON list written as text for in and notIn, value as the regex pattern, plain text in pattern for a text comparison (text with regex syntax still fails, use regex), "ignoreCase": true for "caseSensitive": false, "value": null on empty, notEmpty, true or false, int, float, bool or text as a cast target, from for rename, field for keep, to as the value of default, and a missing replacement. Descriptive keys such as description, comment, label or id are ignored.

Anything that could mean two things, such as an unknown key or a comma-separated string for in, fails the run before a row is read. The message names the rule, the key and the fix, and RUN_SUMMARY.inputErrors lists every rule problem at once, so one retry can fix them all. Each entry has rule, op, reason (MISSING_KEY, BAD_VALUE, UNKNOWN_KEY, UNUSED_KEY, INVALID_REGEX, FIELD_REMOVED, NO_SOURCE, DATASET_ID_ONLY_IN_PAYLOAD), key, the keys the op accepts and message; a key that does not apply is null. A run that stops on its rules after reading rows has one entry with RULE_FAILS_EVERY_ROW, EMPTY_EVERY_ROW or REGEX_TOO_SLOW.

How a run ends

The Actor run succeeds when it could read the source and apply the rules, including when nothing matched. Read RUN_SUMMARY.status:

Status	Meaning
SUCCESS	At least one row was written, and no transform failed.
NO_RESULTS	The source was read and nothing was written.
PARTIAL	Some rows failed a transform (in a source under 50 rows, possibly all of them), the charge limit stopped the run, or a kept row was read but not written. rowsUnwritten counts kept rows that were not stored. transformFailureSamples lists up to 20 source indexes.
SKIPPED_CHARGE_LIMIT	The budget could not pay for one row. Nothing was read.
FAILED	The input, dataset access, or a charge could not be trusted. An input problem names the rule and the key to fix.

rowsSkipped counts rows that were not written because they are not JSON objects or had no field left after the transforms, and skipCodes counts each reason (NON_OBJECT, EMPTY_ROW). Skipped rows pay only the read price.

stoppedReason is COMPLETE, MAX_ITEMS, BUDGET, TIME_LIMIT, DEDUPE_LIMIT or FAILED. TIME_LIMIT means the run stopped 30 seconds before its timeout with the rows it had written and a PARTIAL summary, instead of being cut off; raise the run timeout to process the rest. If the platform restarts the run in the middle of a batch, the Actor checks the charges and rows that batch left behind before it goes on, so rows are not charged twice. When that check cannot tell, the run fails with BILLING_UNCERTAIN instead of guessing. Start a new run and check the rows already written. transformFailureCodes names the rule, for example transforms[0]:CAST_FAILED. REGEX_TIMEOUT means that pattern took longer than 50 milliseconds on that row, so the row was dropped and the run continued. REGEX_FAILED means the replacement does not fit the pattern.

Price

Event	Price	Charged when
processed-row	$0.00002	A row was read and a decision was made.
kept-row	$0.00098	A transformed row was written.
Actor start	$0.00005	Once per run.

1,000 rows, all kept: $1 plus the start. 1,000 rows, none kept: $0.02 plus the start. Rows left unread because of maxItems or maxTotalChargeUsd are not charged.

Limits

Rows of any size work at the default memory: the Actor reads the source in pages sized from the rows it has seen. For heavy rows, such as crawler output with html or markdown, put keep or drop first: those fields are then never downloaded, which makes the run faster and cheaper.

This does not run JavaScript or Python that you supply. It does not join, pivot or call an AI model. dedupeBy removes duplicate rows inside one run. Inline data stops at 1,000 rows. A text value longer than 100,000 characters that trim, lowercase, uppercase, a string cast, regexReplace, regexExtract, template or the regex filter reads drops that row with FIELD_TOO_LARGE. A regular expression that runs longer than 50 milliseconds drops that row with REGEX_TIMEOUT. After 200 such rows the run stops, because the pattern needs to be simpler. outputDatasetName is optional. Leave it empty to use this run's default Dataset, which is empty at the start of every run. A named dataset is appended to, so a schedule writes another copy of the kept rows each time. It cannot be the same Dataset you are reading.

For an agent

Ask it to filter one Dataset or a small JSON array, with explicit maxItems and maxTotalChargeUsd. It should build each rule from the op list in the input schema and read RUN_SUMMARY before fetching rows. If a run fails on its input, RUN_SUMMARY.inputErrors lists every rule to change, with the keys each op accepts. SUCCESS means the kept rows are in the Dataset linked from the run output. NO_RESULTS means the source was read and the filters excluded everything. Do not treat a failed dataset read as an empty export.

What changed is in the changelog.

Direct MCP tool: https://mcp.apify.com?tools=automa-flow/dataset-filter-transform

Example ask: "From dataset DATASET_ID, keep rows whose status is open, trim company, lowercase email, rename company to companyName, and stop at 100 kept rows."