Dataset Cleaner & LLM Processor: Filter, Dedupe, AI per Row

Clean a scraper export before it goes to a CRM, a sheet or the next Actor: filter rows by value or date, remove duplicates, and rename, trim or cast fields. Paste a few rows, or point datasetId at the Dataset an upstream run just wrote. The default input keeps one sample row: Northwind, open, ada@example.com.

Transforms run first, in order. Filters then decide which of those rows are written. The output rows are your fields, not a wrapper. Counts, failures and the next step are in the Key-Value Store record RUN_SUMMARY.

A run that keeps every row costs $1 per 1,000 rows: $0.02 for rows read and $0.98 for rows written. A filter that keeps nothing pays only the $0.02 read price. There is also a $0.00005 start charge. Platform usage is included in these prices. Set maxTotalChargeUsd on API and agent runs. A small test fits in $0.01.

An llm transform puts an AI answer in every row: B2B or B2C, an industry, a field pulled out of a description, a sentiment or a translation. It runs through Apify's own apify/openrouter, so you need no API key of your own. See Add an AI column.

Add an AI column

Classify Google Maps leads as B2B or B2C before your CRM. The prompt reads each row's fields through {{placeholders}}; the answer goes into segment:

{
  "datasetId": "YOUR_DATASET_ID",
  "filters": [{"op": "notEmpty", "field": "website"}],
  "transforms": [
    {
      "op": "llm",
      "prompt": "Is this business B2B or B2C? Name: {{title}}. Category: {{categoryName}}. Answer with one word: B2B or B2C.",
      "outputFields": {"segment": ["B2B", "B2C"]},
      "model": "openai/gpt-4o-mini",
      "maxTokens": 16
    },
    {"op": "keep", "fields": ["title", "categoryName", "website", "phone", "segment", "llmError"]}
  ],
  "maxLlmRows": 1000,
  "maxLlmCostUsd": 1
}

The filter reads website, which the model does not change, so it runs before the model: leads without a website cost no model call. A filter on segment would run after it, for example to keep only B2B rows. Put such filters, and dedupeBy on a raw field such as placeId, in the same input to send fewer rows to the model.

After each scraper run: on the Google Maps scraper (or its Task), open Integrations, connect a Task of this input with datasetId left empty, and pass "datasetId": "{{resource.defaultDatasetId}}" in the integration input.

1,000 leads, all kept with an answer: $0.02 read + $0.98 kept + $1.00 for AI rows = $2.00, plus the model usage that apify/openrouter bills to your Apify account (about $0.01 for 1,000 short gpt-4o-mini answers on a paid plan; the FREE plan pays 10x). RUN_SUMMARY.llmUsageCostUsd shows what the proxy reported for your run.

Pull fields out of job descriptions. With outputFields, each answer is checked against its type and written to its own column; an answer that does not fit is a row failure with llmError, never a guessed value:

{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "llm",
      "prompt": "From this job post, give the salary as written, whether the job is remote, and the seniority.\n\nTitle: {{title}}\n\n{{description}}",
      "outputFields": {
        "salary": "string",
        "remote": "boolean",
        "seniority": ["junior", "mid", "senior", "lead", "unknown"]
      },
      "system": "Answer from the post only. Write \"none\" for a salary the post does not give.",
      "maxTokens": 128
    }
  ]
}

Translate review text, and skip the rows that are empty:

{
  "datasetId": "YOUR_DATASET_ID",
  "filters": [{"op": "notEmpty", "field": "text"}],
  "transforms": [
    {
      "op": "llm",
      "prompt": "Translate this review into English. Reply with the translation only.\n\n{{text}}",
      "to": "textEnglish",
      "maxTokens": 512
    }
  ]
}

How the llm step works:

• to takes a text answer, trimmed. outputFields takes 1-10 typed answers: "string", "number", "integer", "boolean" or a list of allowed answers. Each row the step reaches also gets llmError: null, or the code of why it has no answer.
• model is any OpenRouter model id; the default is openai/gpt-4o-mini. Tested on this Actor: openai/gpt-4o-mini, openai/gpt-4.1-nano, openai/gpt-4.1-mini, google/gemini-2.5-flash-lite and google/gemini-2.5-flash. Another id runs at that model's own price, and RUN_SUMMARY.inputWarning says so.
• maxTokens is 256 by default and at most 2,048, the longest answer apify/openrouter returns on the FREE plan. temperature is 0 by default. system is sent as written before every prompt.
• onError: keep (default) writes the row with null answers and the code in llmError; drop leaves the row out, charged as read only.
• Placeholders read nested fields such as {{metadata.title}}. A missing field is empty, an object or list is inserted as JSON, and a field that no row of the first batch has is named in RUN_SUMMARY.inputWarning. A filled-in prompt over 20,000 characters is a row failure, LLM_PROMPT_TOO_LONG, with no call.
• Limits: llmConcurrency (default 8, at most 32) calls at a time, maxLlmRows (default 1,000) calls per run, and maxLlmCostUsd (default $1) of reported model usage. Rows past a limit get LLM_LIMIT or LLM_COST_LIMIT and follow onError. Calls in flight finish, so usage can pass maxLlmCostUsd by a few calls. Rows are sent to the model a little ahead of writing, so when maxItems, maxTotalChargeUsd or the run timeout stops a run, up to twice llmConcurrency answers can go unwritten; they are not charged as AI rows, but their model usage was spent. llmCache (default true) answers identical filled-in prompts with one call; those rows are still AI rows.
• One call waits at most 90 seconds and is not sent again after a timeout, because a slow generation can still bill model usage. 429 and 5xx answers are retried up to 3 times. If the first 20 rows sent to the model all fail with the same code, for example an unknown model id, the run stops as FAILED before more calls, and nextAction says what to change.
• One llm transform per run. Filters that read fields the llm step and later transforms do not change run before the call. In OR mode that happens only when every filter can.

Try the sample

Leave the prefilled rows, the status = open filter, and the trim / lowercase / rename steps. Run it. The Dataset should contain:

{"companyName": "Northwind", "status": "open", "email": "ada@example.com"}

Closed Co is filtered out. It is still counted in RUN_SUMMARY.rowsRead and rowsFilteredOut.

For a real export, pick the source Dataset. If the sample rows are still filled in, datasetId wins and those rows are ignored. RUN_SUMMARY.inputWarning says so. maxItems is 100 in the demo. Delete it to process the whole Dataset. If at least 50 rows are read and every one fails the same rule, the run stops before you are charged. The same stop happens when those rows all become empty objects, which is what a keep of the wrong field names does. A smaller source with a bad cell finishes as PARTIAL with the failure codes. When fewer than 50 rows come out of keep, drop or rename with no fields left, those rows are not written and are not charged as kept; RUN_SUMMARY.skipCodes.EMPTY_ROW counts them. If a rule names a field that none of those rows have, the run continues and RUN_SUMMARY.inputWarning names the field, so a typo or a leftover sample rule is easy to spot. The same warning names the cast to add when a number or boolean filter reads a field that holds "12.50" or "true" as text.

Run it again

The useful second run is the next export, not a second copy of the same rows. Save a Task with the rules and leave datasetId empty:

{
  "filterMode": "AND",
  "filters": [{"op": "equals", "field": "status", "value": "open"}],
  "transforms": [
    {"op": "trim", "field": "company"},
    {"op": "lowercase", "field": "email"},
    {"op": "rename", "field": "company", "to": "companyName"}
  ]
}

On the scraper, open Integrations, connect this Task, and pass datasetId from {{resource.defaultDatasetId}}. The integration fills that placeholder when the scraper finishes. It also adds a payload object to the input, which this Actor accepts and does not read. If the integration input has no datasetId, the run fails and says what to set, instead of cleaning sample rows saved in the Task. A saved Task stores the text as written, so the placeholder fails the 17-character dataset id check on every start. A schedule starts from the Task input as saved, so it keeps reading one Dataset. Same rules, a new Dataset each time.

Recipes

Copy one into the input, or into a Task you call from a webhook. Each is a complete input apart from datasetId.

Clean crawler output for an LLM or RAG pipeline

Keeps the URL, title and Markdown of each page, drops pages without text and writes each URL once. Website Content Crawler stores the title in metadata.title: rename lifts it to a top-level title, and drop removes the rest of metadata. Because keep comes first and names top-level fields, the heavy html and text fields are never downloaded.

{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "keep",
      "fields": [
        "url",
        "markdown",
        "metadata"
      ]
    },
    {
      "op": "rename",
      "field": "metadata.title",
      "to": "title"
    },
    {
      "op": "drop",
      "fields": [
        "metadata"
      ]
    }
  ],
  "filters": [
    {
      "op": "notEmpty",
      "field": "markdown"
    }
  ],
  "dedupeBy": [
    "url"
  ]
}

Prepare scraped leads for a CRM

Trims the company name, lowercases the email, keeps only leads with an email and writes each email once.

{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "trim",
      "field": "title"
    },
    {
      "op": "rename",
      "field": "title",
      "to": "company"
    },
    {
      "op": "lowercase",
      "field": "email"
    }
  ],
  "filters": [
    {
      "op": "notEmpty",
      "field": "email"
    }
  ],
  "dedupeBy": [
    "email"
  ]
}

Add traffic history to a lead Dataset

Keeps leads that have a website, trims and lowercases it and writes each website once, so the next Actor looks up every site one time.

{
  "datasetId": "YOUR_LEADS_DATASET_ID",
  "transforms": [
    {
      "op": "trim",
      "field": "website"
    },
    {
      "op": "lowercase",
      "field": "website"
    }
  ],
  "filters": [
    {
      "op": "notEmpty",
      "field": "website"
    }
  ],
  "dedupeBy": [
    "website"
  ]
}

Then run Website Traffic Checker with datasetId set to this run's Dataset (in an integration, {{resource.defaultDatasetId}}). It reads the website column, removes scheme, path and www., and returns monthly organic search traffic and the Chrome popularity tier for each site. That Actor runs and charges separately.

Keep only the last 7 days on a schedule

now-7d is measured from each run's start, so a saved Task keeps filtering the last week. Dates must be ISO 8601, such as 2026-10-01 or 2026-10-01T08:00:00Z. If no value of that field in the first rows is such a date, for example Oct 5, 2026 or a Unix timestamp, RUN_SUMMARY.inputWarning says so, so an empty result is not mistaken for no new rows.

{
  "datasetId": "YOUR_DATASET_ID",
  "filters": [
    {
      "op": "gte",
      "field": "postedAt",
      "value": "now-7d"
    }
  ]
}

Clean prices before filtering by amount

Turns "$1,299.00" into 1299 and keeps prices above 10. A cell with no digits, such as "N/A", becomes null and is filtered out instead of failing the row. For prices written with a decimal comma, such as 12,50, replace the comma with a dot first.

{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "regexReplace",
      "field": "price",
      "pattern": "[^0-9.]",
      "replacement": ""
    },
    {
      "op": "cast",
      "field": "price",
      "to": "number"
    }
  ],
  "filters": [
    {
      "op": "gt",
      "field": "price",
      "value": 10
    }
  ]
}

Prepare product images for a bulk download

Keeps active products that have an image URL, names each file after its SKU and writes each SKU and URL pair once. Then pass this run's Dataset ID to Parallel Bulk File Downloader with "urlField": "imageUrl", "fileNameField": "imageName" and "metadataFields": ["sku"]. Use the file extension that matches your images.

{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "trim",
      "field": "imageUrl"
    },
    {
      "op": "template",
      "field": "imageName",
      "template": "{{sku}}-main.png"
    }
  ],
  "filters": [
    {
      "op": "true",
      "field": "active"
    },
    {
      "op": "notEmpty",
      "field": "imageUrl"
    }
  ],
  "dedupeBy": [
    "sku",
    "imageUrl"
  ]
}

The downloader already shares one file and one charge between repeated URLs in the same batch, and it does not charge rows without a URL. This step pays for itself when it removes files you do not need (a file costs $0.004 to download, a kept row $0.001 here, so from about a fifth of the rows), when you need file names built from other fields, or when duplicates span more than one 10,000-row download batch. If every row already holds one URL you want, skip it.

Filter scraper output by a nested value

Many scrapers nest values inside objects, such as price.value in product rows or authorMeta.fans in TikTok rows. A dotted name reads that value. This keeps products rated 4 stars or more that cost at most 50, and writes each product once.

{
  "datasetId": "YOUR_DATASET_ID",
  "filters": [
    {
      "op": "lte",
      "field": "price.value",
      "value": 50
    },
    {
      "op": "gte",
      "field": "stars",
      "value": 4
    }
  ],
  "dedupeBy": [
    "asin"
  ]
}

The same works for TikTok videos from accounts with more than 10,000 followers, with the author name lifted to its own field:

{
  "datasetId": "YOUR_DATASET_ID",
  "transforms": [
    {
      "op": "rename",
      "field": "authorMeta.name",
      "to": "author"
    }
  ],
  "filters": [
    {
      "op": "gt",
      "field": "authorMeta.fans",
      "value": 10000
    }
  ],
  "dedupeBy": [
    "webVideoUrl"
  ]
}

Check the rules on 20 rows first

maxItems stops after 20 kept rows, so a run on a large Dataset costs about two cents. Read RUN_SUMMARY before running it on everything.

{
  "datasetId": "YOUR_DATASET_ID",
  "filters": [
    {
      "op": "equals",
      "field": "status",
      "value": "open",
      "caseSensitive": false
    }
  ],
  "maxItems": 20
}

What you can do

Transform	What it writes
trim, lowercase, uppercase	Changes a string in field, or in each name in fields. With one field and to, writes the result to that new field and keeps the original. Null, a missing field, and other types stay as they are.
rename	Moves field to to. A nested field such as metadata.title moves to the top level.
drop / keep	Removes fields, or writes only the named ones.
cast	to is string, number, integer or boolean. integer fails a value with a fraction instead of rounding it. Null and a missing field stay as they are. A blank string becomes null for number and boolean. A value of the wrong type drops the row. Integers with more than 15 digits are a failed cast.
default	Sets field only when it is missing, null or "". 0 and false stay.
regexReplace	Replaces matches in the field's text. A number or boolean is read as written and the result is text. An empty or missing replacement removes the matches. \1 and \g<name> insert a group; JavaScript $1, $<name> and $& are read the same way when the pattern has that group. Null and a missing field stay as they are.
regexExtract	Writes the first capturing group that matched, or the whole match when there are no groups. group picks one group by number or name instead, and writes null when that group did not take part in the match. A number is read as written. A miss writes null and keeps the row.
template	Writes field from {{other}} placeholders.

Filter	Matches when
equals / notEquals	Values are equal. true is not 1, and the text "5" is not the number 5; inputWarning says so when the field holds the other type. A missing field is not equal to null. Text matches letter case unless "caseSensitive": false.
contains / notContains	A string contains the text, or a list contains the value.
startsWith / endsWith	The field is a string with that prefix or suffix.
regex	The pattern is found in the field's text. A number is matched as written; lists and objects never match.
in / notIn	value is a list.
gt / gte / lt / lte / between	Both sides are JSON numbers, or ISO 8601 dates when value is a date such as "2026-10-01". "now", "now-7d", "now-24h" and "now-30m" count from the run start. A date without a time is midnight UTC and a time without a zone is UTC, so lte "2026-10-31" stops at the start of that day; inputNotes says so. Cast a numeric string first.
empty / notEmpty	Missing, null, "", [] or {}. 0 and false are not empty.
true / false	The field is a boolean, not the string "true". Cast a text field to boolean first.

filterMode is AND or OR. There is no nested expression tree. A dotted name such as price.value or metadata.title reads the value nested inside objects, in filters, dedupeBy, transforms and {{placeholders}}. A top-level field with that exact name wins, so a column literally named metadata.title reads as before. A path does not go into lists. A transform changes a nested value where it is, keep keeps it under its path, and rename moves it to the top-level name in to. A field that does not exist yet, and every to, is a top-level name. Filters and dedupeBy read the field names after the transforms. A name that only rename steps moved is read under its new name and noted in RUN_SUMMARY.inputNotes: after status is renamed to state, a filter on status reads state. A name that a drop or keep removed, or a renamed field that another transform then changed, fails the run before a row is read, and the message names the field to use.

dedupeBy lists the fields that identify a row, such as ["email"] or ["name", "city"]. After the transforms and filters, only the first row for each value is written. Later rows are counted in rowsDuplicate and charged as read, not kept. Values compare exactly, so lowercase an email first. Rows whose key fields are all empty are kept. One run checks up to 1,000,000 distinct keys; past that it stops as DEDUPE_LIMIT with the rows written so far.

Rule format

Each rule is an object with op, field and the keys its op takes: value for comparisons, pattern for regex, and to, fields, replacement or template for transforms. The input form lists them per op.

Text comparisons (equals, notEquals, contains, notContains, startsWith, endsWith, in, notIn, regex) match letter case exactly. Add "caseSensitive": false to match any case.

A spelling with only one possible meaning is read in the documented form and noted in RUN_SUMMARY.inputNotes: a number written as text for gt, gte, lt, lte or between, {"min": 1, "max": 5} or "[1, 5]" for between, values or a JSON list written as text for in and notIn, value as the regex pattern, plain text in pattern for a text comparison (text with regex syntax still fails, use regex), "ignoreCase": true for "caseSensitive": false, "value": null on empty, notEmpty, true or false, int, float, bool or text as a cast target, from for rename, field for keep, to as the value of default, a missing replacement, JavaScript $1 in a replacement, and a filter or dedupeBy on the name a rename moved, also when the rename moved an object that holds it. The CHANGELOG.md lists the other spellings that are read. Descriptive keys such as description, comment, label or id are ignored.

Anything that could mean two things, such as an unknown key or a comma-separated string for in, fails the run before a row is read. The message names the rule, the key and the fix, and RUN_SUMMARY.inputErrors lists every rule problem at once, so one retry can fix them all. Each entry has rule, op, reason (MISSING_KEY, BAD_VALUE, UNKNOWN_KEY, UNUSED_KEY, INVALID_REGEX, FIELD_REMOVED, NO_SOURCE, DATASET_ID_ONLY_IN_PAYLOAD), key, the keys the op accepts and message; a key that does not apply is null. A run that stops on its rules after reading rows has one entry with RULE_FAILS_EVERY_ROW, EMPTY_EVERY_ROW or REGEX_TOO_SLOW.

How a run ends

The Actor run succeeds when it could read the source and apply the rules, including when nothing matched. Read RUN_SUMMARY.status:

Status	Meaning
SUCCESS	At least one row was written, and no transform failed.
NO_RESULTS	The source was read and nothing was written.
PARTIAL	Some rows failed a transform (in a source under 50 rows, possibly all of them), some rows got no model answer (llmFailed counts each code), the charge limit stopped the run, or a kept row was read but not written. rowsUnwritten counts kept rows that were not stored. transformFailureSamples lists up to 20 source indexes.
SKIPPED_CHARGE_LIMIT	The budget could not pay for one row. Nothing was read.
FAILED	The input, dataset access, or a charge could not be trusted. An input problem names the rule and the key to fix.

rowsSkipped counts rows that were not written because they are not JSON objects or had no field left after the transforms, and skipCodes counts each reason (NON_OBJECT, EMPTY_ROW). Skipped rows pay only the read price.

A run with an llm transform adds llmModel, llmRows (rows the step reached), llmSucceeded (with an answer, cache hits included), llmCacheHits, llmCalls, llmFailed, llmLimitReached, llmUsageCostUsd (the usage.cost the proxy reported; null when it reported none) and aiCharged. The codes in llmFailed and llmError are LLM_TIMEOUT, LLM_RATE_LIMITED, LLM_UPSTREAM_ERROR, LLM_TRANSPORT_ERROR, LLM_BAD_RESPONSE, LLM_MODEL_INVALID, LLM_MODEL_UNSUPPORTED (the model cannot return outputFields), LLM_REQUEST_REJECTED, LLM_AUTH_FAILED, LLM_PAYMENT_REQUIRED, LLM_TRUNCATED (the answer hit maxTokens), LLM_EMPTY, LLM_REFUSED, LLM_JSON_INVALID, LLM_FIELD_INVALID, LLM_PROMPT_TOO_LONG, LLM_LIMIT and LLM_COST_LIMIT. nextAction names the fix for the most common one. With an llm step the Actor writes and charges every 100 rows, and it stops reading new rows about two minutes before the run timeout so the calls in flight can finish.

stoppedReason is COMPLETE, MAX_ITEMS, BUDGET, TIME_LIMIT, DEDUPE_LIMIT or FAILED. TIME_LIMIT means the run stopped 30 seconds before its timeout with the rows it had written and a PARTIAL summary, instead of being cut off; raise the run timeout to process the rest. If the platform restarts the run in the middle of a batch, the Actor checks the charges and rows that batch left behind before it goes on, so rows are not charged twice. When that check cannot tell, the run fails with BILLING_UNCERTAIN instead of guessing. Start a new run and check the rows already written. transformFailureCodes names the rule, for example transforms[0]:CAST_FAILED. REGEX_TIMEOUT means that pattern took longer than 50 milliseconds on that row, so the row was dropped and the run continued. REGEX_FAILED means the replacement does not fit the pattern.

Related automa-flow tools

• Text from a cleaned list of PDF URLs: PDF to Text Extractor reads datasetId and a URL field. Clean and dedupe the URL Dataset here, then pass this run's Dataset to it.
• Process a cleaned list of media URLs: Bulk FFmpeg Video Converter converts or cuts the media at the URLs of a Dataset. Clean and dedupe that Dataset here first.

Each Actor runs and charges separately. This Actor does not call them.

Price

Event	Price	Charged when
processed-row	$0.00002	A row was read and a decision was made.
kept-row	$0.00098	A transformed row was written.
ai-row	$0.001	A row was written with a usable answer from the llm transform. Cache hits count; rows with llmError do not.
Actor start	$0.00005	Once per run.

1,000 rows, all kept: $1 plus the start. 1,000 rows, none kept: $0.02 plus the start. Rows left unread because of maxItems or maxTotalChargeUsd are not charged. A run without an llm transform never charges ai-row.

The model itself is not in these prices: apify/openrouter bills its usage to the Apify account that runs this Actor, at the model's rate (the FREE plan pays 10x). maxLlmRows and maxLlmCostUsd bound it; maxTotalChargeUsd bounds this Actor's own events.

Limits

Rows of any size work at the default memory: the Actor reads the source in pages sized from the rows it has seen. For heavy rows, such as crawler output with html or markdown, put keep or drop first: those fields are then never downloaded, which makes the run faster and cheaper. This works for top-level names; a leading keep or drop that names a dotted path downloads whole rows.

This does not run JavaScript or Python that you supply. It does not join or pivot. Only the llm transform calls a model, one prompt per row, with no files, images, web browsing or tools. dedupeBy removes duplicate rows inside one run. Inline data stops at 1,000 rows. A text value longer than 100,000 characters that trim, lowercase, uppercase, a string cast, regexReplace, regexExtract, template or the regex filter reads drops that row with FIELD_TOO_LARGE. A regular expression that runs longer than 50 milliseconds drops that row with REGEX_TIMEOUT. After 200 such rows the run stops, because the pattern needs to be simpler. outputDatasetName is optional. Leave it empty to use this run's default Dataset, which is empty at the start of every run. A named dataset is appended to, so a schedule writes another copy of the kept rows each time. It cannot be the same Dataset you are reading.

For an agent

Ask it to filter one Dataset or a small JSON array, with explicit maxItems and maxTotalChargeUsd. It should build each rule from the op list in the input schema and read RUN_SUMMARY before fetching rows. If a run fails on its input, RUN_SUMMARY.inputErrors lists every rule to change, with the keys each op accepts. SUCCESS means the kept rows are in the Dataset linked from the run output. NO_RESULTS means the source was read and the filters excluded everything. Do not treat a failed dataset read as an empty export. For an llm step, also set maxLlmRows and maxLlmCostUsd, read llmFailed and llmUsageCostUsd, and treat a row with llmError set as unanswered.

What changed is in the changelog.

Direct MCP tool: https://mcp.apify.com?tools=automa-flow/dataset-filter-transform

Example ask: "From dataset DATASET_ID, keep rows whose status is open, trim company, lowercase email, rename company to companyName, and stop at 100 kept rows."