Dataset Filter & Transform: Filter, Pick, Rename, Flatten

Dataset Filter & Transform cleans up any Apify dataset without writing code. Keep only the rows that match your conditions, choose which columns to keep, rename them, flatten nested fields into plain columns, split a list field into one row per element, tag every row with a source, trim spaces, sort and limit. It sits between a scraper and wherever the data goes next (Google Sheets, a CRM import, a file, a Slack alert) and works on one dataset or several, read in order.

• Filter rows with conditions such as equals, contains, regex, greaterThan, in or isNotEmpty. Combine them with AND ("all") or OR ("any").
• Nested fields and lists: dots reach into objects (address.city), and a condition on a list matches when any element matches (categories contains "bakery").
• Shape the columns: keep only some fields (in your order), remove others, rename them, flatten nested objects into a.b.c columns, and join lists into text.
• Split a list into rows: one row per email, phone number or product variant.
• Tag and clean: add fixed fields such as source, trim spaces, turn empty values into null, lowercase emails.
• Sort and limit: sort by any output field, skip the first rows and cap the output.

Why use Dataset Filter & Transform?

• No code, and no code execution. Conditions are plain data ({"field", "op", "value"}) checked by fixed rules. The Actor never runs JavaScript or Python that you supply, so a saved task or an integration can't be turned into a script runner.
• Rows you filter out are free. You pay only for rows written to the output.
• Streams large datasets. Items are read and written in batches of 500, so memory stays flat however big the dataset is (sorting is the one exception, see the limits below).
• Resumes cleanly. If the platform migrates or restarts the run, it continues where it stopped instead of writing duplicates.
• Limited permissions. It can only read the datasets you select and write its own output.

Use cases

• Keep only leads with a website: after a Google Maps or directory scraper, keep the places where website is not empty and rating is at least 4.
• Drop columns before a Google Sheets export: keep only title, phone, website and address.city, in that order, so the sheet has exactly those columns.
• Rename columns to match a CRM import: title → Company Name, phone → Phone Number, address.city → City, so the CSV maps without manual work.
• Split a list field into rows: a scraper returns emails: ["a@x.com", "b@x.com"]; split it so each email gets its own row, ready for an email tool.
• Tag rows with a source: add {"source": "google-maps", "campaign": "october"} to every row before merging results from several scrapers.

How to use Dataset Filter & Transform

1. Pick one or more Datasets (they are read in the order given). You can also type dataset IDs in Dataset IDs (typed), or, for a quick test, paste a JSON list of objects into Items (used only when no dataset is given).
2. Add Filters (optional). Leave them empty to keep every row.
3. Choose how to shape the rows: split a list into rows, keep or remove fields, rename, flatten, join lists, add fields.
4. Optionally clean up values, sort and set Max results.
5. Click Start. The rows are in the run's dataset, and the counts are in the key-value store OUTPUT.

After every scraper run (integration)

1. Open your scraper's saved task, go to the Integrations tab and click Add integration.
2. Search for Dataset Filter & Transform, choose to start it when the run succeeded, and set the input. Apify fills in {{resource.defaultDatasetId}} with that run's dataset:

{
  "datasetIds": ["{{resource.defaultDatasetId}}"],
  "filters": [{ "field": "website", "op": "isNotEmpty" }],
  "pickFields": ["title", "website", "phone", "address.city"],
  "renameFields": { "title": "Company Name", "address.city": "City" }
}

3. Save, and test the integration on a past run.

To send the cleaned rows on, add a second integration on this Actor's runs, for example Google Sheets Sync, with {{resource.defaultDatasetId}}.

Filters

Each condition is an object with a field, an op and (for most ops) a value:

[
  { "field": "website", "op": "isNotEmpty" },
  { "field": "rating", "op": "greaterOrEqual", "value": 4 },
  { "field": "address.country", "op": "in", "value": "US, CA" },
  { "field": "title", "op": "regex", "value": "^(cafe|coffee)\\b" }
]

Op	Keeps a row when the field…
equals / notEquals	equals the value (or doesn't). "value": null matches a missing or null field.
contains / notContains	contains the text (or doesn't).
startsWith / endsWith	starts or ends with the text.
regex	matches the regular expression anywhere in the text (Python syntax).
greaterThan, lessThan, greaterOrEqual, lessOrEqual	compares to the value.
in / notIn	equals one of the values (a list, or text like "US, CA, MX").
exists / notExists	is present and not null (or isn't). No value needed.
isEmpty / isNotEmpty	is missing, null, blank text, an empty list or an empty object (or isn't). No value needed.

How values are compared:

• Text ignores letter case unless you turn on Case-sensitive text matching (or add "caseSensitive": true to one condition).
• Numbers compare as numbers when both sides are numbers, including numeric text such as "4.5". So 9 lessThan 10 is true, even if the 9 was scraped as text. When the value is a number, a field that isn't one (like "N/A") never matches a comparison. Whole numbers are compared exactly, so long IDs never match a neighbouring ID.
• Equals with two texts compares the text, so a zip code "02134" doesn't equal "2134". Write the value as a number ("value": 10) to match 10, "10" and "10.0" alike.
• Dates compare as dates when both sides are ISO 8601 (2026-03-01 or 2026-03-01T10:00:00Z). A date without a time means midnight UTC, and a time without an offset is read as UTC.
• Lists: a condition matches if any element matches (emails endsWith "@gmail.com"). Dots cross lists too: offers.price lessThan 20 checks every offer's price. The negative ops are the opposite of the positive ones: tags notContains "x" keeps rows where no tag contains "x". A missing field counts as "not equal" and "not contains".
• Filter mode "all" keeps rows that match every condition (AND); "any" keeps rows that match at least one (OR).
• Shorthands work too: ==, !=, >, <, >=, <=, and operator instead of op.

An invalid regular expression or an unknown op stops the run before anything is written, with a message saying which condition to fix.

Order of steps

Every item goes through the same steps, in this order:

1. Filter. Conditions are checked on the original item, as the scraper wrote it.
2. Split a list field into rows (unwind).
3. Keep only these fields, then Remove these fields.
4. Rename fields (in the order you list them).
5. Flatten nested objects (and join lists into text).
6. Add fields.
7. Clean up: trim spaces, then empty values to null, then lowercase.

Then the output rows are sorted, the first Skip first rows are skipped, and Max results caps the rest.

Because of this order, later steps use the names produced by earlier ones: rename a picked nested field by its dotted name ("address.city": "City"), and give Lowercase these fields and Sort by field the output name (after renaming, and contact.email after flattening).

Input example

{
  "datasetIds": ["aBcDeFgHiJkLmNoPq"],
  "filters": [
    { "field": "website", "op": "isNotEmpty" },
    { "field": "rating", "op": "greaterOrEqual", "value": 4 }
  ],
  "filterMode": "all",
  "unwindField": "emails",
  "pickFields": ["title", "emails", "website", "address.city", "categories"],
  "renameFields": { "title": "Company Name", "emails": "Email", "address.city": "City" },
  "joinArrays": true,
  "addFields": { "source": "google-maps" },
  "trimStrings": true,
  "lowercaseFields": ["Email"],
  "sortBy": "Company Name",
  "maxResults": 1000
}

Field	Notes
Datasets / Dataset IDs (typed)	One or more datasets, read in order (picked ones first). A dataset listed twice is read once.
Items	A JSON list of objects, used only when no dataset is given (for quick tests or calls from code).
Filters	A JSON list of conditions (see above). Empty = keep every row.
Filter mode	all (AND) or any (OR).
Case-sensitive text matching	Off by default.
Split a list field into rows	One row per element; other fields are copied. A missing or non-list field leaves the row as it is; an empty list keeps one row with the field set to null.
Keep only these fields	In this order. A missing field becomes null, so every row has the same columns. Dotted names stay as the column name (address.city).
Remove these fields	Dots reach nested fields; images.thumbnail removes it from every image.
Rename fields	{"old": "New"}, applied in order. Renamed top-level columns keep their position; a renamed nested field becomes a new column at the end. Missing fields are skipped.
Flatten nested fields	{"address": {"city": "Oslo"}} → {"address.city": "Oslo"}. Lists stay lists.
Join lists into text / List separator	["a", "b"] → "a, b". Lists of objects are kept. Works with or without flattening.
Add fields	Fixed values added to every row; they overwrite a field with the same name.
Trim spaces / Empty values to null / Lowercase these fields	Value cleanups.
Sort by field / Sort descending	Numbers, dates and text sort naturally; empty values go last in both directions; ties keep their input order.
Max results / Skip first rows	Applied to output rows, after sorting.

Output example

With the input above, a scraped place like

{ "title": " Blue Door Cafe ", "website": "https://bluedoor.example", "rating": 4.6,
  "emails": ["Hello@BlueDoor.example", "jobs@bluedoor.example"],
  "address": { "street": "1 Main St", "city": "Oslo" }, "categories": ["Cafe", "Bakery"] }

becomes two rows:

[
  { "Company Name": "Blue Door Cafe", "Email": "hello@bluedoor.example", "website": "https://bluedoor.example",
    "City": "Oslo", "categories": "Cafe, Bakery", "source": "google-maps" },
  { "Company Name": "Blue Door Cafe", "Email": "jobs@bluedoor.example", "website": "https://bluedoor.example",
    "City": "Oslo", "categories": "Cafe, Bakery", "source": "google-maps" }
]

The key-value store OUTPUT holds the counts:

{
  "itemsRead": 1250,
  "itemsMatched": 410,
  "itemsOutput": 655,
  "rowsOutputThisRun": 655,
  "resumedFrom": 0,
  "itemsWithErrors": 0,
  "errors": [],
  "perDataset": [
    { "datasetId": "aBcDeFgHiJkLmNoPq", "itemsRead": 1250, "itemsMatched": 410, "itemsOutput": 655 }
  ],
  "sortedBy": "Company Name",
  "stoppedBy": null
}

itemsMatched counts input items that passed the filters; itemsOutput counts rows written (more than matched when a list is split into rows). stoppedBy is "max results" or "max cost" when the run stopped early.

How much does it cost to filter and transform a dataset?

Dataset Filter & Transform uses pay-per-event pricing: $0.001 per output row, plus $0.005 per run start. Apify platform usage is included. Items that are read but filtered out, and rows skipped with Skip first rows, are free.

Apify Store discounts: the price above is for Apify's Free plan. Paid plans pay less per 1,000 output rows: $0.87 on Starter, $0.70 on Scale, $0.55 on Business. The "from" price at the top of the Store page is the Business price.

Worked examples:

Examples use the Free-plan rate. Where an example doesn't add it, each run also has its run-start charge, counted per GB of Actor memory (minimum one; one at the default memory); Apify Proxy (where used) is billed separately.

• A Google Maps scrape of 10,000 places, of which 3,200 have a website: 3,200 × $0.001 + $0.005 = $3.205.
• A daily integration that keeps 150 new leads per run: $0.155 per run, about $4.65 per month.

If you set a maximum cost per run, the Actor stops cleanly at that limit and writes only the rows you paid for.

FAQ

My filter matches nothing. Why?

Check the field name against a row of your dataset (names are case-sensitive, values are not by default). For nested fields use dots: address.city, not city. If the value is a number, text like "N/A" or "$12" in the field never matches a comparison.

Can I write my own JavaScript or Python condition?

No, on purpose. The Actor doesn't run user code. Most needs are covered by combining conditions, regex and Filter mode.

"Couldn't open dataset …"

Pick the dataset with the dataset picker, or pass a valid dataset ID. In an integration, use ["{{resource.defaultDatasetId}}"] (a list). The placeholder is filled in only when an integration starts the run; started by hand, the Actor asks you to pick a dataset.

Why is a column null in some rows?

Keep only these fields adds every listed field to every row, as null when the item doesn't have it, so spreadsheet columns line up.

I need to remove duplicates too.

Run Merge & Deduplicate Datasets first (it can also keep only items not seen in earlier runs), then this Actor on its output.

Limitations

• Sorting holds rows in memory, up to 200,000 output rows. With more, the run stops before writing anything and says so: add filters or set Max results (with a limit, any number of rows can be read, because only the best "skip + limit" rows are kept in memory; skip + limit can be at most 200,000). Sorting many large rows may need 1 GB of memory instead of the default 256 MB.
• Regular expressions use Python syntax and are limited to 1,000 characters. A search that takes over 1 second on one value (usually a pattern with nested repeats such as (a+)+) stops the run with a message saying which condition to simplify; rows written before that are kept.
• Split a list field into rows works on one field per run.
• Flattening leaves lists of objects as lists (they appear as JSON in a CSV). Join lists of plain values with Join lists into text.
• Items that aren't JSON objects are skipped. If an item can't be processed, it's skipped and counted in itemsWithErrors, with the first few reasons in OUTPUT.

Related Actors

Dataset Filter & Transform is part of a small suite of Actors that work together after any scraper:

Actor	Use it to
HTTP Request & API Calls	Pull data from any REST API into a dataset (with pagination), then reshape it here.
Dataset Join & VLOOKUP	Add columns from another dataset or file by a shared key (email, SKU, URL), like VLOOKUP.
CSV & Excel to Dataset	Turn CSV, Excel or Google Sheets files into a dataset you can filter and reshape here.
Merge & Deduplicate Datasets	Merge datasets, remove duplicates, or keep only items not seen in earlier runs.
Google Sheets Sync	Append or upsert the cleaned rows into a Google Sheet.
Dataset Delivery	Deliver the rows as a CSV, JSON or Excel file to SFTP, FTP, Dropbox or Google Drive.
Send Email Report & Slack, Discord, Teams, Telegram Alerts	Get told about the rows by email, Slack, Discord, Teams or Telegram.
Phone Number Validator & Formatter	Check and format the phone numbers in your filtered leads (E.164, mobile or landline).
Domain WHOIS & DNS Lookup	Add domain age, email provider, SPF and DMARC for each lead's website.
RSS & Atom Feed Reader	Turn RSS, Atom and JSON feeds into a dataset to filter.
Bulk URL Status Checker	Check the links in a dataset for 404s, redirects and SSL problems.
PDF & Image to Text	Turn PDFs and scans into text, Markdown or tables, then filter and reshape the rows here.
Broken Link Checker	Crawl a website for broken links, then filter the results down to the ones you care about.