Dataset Filter & Transform: Filter, Pick, Rename, Flatten

Dataset Filter & Transform cleans up any Apify dataset, or CSV, Excel and JSON files, without writing code. Keep only the rows that match your conditions, remove duplicates, choose which columns to keep, rename them, flatten nested fields into plain columns, split a list field into one row per element, tag every row with a source, trim spaces, sort by one or more fields, skip and limit, and save the result as a CSV, Excel or JSON file too. It sits between a scraper (or a spreadsheet) and wherever the data goes next (Google Sheets, a CRM import, a file, a Slack alert), and reads one source or several, in order.

• Read datasets, files or pasted rows: one or more datasets, public CSV, TSV, JSON, JSON Lines or Excel (XLSX/XLS) file links (Google Sheets, Google Drive and Dropbox share links work), or a JSON list of rows pasted into the input.
• Filter rows with conditions such as equals, contains, regex, greaterThan, in or isNotEmpty. Combine them with AND ("all") or OR ("any").
• Nested fields and lists: dots reach into objects (address.city), and a condition on a list matches when any element matches (categories contains "bakery").
• Shape the columns: keep only some fields (in your order), remove others, rename them, flatten nested objects into a.b.c columns, and join lists into text.
• Split a list into rows: one row per email, phone number or product variant.
• Tag and clean: add fixed fields such as source, trim spaces, turn empty values into null, lowercase emails.
• Remove duplicates by one or more fields (or whole rows), ignoring letter case and extra spaces if you like, keeping the first or the last copy.
• Sort and limit: sort by several fields, each ascending or descending, skip the first rows and cap the output.
• Export files: also save the final rows as CSV, JSON Lines, JSON or Excel in the run's key-value store, with download links in the summary. Free: you pay only for the rows.

Try it in a minute: the input is already filled in with a small example. Click Try for free, then Start, and you'll see real results before you use your own data. Apify's free plan needs no credit card and gives you $5 to spend in Apify Store every month.

Why use Dataset Filter & Transform?

• No code, and no code execution. Conditions are plain data ({"field", "op", "value"}) checked by fixed rules. The Actor never runs JavaScript or Python that you supply, so a saved task or an integration can't be turned into a script runner.
• Rows you filter out are free. You pay only for rows written to the output. Removed duplicates, files that can't be read and the exported files are free too.
• Streams large datasets and files. Items are read and written in batches of 500, so memory stays flat however big the input is. Removing duplicates keeps only a small fingerprint per unique key (about 100 bytes), and "keep the last copy" parks the rows on the run's disk, not in memory. Sorting is the one step that holds rows in memory (see the limits below).
• Resumes cleanly. If the platform migrates or restarts the run, it continues where it stopped instead of writing duplicates.
• Safe file downloads. File links are fetched with the same guarded downloader as CSV & Excel to Dataset: private, loopback and cloud-metadata addresses are refused on every redirect, the size is capped while downloading, zip bombs are refused, and Excel macros and formulas never run.
• Limited permissions. It can only read the datasets you select and write its own output.

Use cases

• Keep only leads with a website: after a Google Maps or directory scraper, keep the places where website is not empty and rating is at least 4.
• Drop columns before a Google Sheets export: keep only title, phone, website and address.city, in that order, so the sheet has exactly those columns.
• Rename columns to match a CRM import: title → Company Name, phone → Phone Number, address.city → City, so the CSV maps without manual work.
• Split a list field into rows: a scraper returns emails: ["a@x.com", "b@x.com"]; split it so each email gets its own row, ready for an email tool.
• Tag rows with a source: add {"source": "google-maps", "campaign": "october"} to every row before merging results from several scrapers.
• Merge scraper runs and a spreadsheet without duplicates: read two datasets and a Google Sheet, remove duplicates by email (ignoring case), and keep the last copy so the newest source wins.
• Top 100 by rating, as an Excel file: sort by rating descending, then reviewsCount descending, set Max results to 100 and export XLSX for a client.
• Clean a CSV export in one step: paste a CSV link, keep the rows where status is active, rename the columns, and get a cleaned CSV back.

How to use Dataset Filter & Transform

1. Pick one or more Datasets (they are read in the order given). You can also type dataset IDs in Dataset IDs (typed), add public File URLs (read after the datasets), or, for a quick test, paste a JSON list of objects into Items (used only when no dataset and no file URL is given).
2. Add Filters (optional). Leave them empty to keep every row.
3. Choose how to shape the rows: split a list into rows, keep or remove fields, rename, flatten, join lists, add fields.
4. Optionally clean up values, remove duplicates, sort, set Max results, and choose Export files.
5. Click Start. The rows are in the run's dataset; the counts (and any file links) are in the key-value store OUTPUT.

After every scraper run (integration)

1. Open your scraper's saved task, go to the Integrations tab and click Add integration.
2. Search for Dataset Filter & Transform, choose to start it when the run succeeded, and set the input. Apify fills in {{resource.defaultDatasetId}} with that run's dataset:

{
  "datasetIds": ["{{resource.defaultDatasetId}}"],
  "filters": [{ "field": "website", "op": "isNotEmpty" }],
  "pickFields": ["title", "website", "phone", "address.city"],
  "renameFields": { "title": "Company Name", "address.city": "City" }
}

3. Save, and test the integration on a past run.

To send the cleaned rows on, add a second integration on this Actor's runs, for example Google Sheets Sync, with {{resource.defaultDatasetId}}.

Filters

Each condition is an object with a field, an op and (for most ops) a value:

[
  { "field": "website", "op": "isNotEmpty" },
  { "field": "rating", "op": "greaterOrEqual", "value": 4 },
  { "field": "address.country", "op": "in", "value": "US, CA" },
  { "field": "title", "op": "regex", "value": "^(cafe|coffee)\\b" }
]

Op	Keeps a row when the field…
equals / notEquals	equals the value (or doesn't). "value": null matches a missing or null field.
contains / notContains	contains the text (or doesn't).
startsWith / endsWith	starts or ends with the text.
regex	matches the regular expression anywhere in the text (Python syntax).
greaterThan, lessThan, greaterOrEqual, lessOrEqual	compares to the value.
in / notIn	equals one of the values (a list, or text like "US, CA, MX").
exists / notExists	is present and not null (or isn't). No value needed.
isEmpty / isNotEmpty	is missing, null, blank text, an empty list or an empty object (or isn't). No value needed.

How values are compared:

• Text ignores letter case unless you turn on Case-sensitive text matching (or add "caseSensitive": true to one condition).
• Numbers compare as numbers when both sides are numbers, including numeric text such as "4.5". So 9 lessThan 10 is true, even if the 9 was scraped as text. When the value is a number, a field that isn't one (like "N/A") never matches a comparison. Whole numbers are compared exactly, so long IDs never match a neighbouring ID.
• Equals with two texts compares the text, so a zip code "02134" doesn't equal "2134". Write the value as a number ("value": 10) to match 10, "10" and "10.0" alike.
• Dates compare as dates when both sides are ISO 8601 (2026-03-01 or 2026-03-01T10:00:00Z). A date without a time means midnight UTC, and a time without an offset is read as UTC.
• Lists: a condition matches if any element matches (emails endsWith "@gmail.com"). Dots cross lists too: offers.price lessThan 20 checks every offer's price. The negative ops are the opposite of the positive ones: tags notContains "x" keeps rows where no tag contains "x". A missing field counts as "not equal" and "not contains".
• Filter mode "all" keeps rows that match every condition (AND); "any" keeps rows that match at least one (OR).
• Shorthands work too: ==, !=, >, <, >=, <=, and operator instead of op.

An invalid regular expression or an unknown op stops the run before anything is written, with a message saying which condition to fix.

Order of steps

1. Read the sources, in this order: the datasets (picked, then typed), then the file URLs. Items are read only when there is no dataset and no file.

Every item then goes through the same steps, in this order:

2. Filter. Conditions are checked on the original item, as the scraper (or file) has it.
3. Split a list field into rows (unwind).
4. Keep only these fields, then Remove these fields.
5. Rename fields (in the order you list them).
6. Flatten nested objects (and join lists into text).
7. Add fields.
8. Clean up: trim spaces, then empty values to null, then lowercase.

Then, on the output rows:

9. Remove duplicates.
10. Sort.
11. Skip first rows, then Max results.
12. Write the rows to the dataset (each written row is charged as it is written).
13. Export files from exactly the rows written.

Because of this order, later steps use the names produced by earlier ones: rename a picked nested field by its dotted name ("address.city": "City"), and give Lowercase these fields, Remove duplicates by and the sort fields the output name (after renaming, and contact.email after flattening). Duplicates removed before the limit don't count toward Max results.

Files as input

File URLs takes public links, one per line: CSV, TSV and other delimited text, JSON (a list of objects, or an object holding one such list, like {"data": [...]}), JSON Lines, and Excel XLSX, XLSM or XLS. The format is detected from the content, so links without an extension work.

• Google Sheets (shared as "Anyone with the link", or published to the web), Google Drive file links and Dropbox share links are turned into download links automatically. OneDrive/SharePoint links and folder links aren't supported.
• Excel sheets: the first sheet by default, or all sheets one after the other. The first row holds the column names. Excel and JSON values keep their types (numbers, booleans; dates as ISO 8601 text).
• CSV values are text unless you turn on Convert numbers and true/false in CSV. Filters and sorting treat numeric text as numbers either way. Delimiter (comma, semicolon, tab, pipe) and encoding (UTF-8, UTF-16, Windows-1252) are detected.
• Add file, sheet and row number adds _file, _sheet and _row to each row read from a file.
• A file that can't be used (not public, not found, a web page instead of a file, damaged) doesn't stop the run and adds nothing to the output: the reason is logged and listed under perFile in OUTPUT, and the status message says how many files failed. Rows read before a damaged part of a file are kept. If no source at all could be read, the run fails with the reason. A link that can never work (not http/https, OneDrive, a folder) stops the run before anything is read.

For more file options (a header row other than the first, a fixed delimiter or encoding, a JSON path, named sheets), convert the file with CSV & Excel to Dataset first and pick its dataset here.

Removing duplicates

Set Remove duplicates by to the output fields that make two rows "the same", for example email, or name and website together (every listed field must match). Use * to compare whole rows.

• Which duplicate to keep: first keeps the first copy and streams. last keeps the last copy's values, at the position where the key first appeared (the same rule as Merge & Deduplicate Datasets); list older sources first so the newest one wins. With last, rows are written only after every source is read.
• Ignore letter case and Ignore extra spaces (leading, trailing and repeated spaces) apply to text in the key.
• A missing field and null are the same key. A number and the same digits as text (1 and "1") are different keys.
• Duplicates are counted in duplicatesRemoved and are free.

This removes duplicates within one run (across all its datasets and files). To keep only items not seen in earlier runs (for a monitor or a daily integration), use Merge & Deduplicate Datasets, which remembers keys between runs.

Sorting

Sort by field (with Sort descending) sorts by one output field. More sort fields adds further keys, one per line, each with its own direction: rating desc, then name. Each key breaks ties of the keys before it; you can also use More sort fields alone.

• Numbers sort as numbers, also when written as text (9 before "10"). ISO 8601 dates sort by time. Other text sorts alphabetically, ignoring case. Numbers come before dates, dates before text.
• Empty values (missing, null, blank) go last for that key, in both directions. Ties keep their input order (the sort is stable).
• From the API, a sort key can also be an object: {"field": "price", "order": "desc"}.

Exporting files

Choose Export files (CSV, JSON Lines, JSON, Excel XLSX) to also save the final rows as files in the run's default key-value store, named File name plus the extension (rows.csv, rows.xlsx…). The files hold exactly the rows written to the dataset, in the same order (also after a restart, and when a cost limit stopped the run early). Their links are listed under exports in OUTPUT and in the status message: they are the records' Apify API URLs, which include an access signature when your account's storages aren't public.

• CSV and Excel get one column per field seen in any row, in first-seen order; a row without a field has an empty cell. Nested objects and lists become JSON text (turn on Flatten nested fields to get address.city columns instead). CSV is UTF-8 (no byte-order mark), with true/false for booleans.
• Excel keeps numbers and booleans typed. Text starting with = stays text (it never becomes a formula). Integers longer than 15 digits are written as text so Excel doesn't round them.
• Exporting is free: you pay only for the rows, once.

Input example

{
  "datasetIds": ["aBcDeFgHiJkLmNoPq"],
  "filters": [
    { "field": "website", "op": "isNotEmpty" },
    { "field": "rating", "op": "greaterOrEqual", "value": 4 }
  ],
  "filterMode": "all",
  "unwindField": "emails",
  "pickFields": ["title", "emails", "website", "address.city", "categories"],
  "renameFields": { "title": "Company Name", "emails": "Email", "address.city": "City" },
  "joinArrays": true,
  "addFields": { "source": "google-maps" },
  "trimStrings": true,
  "lowercaseFields": ["Email"],
  "sortBy": "Company Name",
  "maxResults": 1000
}

Files, duplicates, several sort keys and an Excel export:

{
  "datasetIds": ["aBcDeFgHiJkLmNoPq"],
  "fileUrls": ["https://docs.google.com/spreadsheets/d/1AbC.../edit#gid=0", "https://example.com/leads-march.csv"],
  "filters": [{ "field": "email", "op": "isNotEmpty" }],
  "dedupeBy": ["email"],
  "dedupeIgnoreCase": true,
  "dedupeTrim": true,
  "dedupeKeep": "last",
  "sortFields": ["rating desc", "name"],
  "maxResults": 500,
  "exportFormats": ["xlsx", "csv"],
  "exportFileName": "top-leads"
}

Field	Notes
Datasets / Dataset IDs (typed)	One or more datasets, read in order (picked ones first). A dataset listed twice is read once.
Items	A JSON list of objects (pasted rows), used only when no dataset and no file URL is given (for quick tests or calls from code).
File URLs	Public CSV, TSV, JSON, JSON Lines or Excel links, read after the datasets. A link listed twice is read once.
Excel sheets / Convert numbers and true/false in CSV / Add file, sheet and row number	File reading options (see "Files as input").
Filters	A JSON list of conditions (see above). Empty = keep every row.
Filter mode	all (AND) or any (OR).
Case-sensitive text matching	Off by default.
Split a list field into rows	One row per element; other fields are copied. A missing or non-list field leaves the row as it is; an empty list keeps one row with the field set to null.
Keep only these fields	In this order. A missing field becomes null, so every row has the same columns. Dotted names stay as the column name (address.city).
Remove these fields	Dots reach nested fields; images.thumbnail removes it from every image.
Rename fields	{"old": "New"}, applied in order. Renamed top-level columns keep their position; a renamed nested field becomes a new column at the end. Missing fields are skipped.
Flatten nested fields	{"address": {"city": "Oslo"}} → {"address.city": "Oslo"}. Lists stay lists.
Join lists into text / List separator	["a", "b"] → "a, b". Lists of objects are kept. Works with or without flattening.
Add fields	Fixed values added to every row; they overwrite a field with the same name.
Trim spaces / Empty values to null / Lowercase these fields	Value cleanups.
Remove duplicates by / Which duplicate to keep / Ignore letter case / Ignore extra spaces	See "Removing duplicates". Empty = keep duplicates.
Sort by field / Sort descending / More sort fields	Numbers, dates and text sort naturally; empty values go last in both directions; ties keep their input order.
Max results / Skip first rows	Applied to output rows, after removing duplicates and sorting.
Export files / File name	CSV, JSON Lines, JSON and/or XLSX files of the output rows in the key-value store.

Output example

With the input above, a scraped place like

{ "title": " Blue Door Cafe ", "website": "https://bluedoor.example", "rating": 4.6,
  "emails": ["Hello@BlueDoor.example", "jobs@bluedoor.example"],
  "address": { "street": "1 Main St", "city": "Oslo" }, "categories": ["Cafe", "Bakery"] }

becomes two rows:

[
  { "Company Name": "Blue Door Cafe", "Email": "hello@bluedoor.example", "website": "https://bluedoor.example",
    "City": "Oslo", "categories": "Cafe, Bakery", "source": "google-maps" },
  { "Company Name": "Blue Door Cafe", "Email": "jobs@bluedoor.example", "website": "https://bluedoor.example",
    "City": "Oslo", "categories": "Cafe, Bakery", "source": "google-maps" }
]

The key-value store OUTPUT holds the counts:

{
  "itemsRead": 1250,
  "itemsMatched": 410,
  "itemsOutput": 655,
  "rowsOutputThisRun": 655,
  "resumedFrom": 0,
  "itemsWithErrors": 0,
  "errors": [],
  "perDataset": [
    { "datasetId": "aBcDeFgHiJkLmNoPq", "itemsRead": 1250, "itemsMatched": 410, "itemsOutput": 655 }
  ],
  "sortedBy": "Company Name",
  "stoppedBy": null
}

itemsMatched counts input items that passed the filters; itemsOutput counts rows written (more than matched when a list is split into rows). stoppedBy is "max results" or "max cost" when the run stopped early.

The new options add their own fields, only when you use them. For example, with file URLs, duplicates removed by email, two sort keys and a CSV export:

{
  "itemsRead": 2400,
  "itemsMatched": 2100,
  "itemsOutput": 1730,
  "perDataset": [],
  "perFile": [
    { "url": "https://example.com/leads-march.csv", "format": "csv", "fileSizeBytes": 412000, "itemsRead": 1200,
      "itemsMatched": 1050, "itemsOutput": 880, "error": null },
    { "url": "https://example.com/old-export.xlsx", "format": null, "itemsRead": 0, "itemsMatched": 0,
      "itemsOutput": 0, "error": "HTTP 404 Not Found" }
  ],
  "filesRead": 1,
  "filesFailed": 1,
  "sortedBy": "rating",
  "sortKeys": [{ "field": "rating", "order": "desc" }, { "field": "name", "order": "asc" }],
  "dedupeBy": ["email"],
  "dedupeKeep": "first",
  "duplicatesRemoved": 370,
  "exports": {
    "csv": { "key": "rows.csv", "bytes": 251904, "url": "https://api.apify.com/v2/key-value-stores/<storeId>/records/rows.csv" }
  },
  "stoppedBy": null
}

(Other counts such as rowsOutputThisRun, itemsWithErrors and errors are always there and left out above.) When a run that sorts or keeps the last duplicate is stopped by its cost limit, rowsNotOutput says how many more rows were ready but not written.

How much does it cost to filter and transform a dataset?

Dataset Filter & Transform uses pay-per-event pricing: $0.001 per output row, plus $0.005 per run start. Apify platform usage is included. Items that are read but filtered out, removed duplicates, rows skipped with Skip first rows, files that can't be read, and the exported files are free: a row is paid once, when it is written to the dataset, however many file formats you export.

Apify Store discounts: the price above is for Apify's Free plan. Paid plans pay less per 1,000 output rows: $0.87 on Starter, $0.70 on Scale, $0.55 on Business. The "from" price at the top of the Store page is the Business price.

Worked examples:

Examples use the Free-plan rate. Where an example doesn't add it, each run also has its run-start charge, counted per GB of Actor memory (minimum one; one at the default memory); Apify Proxy (where used) is billed separately.

• A Google Maps scrape of 10,000 places, of which 3,200 have a website: 3,200 × $0.001 + $0.005 = $3.205.
• A daily integration that keeps 150 new leads per run: $0.155 per run, about $4.65 per month.

If you set a maximum cost per run, the Actor stops cleanly at that limit and writes only the rows you paid for (and the exported files contain exactly those rows). When sorting or keeping the last duplicate, the rows are written at the end of the run, in the final order, so a cost limit keeps the first rows of that order.

FAQ

My filter matches nothing. Why?

Check the field name against a row of your dataset (names are case-sensitive, values are not by default). For nested fields use dots: address.city, not city. If the value is a number, text like "N/A" or "$12" in the field never matches a comparison.

Can I write my own JavaScript or Python condition?

No, on purpose. The Actor doesn't run user code. Most needs are covered by combining conditions, regex and Filter mode.

"Couldn't open dataset …"

Pick the dataset with the dataset picker, or pass a valid dataset ID. In an integration, use ["{{resource.defaultDatasetId}}"] (a list). The placeholder is filled in only when an integration starts the run; started by hand, the Actor asks you to pick a dataset.

Why is a column null in some rows?

Keep only these fields adds every listed field to every row, as null when the item doesn't have it, so spreadsheet columns line up.

Should I remove duplicates here or with Merge & Deduplicate Datasets?

Here, when you want duplicates gone within one run (for example across today's datasets and a spreadsheet), together with filtering and reshaping. Use Merge & Deduplicate Datasets when you need only new items compared with earlier runs: it remembers the keys it has seen between runs. You can chain them: Merge & Dedup for "only new", then this Actor to filter and shape.

Why are my CSV numbers text?

CSV files have no types, so values are read as text (ZIP codes like 00123 keep their zeros). Filters and sorting still compare numeric text as numbers. Turn on Convert numbers and true/false in CSV to store them as numbers.

Where are the exported files?

In the run's default key-value store (the Storage tab of the run), with links under exports in OUTPUT. Like the run's dataset, they are kept for your plan's data retention period. To deliver them somewhere else, use Dataset Delivery on this run's dataset.

Limitations

• Sorting holds rows in memory, up to 200,000 output rows. With more, the run stops before writing anything and says so: add filters or set Max results (with a limit, any number of rows can be read, because only the best "skip + limit" rows are kept in memory; skip + limit can be at most 200,000). Sorting many large rows may need 1 GB of memory instead of the default 256 MB.
• Removing duplicates keeps one fingerprint per unique key in memory: about 1 million unique keys at the default 256 MB, and roughly 10 million ("first") or 7 million ("last") more per extra GB. Past that, the run stops with a message (rows already written are unique); give it more memory. "Keep the last copy" also stores the rows temporarily on the run's local disk and writes them only at the end. Fingerprints are 128-bit hashes, so two different keys being treated as one is practically impossible.
• Files: up to 100 file URLs per run, 100 MB each, public links only (no sign-in). The first row of a sheet or CSV holds the column names. If a run is restarted by the platform, the files are downloaded again; if a file changed in the meantime, the output can mix both versions.
• Exported files are built on the run's disk and uploaded at the end. One file can be up to a quarter of the run's memory (64 MB at the default 256 MB; up to 500 MB with 2 GB or more); a larger one is skipped with a note in OUTPUT (the rows are still in the dataset, which Apify can export as CSV, JSON or Excel too). Excel files are limited to 1,048,575 rows and 16,384 columns.
• Regular expressions use Python syntax and are limited to 1,000 characters. A search that takes over 1 second on one value (usually a pattern with nested repeats such as (a+)+) stops the run with a message saying which condition to simplify; rows written before that are kept.
• Split a list field into rows works on one field per run.
• Flattening leaves lists of objects as lists (they appear as JSON in a CSV). Join lists of plain values with Join lists into text.
• Items that aren't JSON objects are skipped. If an item can't be processed, it's skipped and counted in itemsWithErrors, with the first few reasons in OUTPUT.

Related Actors

Dataset Filter & Transform is part of a small suite of Actors that work together after any scraper:

Actor	Use it to
HTTP Request & API Calls	Pull data from any REST API into a dataset (with pagination), then reshape it here.
Dataset Join & VLOOKUP	Add columns from another dataset or file by a shared key (email, SKU, URL), like VLOOKUP.
CSV & Excel to Dataset	Turn CSV, Excel or Google Sheets files into a dataset with more file options (header row, delimiter, encoding, JSON path, named sheets); this Actor can also read simple file links directly.
Merge & Deduplicate Datasets	Keep only items not seen in earlier runs (it remembers keys between runs), or merge datasets cheaply without other transforms.
Google Sheets Sync	Append or upsert the cleaned rows into a Google Sheet.
Dataset Delivery	Deliver the rows as a CSV, JSON or Excel file to SFTP, FTP, Dropbox or Google Drive.
Send Email Report & Slack, Discord, Teams, Telegram Alerts	Get told about the rows by email, Slack, Discord, Teams or Telegram.
Phone Number Validator & Formatter	Check and format the phone numbers in your filtered leads (E.164, mobile or landline).
Domain WHOIS & DNS Lookup	Add domain age, email provider, SPF and DMARC for each lead's website.
RSS & Atom Feed Reader	Turn RSS, Atom and JSON feeds into a dataset to filter.
Bulk URL Status Checker	Check the links in a dataset for 404s, redirects and SSL problems.
PDF & Image to Text	Turn PDFs and scans into text, Markdown or tables, then filter and reshape the rows here.
Broken Link Checker	Crawl a website for broken links, then filter the results down to the ones you care about.