Dataset Filter & Transform: filter, dedupe, clean and export Apify datasets, CSV, Excel and JSON (no code)

Dataset Filter & Transform is a no-code data cleaning tool for scraper output and spreadsheets. It filters rows with simple rules, removes duplicates, renames, selects or drops fields, flattens nested JSON, trims and normalizes text, converts types, sorts and limits. It can also return only the items that are new since the last run.

Feed it one or more Apify dataset IDs, CSV / TSV / XLSX / XLS / ODS / JSON / JSON Lines file URLs, or pasted JSON. You get clean rows in a dataset, plus an optional CSV, Excel (XLSX), JSON or JSONL export file, which can be split into one file (or sheet) per value.

It is built to chain after any scraper, such as Google Maps, Amazon, LinkedIn jobs, e-commerce or real estate, through an Apify integration or webhook. You pay $0.20 per 1,000 delivered rows, which is 10× cheaper than the leading dataset filter actor.

What it does

• Filter rules with AND / OR logic and nested groups:
  • comparisons: equals, notEquals, contains, notContains, startsWith, endsWith, regex, notRegex
  • ranges and lists: gt, gte, lt, lte, between, notBetween, in, notIn
  • presence: exists, notExists, empty, notEmpty, isTrue, isFalse, lengthGt, lengthLt
  • dates: before, after. These also accept relative dates like now-7d, 30 days ago and today.
  • Numbers stored as text ("$1,299.00", "15%") compare as numbers.
  • Text matching ignores case unless you set "caseSensitive": true.
• Deduplicate by one or more fields, or by the whole row. Case and surrounding spaces are ignored by default, and dedupe works across all input sources.
• Only new items since last run: the actor remembers delivered keys in a named key-value store. This is ideal for scheduled scrapers where you only want new listings, jobs, products or reviews.
• Field mapping:
  • keep only some fields, in your order (dot paths like address.city work)
  • remove fields or rename them
  • remove empty fields
  • add computed fields from templates: "{title} ({price} {currency})"
• Clean text: trim, collapse whitespace, strip HTML tags, turn empty strings into null, change case (lower / upper / title / capitalize / slug).
• Convert types: number, integer, boolean, date, datetime, Unix timestamp, array, JSON. Values that cannot be converted become null, keep their original value, or drop the row.
• Flatten nested objects: {"address": {"city": "Paris"}} becomes {"address.city": "Paris"}. Arrays can be kept, joined or expanded into indexed columns.
• Sort by several fields (asc / desc). Numbers and dates sort naturally, and empty values go last. Offset and limit let you take the top N rows.
• Export:
  • CSV (UTF-8 with BOM, opens cleanly in Excel)
  • styled XLSX (bold header, frozen first row, autofilter)
  • JSON or JSON Lines
  • Split by field: one file per value, or one sheet per value in a single XLSX.

Use cases

• Clean a Google Maps / Yelp scrape: keep rating ≥ 4.5 with a phone number, dedupe by placeId, export to Excel for sales.
• Daily job or real-estate monitoring: get only listings that are new since yesterday, then send them to Sheets, Slack or a CRM.
• Merge the datasets of several scraper runs, remove duplicate URLs, and keep only the columns your database needs.
• Prepare data for AI agents and RAG: drop noisy fields, strip HTML, flatten JSON, and cap the number of rows.
• Filter a big CSV or Excel export (e.g. a product feed) without opening Excel, using rules such as "price between 10 and 50, brand in [A, B], title not matching /refurbished/".

Input example

{
  "datasetIds": ["{{resource.defaultDatasetId}}"],
  "fileUrls": ["https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv"],
  "filters": [
    {"field": "species", "op": "in", "value": ["Adelie", "Gentoo"]},
    {"field": "body_mass_g", "op": "between", "value": [3500, 5000]},
    {"any": [
      {"field": "island", "op": "equals", "value": "Biscoe"},
      {"field": "bill_length_mm", "op": "gt", "value": 45}
    ]},
    {"field": "sex", "op": "notEmpty"}
  ],
  "dedupeFields": ["species", "island", "body_mass_g"],
  "textCase": {"sex": "lower"},
  "castFields": {"body_mass_g": "number"},
  "sortBy": ["body_mass_g desc"],
  "limit": 100,
  "renameFields": {"body_mass_g": "bodyMassGrams"},
  "exportFormat": "xlsx",
  "splitByField": "species"
}

The rules are processed in this fixed order:

1. read
2. flatten
3. clean text
4. change case
5. convert types
6. add fields
7. filter
8. dedupe
9. only-new
10. sort
11. offset / limit
12. select / drop / rename / remove empty
13. output

Filters, dedupe and sort use the original field names, because renaming happens last.

Output example

Dataset item (from the 1-click example):

{
  "species": "Gentoo",
  "island": "Biscoe",
  "bill_length_mm": 45.1,
  "bill_depth_mm": 14.5,
  "flipper_length_mm": 215,
  "bodyMassGrams": 5000,
  "sex": "female"
}

The OUTPUT record in the key-value store holds the run summary:

{
  "rowsRead": 344, "rowsKept": 153, "rowsDelivered": 153, "filteredOut": 191, "duplicatesRemoved": 0,
  "alreadySeenSkipped": 0, "castErrors": 0, "stoppedAtMaxCharge": false,
  "sources": [{"source": "https://raw.githubusercontent.com/.../penguins.csv", "status": "ok", "rowsRead": 344, "fileType": "csv"}],
  "files": [{"key": "filtered-data.xlsx", "rows": 153, "url": "https://api.apify.com/v2/key-value-stores/.../records/filtered-data.xlsx"}],
  "charged": {"rows-read-1k": 1, "row-output": 153, "file-exported": 1}
}

Pricing (pay per event)

Event	Price
Row delivered (after filtering / dedupe)	$0.0002 ($0.20 per 1,000)
1,000 source rows scanned	$0.002
Export file (CSV / XLSX / JSON / JSONL)	$0.003
Actor start	$0.00005

Some examples of what a run costs:

• The 1-click example costs about $0.03.
• 100,000 scraped rows → 20,000 kept + CSV costs $4.20.
• A scheduled "only new" run over 100,000 rows that finds 500 new ones costs $0.30.

Filtered-out rows, duplicates, failed sources and invalid input are never charged as rows. Set a maximum cost per run and the actor stops cleanly at that budget. Delivered rows are always complete, and the export file still gets written.

Actor (Apify Store, Oct 2026)	Price per kept row	Export file
This actor	$0.0002	$0.003
nerolabs/dataset-filter-transform	$0.002	$0.01
humble-echidna dataset tools	$0.001	–
automa-flow	$0.00098	–
lukaskrivka merge-dedup (legacy)	free, but needs JavaScript functions	–

Use with AI agents / Apify MCP

The actor works well as a tool for AI agents through the Apify MCP server. An agent can:

1. run a scraper,
2. call this actor with datasetIds: [<that run's dataset>] and a JSON list of rules,
3. read back only the compact, relevant rows.

Pasting a few items directly into items is handy for quick agent calls. Rules are plain JSON objects, so an LLM can write them reliably. Invalid rules fail fast with a clear message and are not charged.

FAQ

How do I connect it to my scraper? Open the scraper's actor or task, go to Integrations, choose Run Actor, pick Dataset Filter & Transform and set "datasetIds": ["{{resource.defaultDatasetId}}"]. Every scraper run is then cleaned automatically.

How does "only new items" work? The actor stores a 12-byte hash of each delivered row's dedupe key (or of the whole row) in the named key-value store stateStoreName. Later runs skip those rows. Use a different store name for each pipeline. To start over, delete the store or change its name.

Can I filter on nested fields? Yes. Use dot paths such as address.city or images.0.url, with or without flattening.

Which file formats can I read? CSV/TSV (encoding and delimiter are detected automatically), XLSX, XLSM, XLS, XLSB, ODS, JSON (an array, or an object with items/data/results) and JSON Lines. Google Sheets, Google Drive, Dropbox, OneDrive and GitHub share links work.

How big can the input be? Streaming filters handle millions of rows at constant memory. Sort and dedupe keep the kept rows or keys in memory: 1 GB handles several hundred thousand sorted rows. Use 2-4 GB for more.

What happens when a URL is broken? It is listed with its error in OUTPUT, the other sources are still processed, and nothing is charged for it.

Is my data stored anywhere else? No. Rows are only written to your own run's dataset and key-value store.