# Dataset Filter & Transform - Dedupe, Clean, Export CSV (`gazidev/dataset-filter-transform`) Actor

No-code filter, dedupe and transform for Apify datasets, CSV, XLSX and JSON files: rules (equals, contains, regex, ranges, dates, AND/OR), dedupe by fields, only-new-since-last-run, rename/select/drop, flatten, cast, sort, limit. Output to dataset plus CSV/XLSX/JSON export. Pay per row.

- **URL**: https://apify.com/gazidev/dataset-filter-transform.md
- **Developed by:** [Cemal Atakli](https://apify.com/gazidev) (community)
- **Stats:** 2 total users, 2 monthly users, 50.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.20 / 1,000 row delivereds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dataset Filter & Transform: filter, dedupe, clean and export Apify datasets, CSV, Excel and JSON (no code)

**Dataset Filter & Transform** is a no-code data cleaning tool for scraper output and spreadsheets. It filters rows
with simple rules, removes duplicates, renames, selects or drops fields, flattens nested JSON, trims and normalizes text,
converts types, sorts and limits. It can also return **only the items that are new since the last run**.

Feed it one or more **Apify dataset IDs**, **CSV / TSV / XLSX / XLS / ODS / JSON / JSON Lines file URLs**, or pasted
JSON. You get clean rows in a dataset, plus an optional **CSV, Excel (XLSX), JSON or JSONL export file**, which can be
split into one file (or sheet) per value.

It is built to chain after any scraper, such as Google Maps, Amazon, LinkedIn jobs, e-commerce or real estate, through
an Apify integration or webhook. You pay **$0.20 per 1,000 delivered rows**, which is 10× cheaper than the leading
dataset filter actor.

### What it does

- **Filter rules** with AND / OR logic and nested groups:
  - comparisons: `equals`, `notEquals`, `contains`, `notContains`, `startsWith`, `endsWith`, `regex`, `notRegex`
  - ranges and lists: `gt`, `gte`, `lt`, `lte`, `between`, `notBetween`, `in`, `notIn`
  - presence: `exists`, `notExists`, `empty`, `notEmpty`, `isTrue`, `isFalse`, `lengthGt`, `lengthLt`
  - dates: `before`, `after`. These also accept relative dates like `now-7d`, `30 days ago` and `today`.
  - Numbers stored as text (`"$1,299.00"`, `"15%"`) compare as numbers.
  - Text matching ignores case unless you set `"caseSensitive": true`.
- **Deduplicate** by one or more fields, or by the whole row. Case and surrounding spaces are ignored by default, and
  dedupe works across all input sources.
- **Only new items since last run**: the actor remembers delivered keys in a named key-value store. This is ideal for
  scheduled scrapers where you only want new listings, jobs, products or reviews.
- **Field mapping**:
  - keep only some fields, in your order (dot paths like `address.city` work)
  - remove fields or rename them
  - remove empty fields
  - add computed fields from templates: `"{title} ({price} {currency})"`
- **Clean text**: trim, collapse whitespace, strip HTML tags, turn empty strings into null, change case (lower /
  upper / title / capitalize / slug).
- **Convert types**: number, integer, boolean, date, datetime, Unix timestamp, array, JSON. Values that cannot be
  converted become null, keep their original value, or drop the row.
- **Flatten nested objects**: `{"address": {"city": "Paris"}}` becomes `{"address.city": "Paris"}`. Arrays can be
  kept, joined or expanded into indexed columns.
- **Sort** by several fields (asc / desc). Numbers and dates sort naturally, and empty values go last. **Offset and
  limit** let you take the top N rows.
- **Export**:
  - CSV (UTF-8 with BOM, opens cleanly in Excel)
  - styled XLSX (bold header, frozen first row, autofilter)
  - JSON or JSON Lines
  - **Split by field**: one file per value, or one sheet per value in a single XLSX.

### Use cases

- Clean a Google Maps / Yelp scrape: keep rating ≥ 4.5 with a phone number, dedupe by `placeId`, export to Excel for
  sales.
- Daily job or real-estate monitoring: get only listings that are new since yesterday, then send them to Sheets, Slack
  or a CRM.
- Merge the datasets of several scraper runs, remove duplicate URLs, and keep only the columns your database needs.
- Prepare data for AI agents and RAG: drop noisy fields, strip HTML, flatten JSON, and cap the number of rows.
- Filter a big CSV or Excel export (e.g. a product feed) without opening Excel, using rules such as "price between 10
  and 50, brand in \[A, B], title not matching /refurbished/".

### Input example

```json
{
  "datasetIds": ["{{resource.defaultDatasetId}}"],
  "fileUrls": ["https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv"],
  "filters": [
    {"field": "species", "op": "in", "value": ["Adelie", "Gentoo"]},
    {"field": "body_mass_g", "op": "between", "value": [3500, 5000]},
    {"any": [
      {"field": "island", "op": "equals", "value": "Biscoe"},
      {"field": "bill_length_mm", "op": "gt", "value": 45}
    ]},
    {"field": "sex", "op": "notEmpty"}
  ],
  "dedupeFields": ["species", "island", "body_mass_g"],
  "textCase": {"sex": "lower"},
  "castFields": {"body_mass_g": "number"},
  "sortBy": ["body_mass_g desc"],
  "limit": 100,
  "renameFields": {"body_mass_g": "bodyMassGrams"},
  "exportFormat": "xlsx",
  "splitByField": "species"
}
```

The rules are processed in this fixed order:

1. read
2. flatten
3. clean text
4. change case
5. convert types
6. add fields
7. **filter**
8. **dedupe**
9. only-new
10. sort
11. offset / limit
12. select / drop / rename / remove empty
13. output

Filters, dedupe and sort use the original field names, because renaming happens last.

### Output example

Dataset item (from the 1-click example):

```json
{
  "species": "Gentoo",
  "island": "Biscoe",
  "bill_length_mm": 45.1,
  "bill_depth_mm": 14.5,
  "flipper_length_mm": 215,
  "bodyMassGrams": 5000,
  "sex": "female"
}
```

The `OUTPUT` record in the key-value store holds the run summary:

```json
{
  "rowsRead": 344, "rowsKept": 153, "rowsDelivered": 153, "filteredOut": 191, "duplicatesRemoved": 0,
  "alreadySeenSkipped": 0, "castErrors": 0, "stoppedAtMaxCharge": false,
  "sources": [{"source": "https://raw.githubusercontent.com/.../penguins.csv", "status": "ok", "rowsRead": 344, "fileType": "csv"}],
  "files": [{"key": "filtered-data.xlsx", "rows": 153, "url": "https://api.apify.com/v2/key-value-stores/.../records/filtered-data.xlsx"}],
  "charged": {"rows-read-1k": 1, "row-output": 153, "file-exported": 1}
}
```

### Pricing (pay per event)

| Event | Price |
|---|---|
| Row delivered (after filtering / dedupe) | **$0.0002** ($0.20 per 1,000) |
| 1,000 source rows scanned | $0.002 |
| Export file (CSV / XLSX / JSON / JSONL) | $0.003 |
| Actor start | $0.00005 |

Some examples of what a run costs:

- The 1-click example costs about **$0.03**.
- 100,000 scraped rows → 20,000 kept + CSV costs **$4.20**.
- A scheduled "only new" run over 100,000 rows that finds 500 new ones costs **$0.30**.

Filtered-out rows, duplicates, failed sources and invalid input are never charged as rows. Set a *maximum cost per
run* and the actor stops cleanly at that budget. Delivered rows are always complete, and the export file still gets
written.

| Actor (Apify Store, Oct 2026) | Price per kept row | Export file |
|---|---|---|
| **This actor** | **$0.0002** | $0.003 |
| nerolabs/dataset-filter-transform | $0.002 | $0.01 |
| humble-echidna dataset tools | $0.001 | – |
| automa-flow | $0.00098 | – |
| lukaskrivka merge-dedup (legacy) | free, but needs JavaScript functions | – |

### Use with AI agents / Apify MCP

The actor works well as a tool for AI agents through the [Apify MCP server](https://mcp.apify.com). An agent can:

1. run a scraper,
2. call this actor with `datasetIds: [<that run's dataset>]` and a JSON list of rules,
3. read back only the compact, relevant rows.

Pasting a few items directly into `items` is handy for quick agent calls. Rules are plain JSON objects, so an LLM
can write them reliably. Invalid rules fail fast with a clear message and are not charged.

### FAQ

**How do I connect it to my scraper?** Open the scraper's actor or task, go to *Integrations*, choose *Run Actor*,
pick Dataset Filter & Transform and set `"datasetIds": ["{{resource.defaultDatasetId}}"]`. Every scraper run is then
cleaned automatically.

**How does "only new items" work?** The actor stores a 12-byte hash of each delivered row's dedupe key (or of the whole
row) in the named key-value store `stateStoreName`. Later runs skip those rows. Use a different store name for each
pipeline. To start over, delete the store or change its name.

**Can I filter on nested fields?** Yes. Use dot paths such as `address.city` or `images.0.url`, with or without
flattening.

**Which file formats can I read?** CSV/TSV (encoding and delimiter are detected automatically), XLSX, XLSM, XLS, XLSB,
ODS, JSON (an array, or an object with `items`/`data`/`results`) and JSON Lines. Google Sheets, Google Drive, Dropbox,
OneDrive and GitHub share links work.

**How big can the input be?** Streaming filters handle millions of rows at constant memory. Sort and dedupe keep the
kept rows or keys in memory: 1 GB handles several hundred thousand sorted rows. Use 2-4 GB for more.

**What happens when a URL is broken?** It is listed with its error in `OUTPUT`, the other sources are still processed,
and nothing is charged for it.

**Is my data stored anywhere else?** No. Rows are only written to your own run's dataset and key-value store.

# Actor input Schema

## `datasetIds` (type: `array`):

One or more Apify dataset IDs (or 'username~dataset-name'), e.g. the output of a scraper run. Several datasets are processed one after another and merged into one output. In an integration / webhook use {{resource.defaultDatasetId}}.

## `fileUrls` (type: `array`):

Public URLs of CSV, TSV, XLSX, XLS, ODS, JSON or JSON Lines files. Google Sheets ('Anyone with the link'), Google Drive, Dropbox, OneDrive and GitHub links are converted to direct downloads. The format is detected from the file content.

## `items` (type: `array`):

Optional: a JSON array of objects to process directly (handy for quick tests and AI agents).

## `sheets` (type: `array`):

Sheet names or 1-based numbers to read from workbooks. Empty = all visible sheets.

## `maxRowsToRead` (type: `integer`):

Stop reading after this many source rows in total (0 = read everything). Useful for previews.

## `filters` (type: `array`):

Keep only rows matching these rules. Each rule: {"field": "price", "op": "between", "value": \[10, 50]}. Operators: equals, notEquals, contains, notContains, startsWith, endsWith, regex, notRegex, gt, gte, lt, lte, between, notBetween, in, notIn, exists, notExists, empty, notEmpty, before, after (dates, also "now-7d" / "30 days ago"), lengthGt, lengthLt, isTrue, isFalse. Text matching is case-insensitive unless "caseSensitive": true. Nest groups with {"any": \[rules]} (OR) or {"all": \[rules]} (AND), negate with {"not": rule}. Fields support dot paths (address.city, images.0.url).

## `filterLogic` (type: `string`):

AND = a row must match every rule. OR = matching any rule is enough.

## `dedupeFields` (type: `array`):

Keep only the first row for each combination of these fields (e.g. 'url', or 'name' + 'city'). Rows where all of these fields are empty are never treated as duplicates. Works across all input sources.

## `dedupeWholeRow` (type: `boolean`):

When no dedupe fields are given: drop rows that are identical in every field.

## `dedupeIgnoreCase` (type: `boolean`):

'ACME Inc ' and 'acme inc' count as the same value.

## `onlyNewItems` (type: `boolean`):

Remember the dedupe key (or the whole row) of every delivered row in a named key-value store and skip rows already delivered in earlier runs. Ideal for scheduled scraper runs: you only get new listings / jobs / products.

## `stateStoreName` (type: `string`):

Named key-value store that holds the remembered keys. Use a different name per pipeline.

## `flattenNested` (type: `boolean`):

Turn {"address": {"city": "Paris"}} into {"address.city": "Paris"}. Applied first, so rules and fields can use the flat names.

## `flattenArrays` (type: `string`):

Keep arrays as they are, join simple arrays into 'a, b, c' text, or expand them to indexed columns (images.0, images.1, ...).

## `flattenSeparator` (type: `string`):

Separator used between nested key names.

## `trimText` (type: `boolean`):

Remove leading / trailing spaces from every text value (also nested).

## `collapseWhitespace` (type: `boolean`):

Replace runs of spaces, tabs and line breaks inside text with a single space.

## `stripHtml` (type: `boolean`):

Remove HTML tags and decode entities (& -> &) in text values.

## `emptyStringsToNull` (type: `boolean`):

Turn empty strings into null.

## `textCase` (type: `object`):

Field -> lower | upper | title | capitalize | slug, e.g. {"email": "lower", "name": "title"}.

## `castFields` (type: `object`):

Field -> string | number | integer | boolean | date | datetime | timestamp | array | json, e.g. {"price": "number", "postedAt": "date"}. Numbers like '$1,299.00' and '15%' are understood. Runs before the filter.

## `castFailure` (type: `string`):

Set it to null, keep the original value, or drop the whole row.

## `addFields` (type: `object`):

New field -> template with {field} placeholders, e.g. {"label": "{title} ({price} {currency})", "source": "google-maps"}. A template that is just "{field}" copies the value with its type.

## `sortBy` (type: `array`):

Fields to sort by, each optionally followed by 'desc', e.g. 'rating desc', 'name'. Numbers and dates sort naturally; empty values go last.

## `offset` (type: `integer`):

Skip this many kept rows (after sorting) before output.

## `limit` (type: `integer`):

Output at most this many rows (0 = no limit). Without sorting, reading stops as soon as the limit is reached.

## `selectFields` (type: `array`):

Output only these fields, in this order (dot paths allowed, e.g. 'address.city'). Empty = keep all.

## `dropFields` (type: `array`):

Fields to remove from the output (dot paths allowed).

## `renameFields` (type: `object`):

Old name -> new name, e.g. {"body_mass_g": "bodyMassGrams"}. Applied last (filters, dedupe and sort use the original names).

## `removeEmptyFields` (type: `boolean`):

Drop fields whose value is null, empty text, \[] or {} from each row.

## `addSourceField` (type: `boolean`):

Add the source dataset ID / file name (and \_sheet for workbooks) to every row.

## `outputToDataset` (type: `boolean`):

Write the result rows to this run's dataset (download as JSON, CSV, Excel, or use via API / integrations).

## `exportFormat` (type: `string`):

Save the result as one file in the key-value store (link in the OUTPUT record). CSV and XLSX flatten nested fields automatically.

## `exportFileName` (type: `string`):

Base name of the export file (extension is added).

## `splitByField` (type: `string`):

One export file per distinct value of this field (max 200, the rest goes to '-other'). For XLSX: one sheet per value in a single workbook.

## `maxFileSizeMb` (type: `integer`):

Larger input files are skipped (not charged).

## Actor input object example

```json
{
  "fileUrls": [
    "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv"
  ],
  "maxRowsToRead": 0,
  "filters": [
    {
      "field": "species",
      "op": "in",
      "value": [
        "Adelie",
        "Gentoo"
      ]
    },
    {
      "field": "body_mass_g",
      "op": "between",
      "value": [
        3500,
        5000
      ]
    },
    {
      "field": "sex",
      "op": "notEmpty"
    }
  ],
  "filterLogic": "and",
  "dedupeWholeRow": false,
  "dedupeIgnoreCase": true,
  "onlyNewItems": false,
  "stateStoreName": "dataset-filter-transform-state",
  "flattenNested": false,
  "flattenArrays": "keep",
  "flattenSeparator": ".",
  "trimText": true,
  "collapseWhitespace": false,
  "stripHtml": false,
  "emptyStringsToNull": false,
  "textCase": {
    "sex": "lower"
  },
  "castFailure": "null",
  "sortBy": [
    "body_mass_g desc"
  ],
  "offset": 0,
  "limit": 0,
  "renameFields": {
    "body_mass_g": "bodyMassGrams"
  },
  "removeEmptyFields": false,
  "addSourceField": false,
  "outputToDataset": true,
  "exportFormat": "none",
  "exportFileName": "filtered-data",
  "maxFileSizeMb": 500
}
```

# Actor output Schema

## `rows` (type: `string`):

No description

## `files` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "fileUrls": [
        "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv"
    ],
    "filters": [
        {
            "field": "species",
            "op": "in",
            "value": [
                "Adelie",
                "Gentoo"
            ]
        },
        {
            "field": "body_mass_g",
            "op": "between",
            "value": [
                3500,
                5000
            ]
        },
        {
            "field": "sex",
            "op": "notEmpty"
        }
    ],
    "textCase": {
        "sex": "lower"
    },
    "sortBy": [
        "body_mass_g desc"
    ],
    "renameFields": {
        "body_mass_g": "bodyMassGrams"
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("gazidev/dataset-filter-transform").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "fileUrls": ["https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv"],
    "filters": [
        {
            "field": "species",
            "op": "in",
            "value": [
                "Adelie",
                "Gentoo",
            ],
        },
        {
            "field": "body_mass_g",
            "op": "between",
            "value": [
                3500,
                5000,
            ],
        },
        {
            "field": "sex",
            "op": "notEmpty",
        },
    ],
    "textCase": { "sex": "lower" },
    "sortBy": ["body_mass_g desc"],
    "renameFields": { "body_mass_g": "bodyMassGrams" },
}

# Run the Actor and wait for it to finish
run = client.actor("gazidev/dataset-filter-transform").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "fileUrls": [
    "https://raw.githubusercontent.com/mwaskom/seaborn-data/master/penguins.csv"
  ],
  "filters": [
    {
      "field": "species",
      "op": "in",
      "value": [
        "Adelie",
        "Gentoo"
      ]
    },
    {
      "field": "body_mass_g",
      "op": "between",
      "value": [
        3500,
        5000
      ]
    },
    {
      "field": "sex",
      "op": "notEmpty"
    }
  ],
  "textCase": {
    "sex": "lower"
  },
  "sortBy": [
    "body_mass_g desc"
  ],
  "renameFields": {
    "body_mass_g": "bodyMassGrams"
  }
}' |
apify call gazidev/dataset-filter-transform --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gazidev/dataset-filter-transform"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3pWx1pl0aID38XCeA/builds/jHrYNfriKJnJ47wMv/openapi.json