# Dataset Transformer – Filter, Dedupe, Flatten & JSON to CSV (`chris_elder/dataset-transformer`) Actor

Clean and reshape data from an Apify dataset, pasted JSON, or a JSON, JSONL or CSV file URL: filter rows, remove duplicates by any field, flatten nested JSON, pick and rename fields, sort, and export CSV. $0.50 per 1,000 output rows; dropped rows are free.

- **URL**: https://apify.com/chris_elder/dataset-transformer.md
- **Developed by:** [Chris Elder](https://apify.com/chris_elder) (community)
- **Categories:** Developer tools, Automation, Integrations
- **Stats:** 3 total users, 2 monthly users, 60.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.30 / 1,000 row outputs

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Dataset Transformer – Filter, Dedupe, Flatten & JSON to CSV

**Dataset Transformer** cleans and reshapes tabular data in one step: filter rows, remove duplicates, flatten nested JSON, pick or rename fields, sort, and export **CSV**. Feed it an Apify dataset ID (for example the output of another scraper), a pasted JSON array, or a public JSON / JSONL / CSV file URL.

Store page: [apify.com/chris_elder/dataset-transformer](https://apify.com/chris_elder/dataset-transformer)

✅ Pay per result: **$0.50 per 1,000 output rows** · 🆓 Dropped / filtered rows are never charged · 🔑 No API key

[▶ Try it with one click](#how-to-use-it) · [📄 Real example output](#output-example-from-a-real-public-run) · [💲 Pricing](#how-much-does-it-cost-to-clean-a-dataset) · [🔌 API & integrations](#integrations-api-ai-agents-make-zapier) · [❓ FAQ](#faq) · [🛟 Support](#support-reliability-and-feature-requests)

### What does Dataset Transformer do?

- 🧹 **Filter rows** with 14 operators (`eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `contains`, `notContains`, `startsWith`, `endsWith`, `exists`, `notExists`, `in`, `regex`) combined with AND or OR
- 🔁 **Remove duplicates** by one or more fields (dot paths like `meta.city`); the first occurrence is kept
- 📦 **Flatten nested JSON** into dotted keys (`{ "meta": { "city": "London" } }` → `meta.city`)
- ✏️ **Pick, omit or rename** fields
- ↕️ **Sort** ascending or descending by any field
- 📄 **JSON to CSV** – optional `OUTPUT.csv` in the Key-Value store; CSV files you load come back as JSON dataset rows (**CSV to JSON**)
- 🆓 **Dropped rows free** – only rows written to the output dataset are charged

### Use cases

- **Scraper users** – clean and dedupe a noisy dataset before it goes to Google Sheets, Airtable or a warehouse
- **Agents & n8n/Make** – “filter + dedupe this run’s dataset” as one step after another Actor
- **Developers** – flatten nested JSON and convert JSON to CSV without writing a script
- **Ops / QA** – keep only rows that pass rules (e.g. price exists, rating ≥ 4) after a crawl

### How to use it

1. Click **Try for free** – the prefill filters `score ≥ 5`, removes the duplicate Ada/London row, flattens and renames `meta.city` → `city`, sorts by score and exports CSV (2 output rows ≈ $0.0015 including the start fee).
2. To clean real data, replace `items` with `datasetId` (any dataset in your account, or a public one) or `fileUrl` (JSON array, JSONL or CSV).
3. Download the cleaned dataset (JSON, CSV, Excel from the Storage tab) or `OUTPUT.csv` from the run’s Key-Value store.

#### Input example

```json
{
  "items": [
    { "id": 1, "name": "Ada", "meta": { "city": "London" }, "score": 9 },
    { "id": 2, "name": "Ada", "meta": { "city": "London" }, "score": 9 },
    { "id": 3, "name": "Bob", "meta": { "city": "Berlin" }, "score": 4 },
    { "id": 4, "name": "Cara", "meta": { "city": "Paris" }, "score": 7 }
  ],
  "filters": [{ "field": "score", "op": "gte", "value": 5 }],
  "dedupeBy": ["name", "meta.city"],
  "flatten": true,
  "rename": { "meta.city": "city" },
  "sortBy": "score",
  "sortDir": "desc",
  "exportCsv": true,
  "maxRows": 10
}
```

Other options: `filterMode` (`and`/`or`), `pick`, `omit`, `csvKey`.

#### Example: 4 rows in → 2 clean rows out

| Step | Rows | What happened |
|---|---|---|
| Input | 4 | Ada (9), Ada (9, duplicate), Bob (4), Cara (7) |
| Filter `score ≥ 5` | 3 | Bob dropped (free) |
| Dedupe by `name` + `meta.city` | 2 | second Ada/London dropped (free) |
| Flatten + rename + sort desc | 2 | `meta.city` → `city`, highest score first |

#### Output example (from a real public run)

Public run data (no login needed): [dataset JSON](https://api.apify.com/v2/datasets/DRGCnvRQfbra3eYwm/items?format=json\&clean=1) · [dataset CSV](https://api.apify.com/v2/datasets/DRGCnvRQfbra3eYwm/items?format=csv\&clean=1) · [OUTPUT.csv](https://api.apify.com/v2/key-value-stores/nupfh3En1qhJezKdn/records/OUTPUT.csv) · [RUN_SUMMARY](https://api.apify.com/v2/key-value-stores/nupfh3En1qhJezKdn/records/RUN_SUMMARY)

Dataset rows:

```json
[
  { "id": 1, "name": "Ada", "score": 9, "city": "London" },
  { "id": 4, "name": "Cara", "score": 7, "city": "Paris" }
]
```

`OUTPUT.csv`:

```
id,name,score,city
1,Ada,9,London
4,Cara,7,Paris
```

`RUN_SUMMARY`:

```json
{ "inputCount": 4, "outputCount": 2, "droppedFilter": 1, "droppedDedupe": 1, "skippedBudget": 0, "budgetHit": false, "exportCsv": true, "csvKey": "OUTPUT.csv", "source": "items" }
```

### How much does it cost to clean a dataset?

| Event | Free / Bronze | Silver | Gold+ |
|---|---|---|---|
| Actor start (× memory GB, min 1) | $0.0005 | $0.0005 | $0.0005 |
| Row output | $0.0005 | $0.0004 | $0.0003 |

**Filtered, deduped and invalid rows are free.** Only rows written to the output dataset are charged.

| Job | Approx. cost |
|---|---|
| Prefill (2 output rows, 512 MB) | 2 × $0.0005 + $0.0005 start = $0.0015 |
| 1,000 output rows | $0.50 + start |
| 10,000 output rows (Gold) | 10,000 × $0.0003 = $3.00 + start |

Use **Max cost per run** to cap spend; the Actor stops writing rows when the cap is reached (`budgetHit: true` in `RUN_SUMMARY`).

### Integrations (API, AI agents, Make, Zapier)

Clean another Actor’s output: pass that run’s `defaultDatasetId` as `datasetId`.

```bash
curl -X POST "https://api.apify.com/v2/acts/chris_elder~dataset-transformer/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"datasetId":"YOUR_DATASET_ID","dedupeBy":["url"],"flatten":true,"exportCsv":true}'
```

Python (`pip install apify-client`):

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("chris_elder/dataset-transformer").call(run_input={
    "datasetId": "YOUR_DATASET_ID",
    "filters": [{"field": "price", "op": "exists"}],
    "dedupeBy": ["url"],
    "exportCsv": True,
})
rows = client.dataset(run["defaultDatasetId"]).list_items().items
```

MCP (AI agents): `https://mcp.apify.com/?tools=chris_elder/dataset-transformer`

Make / Zapier / n8n: add the Apify **Run Actor** step after your scraper, map its dataset ID into `datasetId`, and read the cleaned rows. Works with Apify schedules and webhooks.

### Limits and good to know

- One source per run (`items`, `datasetId` or `fileUrl`). It does not merge several datasets.
- `maxRows` default 10,000 (max 100,000). For very large datasets, filter first or raise memory.
- CSV input is parsed with a header row and values arrive as text; `gt`/`gte`/`lt`/`lte` convert values to numbers, `eq` compares loosely (`"9"` equals `9`).
- Excel (XLSX) files are not supported as input yet.

### FAQ

**How do I convert JSON to CSV?** Load the JSON (pasted, dataset or file URL), turn on `flatten` for nested fields and `exportCsv`, then download `OUTPUT.csv`.

**How do I remove duplicates from an Apify dataset?** Set `datasetId` and list the fields that identify a duplicate in `dedupeBy` (e.g. `["url"]`).

**Does it modify my source dataset?** No. It only reads, and writes a new dataset on this run.

**Nested fields?** Use dot paths (`meta.city`) and/or turn on `flatten`.

**Why was I charged less than the input size?** Filtered and deduped rows are free; you pay only for output rows.

### Support, reliability and feature requests

- Dropped rows are free.
- Every release is tested on real inputs; the one-click example works.
- Issues answered within 2 business days via the Actor **Issues** tab.
- Breaking input or price changes are announced in this README first.

Built by Chris Elder with AI-assisted development; every release is reviewed and tested.

### Need the data done for you?

If you'd rather not set up runs yourself, the UK company behind these tools (GPT Shop Ltd) offers a fixed-price done-for-you scrape. Tell us the website and the fields you need, and you get a clean CSV or Google Sheet within 48 hours of the confirmed brief, up to 5,000 rows, for £149 or $179. If we can't get the data, you're refunded in full. [See the done-for-you scraper service](https://sme-free-tools.pages.dev/done-for-you-scraper/?utm_source=apify\&utm_medium=readme\&utm_campaign=dataset-transformer)

### More tools by Chris Elder

[Bulk File Downloader](https://apify.com/chris_elder/bulk-file-downloader) · [HTTP Request Sender](https://apify.com/chris_elder/http-request-sender) · [Sitemap URL Extractor](https://apify.com/chris_elder/sitemap-url-extractor) · [PDF to Markdown](https://apify.com/chris_elder/pdf-to-markdown) · [all tools](https://apify.com/chris_elder)

### More use cases (FAQ)

- **How do I convert an Apify dataset or JSON to CSV/Excel?** Set `exportCsv: true`.
- **How do I remove duplicate rows from scraped data?** Use `dedupeBy` with one or more fields (nested paths such as `meta.city` work).
- **Can I flatten nested JSON for Google Sheets?** Yes. Set `flatten: true`, then rename columns with `rename`.

# Actor input Schema

## `items` (type: `array`):

Paste rows directly. Use this OR datasetId OR fileUrl.

## `datasetId` (type: `string`):

Read rows from an existing Apify dataset (same account or shared). Read-only access to this one dataset is requested. Use this OR items OR fileUrl.

## `fileUrl` (type: `string`):

http(s) URL of a JSON array, JSONL or CSV file to transform.

## `filters` (type: `array`):

Keep rows matching these rules. Ops: eq, ne, gt, gte, lt, lte, contains, notContains, startsWith, endsWith, exists, notExists, in, regex. Dot paths supported (meta.city).

## `filterMode` (type: `string`):

Combine filters with AND or OR.

## `dedupeBy` (type: `array`):

Field names (dot paths OK). First occurrence kept.

## `flatten` (type: `boolean`):

Turn { meta: { city } } into { "meta.city": ... }.

## `pick` (type: `array`):

If set, keep only these fields (after flatten).

## `omit` (type: `array`):

Drop these fields (ignored when pick is set).

## `rename` (type: `object`):

Map of oldField → newField.

## `sortBy` (type: `string`):

Field name to sort by (dot paths OK).

## `sortDir` (type: `string`):

Ascending or descending sort.

## `maxRows` (type: `integer`):

maxRows

## `exportCsv` (type: `boolean`):

When true, also write OUTPUT.csv to the Key-Value store.

## `csvKey` (type: `string`):

Key for the CSV record in the default Key-Value store.

## Actor input object example

```json
{
  "items": [
    {
      "id": 1,
      "name": "Ada",
      "meta": {
        "city": "London"
      },
      "score": 9
    },
    {
      "id": 2,
      "name": "Ada",
      "meta": {
        "city": "London"
      },
      "score": 9
    },
    {
      "id": 3,
      "name": "Bob",
      "meta": {
        "city": "Berlin"
      },
      "score": 4
    },
    {
      "id": 4,
      "name": "Cara",
      "meta": {
        "city": "Paris"
      },
      "score": 7
    }
  ],
  "filters": [
    {
      "field": "score",
      "op": "gte",
      "value": 5
    }
  ],
  "filterMode": "and",
  "dedupeBy": [
    "name",
    "meta.city"
  ],
  "flatten": true,
  "rename": {
    "meta.city": "city"
  },
  "sortBy": "score",
  "sortDir": "desc",
  "maxRows": 10000,
  "exportCsv": true,
  "csvKey": "OUTPUT.csv"
}
```

# Actor output Schema

## `overview` (type: `string`):

No description

## `csv` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "items": [
        {
            "id": 1,
            "name": "Ada",
            "meta": {
                "city": "London"
            },
            "score": 9
        },
        {
            "id": 2,
            "name": "Ada",
            "meta": {
                "city": "London"
            },
            "score": 9
        },
        {
            "id": 3,
            "name": "Bob",
            "meta": {
                "city": "Berlin"
            },
            "score": 4
        },
        {
            "id": 4,
            "name": "Cara",
            "meta": {
                "city": "Paris"
            },
            "score": 7
        }
    ],
    "filters": [
        {
            "field": "score",
            "op": "gte",
            "value": 5
        }
    ],
    "dedupeBy": [
        "name",
        "meta.city"
    ],
    "rename": {
        "meta.city": "city"
    },
    "sortBy": "score"
};

// Run the Actor and wait for it to finish
const run = await client.actor("chris_elder/dataset-transformer").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "items": [
        {
            "id": 1,
            "name": "Ada",
            "meta": { "city": "London" },
            "score": 9,
        },
        {
            "id": 2,
            "name": "Ada",
            "meta": { "city": "London" },
            "score": 9,
        },
        {
            "id": 3,
            "name": "Bob",
            "meta": { "city": "Berlin" },
            "score": 4,
        },
        {
            "id": 4,
            "name": "Cara",
            "meta": { "city": "Paris" },
            "score": 7,
        },
    ],
    "filters": [{
            "field": "score",
            "op": "gte",
            "value": 5,
        }],
    "dedupeBy": [
        "name",
        "meta.city",
    ],
    "rename": { "meta.city": "city" },
    "sortBy": "score",
}

# Run the Actor and wait for it to finish
run = client.actor("chris_elder/dataset-transformer").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "items": [
    {
      "id": 1,
      "name": "Ada",
      "meta": {
        "city": "London"
      },
      "score": 9
    },
    {
      "id": 2,
      "name": "Ada",
      "meta": {
        "city": "London"
      },
      "score": 9
    },
    {
      "id": 3,
      "name": "Bob",
      "meta": {
        "city": "Berlin"
      },
      "score": 4
    },
    {
      "id": 4,
      "name": "Cara",
      "meta": {
        "city": "Paris"
      },
      "score": 7
    }
  ],
  "filters": [
    {
      "field": "score",
      "op": "gte",
      "value": 5
    }
  ],
  "dedupeBy": [
    "name",
    "meta.city"
  ],
  "rename": {
    "meta.city": "city"
  },
  "sortBy": "score"
}' |
apify call chris_elder/dataset-transformer --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,chris_elder/dataset-transformer"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/N7HixZ52haD5ec7th/builds/aINYLQ68oFjW7T0cm/openapi.json