# Dataset Transformer — Filter, Dedupe, Sort & Reshape Data (`nexgensignal/dataset-transformer-api`) Actor

Transform Apify datasets or JSON / CSV links: filter rows, remove duplicates, sort, flatten nested fields, unwind lists, pick and rename columns. Merge several datasets into one clean result for export as JSON, CSV or Excel. Part of NexGen Signal — data tools and official-source data products

- **URL**: https://apify.com/nexgensignal/dataset-transformer-api.md
- **Developed by:** [NexGen Signal](https://apify.com/nexgensignal) (community)
- **Categories:** Integrations, Automation, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $33.50 / 1,000 output rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## 🧮 Dataset Transformer — Filter, Dedupe, Sort & Reshape Data

Take rows from **your Apify datasets, JSON / JSON Lines / CSV links, or pasted JSON**, and get a clean new dataset: unwind list fields into rows, flatten nested objects into columns, filter with 17 operators, remove duplicates, sort, keep and rename fields, skip and limit.

Several datasets merge into one. Field names can be dot paths (`properties.mag`, `offers.0.price`). Numbers compare as numbers even when they arrive as text from CSV; text matching ignores case unless you ask otherwise. The result exports from Apify as JSON, CSV, Excel or XML like any dataset.

The unit of output is one output row delivered. Rows you filter out are not billed.

### Who buys this and for what job

- **Apify users** cleaning or merging the output of scrapers before export or integration.
- **Analysts** turning nested JSON from an API into a flat table.
- **Automation builders** adding a filter / dedupe step between two actors.
- **Data teams** sorting and trimming large CSV files without writing code.

Buyers usually search for this as **"dataset transformer"**, **"merge datasets"** or **"JSON to CSV flatten"**.

### 📊 Sample Output

![Table of transformed rows: magnitude, place, time and URL of the strongest recent earthquakes.](https://api.apify.com/v2/key-value-stores/IXCaMKjxSmUTLHhmq/records/dataset-transformer-api.png)

| magnitude | place | time_ms | url |
|---|---|---|---|
| 6.9 | 76 km NE of Ozernovskiy, Russia | 1792121731925 | https://earthquake.usgs.gov/earthquakes/eventpage/us6000u2mq |
| 6.8 | central Mid-Atlantic Ridge | 1791894187248 | https://earthquake.usgs.gov/earthquakes/eventpage/us6000u214 |
| 6.7 | central Mid-Atlantic Ridge | 1791870286919 | https://earthquake.usgs.gov/earthquakes/eventpage/us6000u1z5 |
| 6.2 | 79 km SSE of Panguna, Papua New Guinea | 1791727643188 | https://earthquake.usgs.gov/earthquakes/eventpage/us6000u1kv |
| 6.1 | 17 km W of Río Grande, Panama | 1791847236003 | https://earthquake.usgs.gov/earthquakes/eventpage/us6000u1wn |

Rows from live run `8LuJln2xafXxCOetY` on Apify (the Store example: the USGS magnitude 4.5+ weekly GeoJSON, flattened, filtered to magnitude 5 and above, sorted strongest first, with four renamed columns). Notice how nested GeoJSON features became four flat, named columns.

### ✅ What you get

Every field of each record:

- `Your fields` — the output has the fields you choose; with no field list, every field of each row is kept.

### ⚙️ Sample inputs

**Strongest recent earthquakes (the Store example)**

```json
{
  "dataUrls": ["https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/4.5_week.geojson"],
  "flatten": true,
  "filters": [{"field": "properties.mag", "op": "gte", "value": 5}],
  "sortBy": "properties.mag",
  "sortDescending": true,
  "fields": ["properties.mag", "properties.place", "properties.time", "properties.url"],
  "rename": {"properties.mag": "magnitude", "properties.place": "place", "properties.time": "time_ms", "properties.url": "url"},
  "maxRows": 10
}
```

From GeoJSON to a flat table.

**Merge two of your datasets and dedupe**

```json
{
  "datasetIds": ["DATASET_ID_1", "DATASET_ID_2"],
  "dedupeBy": ["url"],
  "maxRows": 100000
}
```

Pick the datasets in the Source datasets field.

**One row per order line**

```json
{
  "rows": [{"order": "A1", "items": [{"sku": "x", "qty": 2}, {"sku": "y", "qty": 1}]}],
  "unwind": "items",
  "flatten": true
}
```

Unwind a list, then flatten.

Input fields:

| Field | Meaning |
|---|---|
| `datasetIds` | your Apify datasets (picked in the input form). |
| `dataUrls` | JSON, JSON Lines or CSV links (up to 100). |
| `rows` | pasted JSON array (up to 100,000). |
| `rowsPath` | where the row list sits inside a JSON object. |
| `unwind` | list field to split into one row per element. |
| `flatten` | nested fields to dot-named columns. |
| `filters`, `filterMode` | conditions (eq, ne, gt, gte, lt, lte, contains, notContains, startsWith, endsWith, in, notIn, exists, notExists, empty, notEmpty, regex), all or any. |
| `dedupeBy` | keep the first row per combination of these fields. |
| `sortBy`, `sortDescending` | sort field and direction. |
| `fields`, `rename`, `dropFields` | pick, rename and drop columns. |
| `offset`, `maxRows`, `maxInputRows` | skip, limit (default 100,000 output rows; each billed) and input cap (default 200,000). |

### 🧾 JSON sample record

```json
{
  "magnitude": 6.9,
  "place": "76 km NE of Ozernovskiy, Russia",
  "time_ms": 1792121731925,
  "url": "https://earthquake.usgs.gov/earthquakes/eventpage/us6000u2mq"
}
```

### 🔧 How it works

Steps run in this order: unwind, flatten, filter, de-duplicate, sort, pick / rename / drop, skip and limit. Apify datasets are read through the API with the access you grant in the input form (the actor runs with limited permissions and cannot see other storages). Links are read with a robots.txt check.

Delivery is pushed in batches of 500 and charged after each batch. The run receipt (`RUN_RECEIPT`) lists every source with rows read and used, the row count after each step, and the delivered and charged counts.

### 💰 Pricing example

Per row delivered, no start fee, no monthly charge:

| Apify plan | Price per row |
|---|---|
| Free | $0.05 |
| Bronze | $0.045 |
| Silver | $0.04 |
| Gold, Platinum, Diamond | $0.0335 |

- 10 rows: $0.50 on the Free plan, $0.335 on Gold.
- 10,000 rows: $500.00 on Free, $335.00 on Gold.
- 100,000 rows: $5,000.00 on Free, $3,350.00 on Gold.

**Coverage and size.** Proof runs on 17 Oct 2026: 622 USGS rows read and 40 delivered in about 3 seconds; one of our own Apify datasets (30 rows) read through the dataset picker; a 139-row USGS CSV deduplicated by magnitude type to 4 rows.

### ⚖️ Legal & ToS

It processes data you supply or point it at. You choose the links. Make sure you have the right to download and use the content you point it at; the actor fetches only what you give it (and, where you turn it on, what those pages link to).

### ❓ FAQ

**Does it change my source datasets?**
No. Sources are only read; the result is a new dataset.

**How large can the input be?**
Up to 2 million rows read in one run (maxInputRows); give the run more memory for large inputs.

**Can I filter on nested fields?**
Yes, with dot paths, with or without flatten.

### 🆘 Troubleshooting

- **dataset not found or not shared with this run** — Pick the dataset in the Source datasets field so the run is allowed to read it.
- **rowsPath not found** — Check where the list sits inside the JSON; leave empty when the JSON has one list of objects.
- **SOURCE UNAVAILABLE** — None of your sources could be reached after retries. Nothing was charged.

### 🔗 Related Actors

- [Website Contact Scraper](https://apify.com/nexgensignal/bulk-website-contact-extractor) — contact details from website lists
- [PR Newswire Scraper](https://apify.com/nexgensignal/press-release-monitor-api) — press releases by company or topic
- [Website Tech Stack Detector](https://apify.com/nexgensignal/website-tech-stack-api) — what a site is built with

🏢 **About NexGen Signal** — data tools and official-source data products on Apify. Browse the catalog: [apify.com/nexgensignal](https://apify.com/nexgensignal)

### ⭐ Found this useful?

If this actor saved you time, a rating on its Apify Store page helps other buyers find it.

Part of NexGen Signal — data tools and official-source data products.

# Actor input Schema

## `datasetIds` (type: `array`):

Apify datasets to read (merged in order). Pick them here so this run is allowed to read them.

## `dataUrls` (type: `array`):

Links to JSON (array, or an object holding one list), JSON Lines or CSV. Up to 100.

## `rows` (type: `array`):

A JSON array of rows to transform, up to 100,000.

## `rowsPath` (type: `string`):

Where the list of rows sits inside a JSON object, e.g. data.items. Empty = the array itself, or the object's only list of objects.

## `unwind` (type: `string`):

A list field to split into one row per element, e.g. tags or offers.

## `flatten` (type: `boolean`):

Turn nested objects and lists into dot-named columns, e.g. properties.mag, tags.0.

## `filters` (type: `array`):

Conditions like {"field": "price", "op": "lt", "value": 50}. op: eq, ne, gt, gte, lt, lte, contains, notContains, startsWith, endsWith, in, notIn, exists, notExists, empty, notEmpty, regex. Text matching ignores case unless "caseSensitive": true. Fields use dot paths.

## `filterMode` (type: `string`):

How several filters combine.

## `dedupeBy` (type: `array`):

Keep the first row for each combination of these fields.

## `sortBy` (type: `string`):

Field to sort on (numbers sort as numbers). Rows without it go last.

## `sortDescending` (type: `boolean`):

Largest / latest first.

## `fields` (type: `array`):

Keep only these fields, in this order (dot paths allowed). Empty = all fields.

## `rename` (type: `object`):

Old name -> new name, e.g. {"properties.mag": "magnitude"}.

## `dropFields` (type: `array`):

Remove these fields from the output.

## `offset` (type: `integer`):

Skip this many result rows (after sorting).

## `maxRows` (type: `integer`):

Stop after this many output rows. Each output row is billed.

## `maxInputRows` (type: `integer`):

Read at most this many rows across all sources.

## Actor input object example

```json
{
  "dataUrls": [
    "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/4.5_week.geojson"
  ],
  "flatten": true,
  "filters": [
    {
      "field": "properties.mag",
      "op": "gte",
      "value": 5
    }
  ],
  "filterMode": "all",
  "dedupeBy": [],
  "sortBy": "properties.mag",
  "sortDescending": true,
  "fields": [
    "properties.mag",
    "properties.place",
    "properties.time",
    "properties.url"
  ],
  "rename": {
    "properties.mag": "magnitude",
    "properties.place": "place",
    "properties.time": "time_ms",
    "properties.url": "url"
  },
  "dropFields": [],
  "offset": 0,
  "maxRows": 10,
  "maxInputRows": 200000
}
```

# Actor output Schema

## `results` (type: `string`):

The delivered (and billed) records.

## `receipt` (type: `string`):

Per-link outcome, robots result, delivered and charged counts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "dataUrls": [
        "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/4.5_week.geojson"
    ],
    "flatten": true,
    "filters": [
        {
            "field": "properties.mag",
            "op": "gte",
            "value": 5
        }
    ],
    "sortBy": "properties.mag",
    "sortDescending": true,
    "fields": [
        "properties.mag",
        "properties.place",
        "properties.time",
        "properties.url"
    ],
    "rename": {
        "properties.mag": "magnitude",
        "properties.place": "place",
        "properties.time": "time_ms",
        "properties.url": "url"
    },
    "maxRows": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("nexgensignal/dataset-transformer-api").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "dataUrls": ["https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/4.5_week.geojson"],
    "flatten": True,
    "filters": [{
            "field": "properties.mag",
            "op": "gte",
            "value": 5,
        }],
    "sortBy": "properties.mag",
    "sortDescending": True,
    "fields": [
        "properties.mag",
        "properties.place",
        "properties.time",
        "properties.url",
    ],
    "rename": {
        "properties.mag": "magnitude",
        "properties.place": "place",
        "properties.time": "time_ms",
        "properties.url": "url",
    },
    "maxRows": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("nexgensignal/dataset-transformer-api").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "dataUrls": [
    "https://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/4.5_week.geojson"
  ],
  "flatten": true,
  "filters": [
    {
      "field": "properties.mag",
      "op": "gte",
      "value": 5
    }
  ],
  "sortBy": "properties.mag",
  "sortDescending": true,
  "fields": [
    "properties.mag",
    "properties.place",
    "properties.time",
    "properties.url"
  ],
  "rename": {
    "properties.mag": "magnitude",
    "properties.place": "place",
    "properties.time": "time_ms",
    "properties.url": "url"
  },
  "maxRows": 10
}' |
apify call nexgensignal/dataset-transformer-api --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nexgensignal/dataset-transformer-api"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/erIheJXdJwf6FZdv1/builds/tg7iiJbefXezYB4U6/openapi.json