Dataset Transformer: Filter, Dedupe, JSON to CSV

What does Dataset Transformer do?

Dataset Transformer cleans up Apify datasets and CSV, JSON or Excel files: filter rows, remove duplicates, sort, keep and rename fields, and flatten nested JSON. Give it one of your datasets, a file link, or the rows themselves as pasted JSON; you get a new dataset, and optionally a CSV or JSONL file (JSON to CSV, CSV to JSON or Excel to JSON in one step).

It's the clean-up step between a scraper and your spreadsheet, CRM or database: "only the jobs in Berlin, one row per URL, newest first, with these five columns".

Input fields · API · Use it from Claude, ChatGPT or Cursor (MCP)

Jump to: Fields · Price · How to use · Input · Output · AI agents (MCP) · Limits · FAQ

Try it in one click: the input comes pre-filled with the US Treasury's official exchange rates (a public CSV of 174 currencies). The example keeps the currencies worth more than one US dollar, removes a repeated one, sorts by rate and keeps four renamed columns: 9 rows, about $0.009 (9 × $0.001, plus $0.00005 for the run start). Then pick your own dataset or file. Cleaning a scraper's results? See scraper results to a clean CSV.

What data does Dataset Transformer return?

Your own rows, with the fields you chose. From the example:

Fields you asked for that a row doesn't have are null, so every row has the same columns. Nested fields can be flattened into plain columns (address.city). The run's RUN_STATS record counts the rows read, filtered out, duplicates, skipped and written. More under Output.

How much does it cost to filter and dedupe a dataset?

You pay per row written to the output: $1.00 per 1,000 rows, plus $0.00005 each time a run starts.

It's cheaper on paid Apify plans: $0.90 per 1,000 rows on Starter, $0.80 on Scale and $0.70 on Business. The prices on this page are the Free-plan price, so on a paid plan you pay less than the examples show.

• The example below: 9 rows × $0.001 = $0.009, plus the start fee; the 164 rows filtered out and the 1 duplicate were free.
• A month, for example: a daily scraper's output cleaned down to 200 rows a day: 6,000 rows × $0.001 = $6.00, plus 30 starts ($0.0015).
• Caps: Max rows per run in the input (with sorting, these are the top rows), and Maximum cost per run in the run options. The run stops cleanly at whichever comes first.

Never charged: rows that are filtered out, duplicates, empty after your field selection, or too large. The optional CSV or JSONL file is free.

How to filter, dedupe and convert a dataset

1. Open Dataset Transformer and click Try for free (or Start if you're signed in).
2. Pick one of your datasets in Apify dataset, upload a file or paste its link in Or: file URL or upload, or paste the rows themselves in Or: paste rows (JSON array).
3. Add Filters (one per line, e.g. price > 10), Remove duplicates by, Sort by and Fields to keep (and rename) as you need.
4. Optional: Also save as a file (CSV or JSON Lines).
5. Click Start, then open the Output tab and export as JSON, CSV or Excel.

Example: the US Treasury's exchange rates

The pre-filled input (a public CSV, filtered, deduplicated, sorted and trimmed to four renamed columns):

{"fileUrl": "https://api.fiscaldata.treasury.gov/services/api/fiscal_service/v1/accounting/od/rates_of_exchange?format=csv&filter=record_date:eq:2025-12-31&page[size]=300",
 "filters": ["exchange_rate < 1"], "dedupeBy": ["currency", "exchange_rate"], "sortBy": ["exchange_rate"],
 "fields": ["country", "currency", "exchange_rate -> unitsPerUsd", "record_date -> date"]}

It read 174 rows and wrote 9. The first three (real output from a local run on 2026-09-25):

[
  {"country": "Kuwait", "currency": "Dinar", "unitsPerUsd": "0.308", "date": "2025-12-31"},
  {"country": "Bahrain", "currency": "Dinar", "unitsPerUsd": "0.377", "date": "2025-12-31"},
  {"country": "Bulgaria", "currency": "Lev New", "unitsPerUsd": "0.86", "date": "2025-12-31"}
]

Example: scraper results to a clean CSV

A typical job for Dataset Transformer: a Google Maps-style scraper ran two searches over the same area ("coffee shop in Austin, TX" and "cafe in Austin, TX"), so places repeat, and every row has 24 fields, most of them nested or not needed in a spreadsheet (opening hours, image lists, search rank, scrape time). The goal is one CSV with one row per open, well-rated place that has a phone number, busiest first, with plain column names. The places, phone numbers (555-01xx) and websites (.example) are made up for this example.

Before: the scraper's 15 rows (6 of the 24 fields shown; the rest include placeId, url, location.lat, location.lng, openingHours, imageUrls, additionalInfo, rank and scrapedAt):

The input (replace YOUR-SCRAPER-DATASET-ID with your scraper run's dataset, or pick it in Apify dataset):

{
  "datasetId": "YOUR-SCRAPER-DATASET-ID",
  "filters": ["permanentlyClosed != true", "totalScore >= 4.3", "phone is not empty"],
  "dedupeBy": ["placeId"],
  "sortBy": ["reviewsCount desc"],
  "fields": ["title -> name", "categoryName -> category", "totalScore -> rating", "reviewsCount -> reviews",
             "phone", "website", "street", "city", "postalCode -> zip",
             "location.lat -> lat", "location.lng -> lng", "url -> googleMapsUrl"],
  "exportFormat": "csv"
}

After: 7 rows with 12 columns, in the dataset and in OUTPUT.csv (6 of the 12 columns shown; the others are category, street, city, lat, lng and googleMapsUrl):

The first lines of OUTPUT.csv (real output from a local run on 2026-10-04):

name,category,rating,reviews,phone,website,street,city,zip,lat,lng,googleMapsUrl
Riverbend Bakery & Coffee,Bakery,4.8,2140,(512) 555-0105,https://riverbendbakery.example,3300 Riverbend Dr,Austin,78746,30.2851,-97.8012,https://www.google.com/maps/place/?q=place_id:ChIJexample000005
Juniper & Oak Coffee,Coffee shop,4.7,1284,(512) 555-0101,https://juniperoak.example,418 Juniper St,Austin,78702,30.2641,-97.7289,https://www.google.com/maps/place/?q=place_id:ChIJexample000001

The run's status: "7 rows written from 15 rows read; 4 filtered out; 4 duplicates removed; file saved as OUTPUT.csv". You pay for the 7 rows written (7 × $0.001 = $0.007 on the Free plan, plus the run start); the 4 filtered rows and the 4 duplicates are free. Filters, duplicates and sorting use the scraper's own field names (totalScore, placeId); the new names (rating, zip) only appear in the output.

Ready-to-run examples

Each example opens with the input already filled in. Run it as it is, or change the input first.

• Filter, dedupe and sort a CSV file from a URL: Clean a CSV, JSON or JSONL file in one step: keep only the rows that match your filters, remove duplicates, sort, and keep or rename the columns you need.
• Convert a JSON API response to a CSV file: Turn nested JSON from an API into flat rows: pick the list inside the response, flatten nested fields, rename the columns you need and get a CSV file.
• Convert a CSV file to JSON or JSON Lines: Load a CSV from a URL and get it as a dataset you can download as JSON, Excel or XML, plus a JSON Lines file, with the columns renamed the way you want.
• Sort a CSV file and keep the top 10 rows: Sort any CSV by one or more columns, numbers as numbers, and keep only the top rows and the columns you need.
• Convert an Excel file to JSON or CSV: Read an Excel workbook (.xlsx) from a URL into rows, with numbers kept as numbers and dates as ISO text, then filter, rename and save a CSV.
• Read an Excel table that starts below a title: When a title and notes sit above the table, as in many official spreadsheets, set the header row and get clean rows from the table only.
• Filter Google Maps leads by rating and reviews: Keep only places with a website and phone, rated 4.5 or more with 50+ reviews and still open, one row per place, as a CSV.
• Convert XLSX to CSV: an Excel spreadsheet to a CSV file: Turn an Excel workbook at a web address into a CSV file and a dataset, numbers kept as numbers.

Input

Filters. == != > >= < <= compare numbers as numbers (also numbers stored as text, as in CSV files) and anything else as text, so ISO dates like 2026-01-31 compare correctly. contains, not contains, starts with and ends with ignore upper and lower case. is empty and is not empty take no value. Examples:

price > 10
country == US
title contains remote
postedAt >= 2026-09-01
email is not empty
address.city == "New York"

Filters, duplicates and sorting use your data's original field names; renaming happens last.

A full example, for a dataset of job postings:

{
  "datasetId": "YOUR-DATASET-ID",
  "filters": ["location contains Berlin", "salary is not empty"],
  "dedupeBy": ["url"],
  "sortBy": ["postedAt desc"],
  "fields": ["title", "company.name -> company", "salary", "url", "postedAt"],
  "exportFormat": "csv",
  "maxResults": 500
}

Output

One dataset row per output row, with the fields you chose (or all of the source's fields). Fields you asked for that a row doesn't have are null, so every row has the same columns. With Also save as a file, the same rows are saved as OUTPUT.csv or OUTPUT.jsonl in the run's key-value store (nested values become JSON text in CSV cells). The RUN_STATS record has the counts: rows read, filtered out, duplicates, skipped, written.

To download your latest file from code: GET https://api.apify.com/v2/acts/humble-echidna~dataset-transform/runs/last/key-value-store/records/OUTPUT.csv?status=SUCCEEDED with your API token (for a saved task, actor-tasks/<task id> in place of acts/humble-echidna~dataset-transform).

The dataset itself can also be downloaded from Apify as CSV, JSON, Excel, XML or HTML at any time.

Run it on a schedule, or from your own code

1. Save your input as a task (Save as a new task, top right of the actor page) and add it to a schedule (Console → Schedules → Create new): for example daily, a few minutes after the scraper it cleans up.
2. Collect results: download the dataset as JSON, CSV or Excel; fetch the latest run's results from the API (GET https://api.apify.com/v2/actor-tasks/<task id>/runs/last/dataset/items?status=SUCCEEDED&format=csv, with your API token); let a webhook tell your system when a run succeeds; or connect it to Make, Zapier or n8n through Apify's integrations.

To run it right after another actor finishes instead, chain it with an integration (any actor, ours or not):

1. Open the other actor or its saved task, go to the Integrations tab and add Apify actor → Dataset Transformer, triggered when a run succeeds.
2. Set "datasetId": "{{resource.defaultDatasetId}}" in the input (Apify fills in the finished run's dataset) and add your filters, dedupe keys and fields as usual.

Every successful scrape then produces a cleaned copy, with no code. From the API it's the same input: pass the finished run's defaultDatasetId as datasetId.

Run it automatically after your scraper finishes

The worked example above, as an Actor-to-Actor integration on the scraper (or its saved task):

1. On the scraper's Integrations tab, add Dataset Transformer and set Start when to run succeeded.
2. In Actor input, check that Apify dataset says {{resource.defaultDatasetId}}; Apify replaces it with the dataset of the run that just finished. If the field is empty, the run still reads the finished run's dataset (from the integration's details).
3. Fill in the rest as in the example: the filters, placeId under Remove duplicates by, reviewsCount desc under Sort by, the fields, and Also save as a file: CSV. In JSON, the whole input is the example's input with "datasetId": "{{resource.defaultDatasetId}}".
4. Save, then test it on one of the scraper's past runs (the integration's test menu). Each successful scrape then leaves a deduplicated OUTPUT.csv in the Dataset Transformer run's key-value store.

The integration also sends a payload field with the finished run's details. A dataset named in Apify dataset wins; when that field is empty, Dataset Transformer reads payload.resource.defaultDatasetId instead.

Without the Integrations form, an integration is a webhook that starts a run. Create it with the Create webhook API on the ACTOR.RUN.SUCCEEDED event, with the input as the payload template (variables):

{
  "eventTypes": ["ACTOR.RUN.SUCCEEDED"],
  "condition": {"actorId": "<your scraper's actor id>"},
  "requestUrl": "https://api.apify.com/v2/acts/humble-echidna~dataset-transform/runs",
  "shouldInterpolateStrings": true,
  "payloadTemplate": "{\"datasetId\": \"{{resource.defaultDatasetId}}\", \"filters\": [\"totalScore >= 4.3\"], \"dedupeBy\": [\"placeId\"], \"exportFormat\": \"csv\"}"
}

Use "actorTaskId" in condition for a saved task. Apify adds your token to requests to its own API, and the Console lists this one as an HTTP webhook rather than an integration (details).

Use it from n8n

With n8n's Apify node (Apify's n8n guide):

1. Add an Apify node with the operation Run an Actor, pick humble-echidna/dataset-transform, paste your input JSON (for example the one above, with your scraper's dataset id), and turn on Wait for finish.
2. Add a second Apify node with the operation Get dataset items, and set Dataset ID to the defaultDatasetId from the first node's output.
3. Send the rows on: Google Sheets, a CRM, a database. For the CSV file instead, use Get key-value store record with the first node's defaultKeyValueStoreId and the key OUTPUT.csv.

Run an Actor and get dataset does steps 1 and 2 in one node. Don't replace step 1 with a fixed Wait node, and don't read the actor's "last run" instead of the run's own defaultDatasetId: a fixed wait breaks when a run takes longer than usual, and a "last run" read returns another run's rows when two runs overlap (a schedule and a manual run, or two workflows).

Can I use Dataset Transformer from an AI agent (MCP)?

Yes, through Apify's MCP server, from Claude, ChatGPT, Cursor or any other MCP client. Add this to your client's MCP configuration (or let the agent find it with the server's actor search); your client signs you in to Apify:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=humble-echidna/dataset-transform"
    }
  }
}

To use an Apify API token instead of signing in, add "headers": {"Authorization": "Bearer <APIFY_TOKEN>"} next to url.

An agent that has just run a scraper can pass its dataset, e.g. {"datasetId": "<dataset id>", "filters": ["price > 10"], "fields": ["title", "price", "url"]}, to get a smaller, cleaner result back. An agent holding JSON can pass the rows themselves to get a CSV: {"items": [{"name": "Ada", "city": "London"}, {"name": "Bo", "city": "Paris"}], "exportFormat": "csv"}.

Who it's for

Anyone who already runs scrapers on Apify (ours or anyone's) and hands the results to a spreadsheet, CRM or database: it's the scheduled clean-up step in between, so the export has only the rows and columns you want, with no duplicates, every time.

Why this one?

• One flat price, no usage bill. You pay per result at the price above and nothing else: no compute, proxy or storage charges on top, apart from the $0.00005 run-start fee, so you know what a run costs before it starts. Results you don't get aren't charged.
• Datasets and files alike. Your own Apify datasets, or CSV/TSV, JSON, JSON Lines and Excel (.xlsx) files from any public web address, gzip-compressed or not. Format, delimiter and text encoding (UTF-8, UTF-16, Windows-1252) are detected for you, and a broken character (half an emoji, from text cut mid-character) never fails the run.
• Pay only for the rows you keep: $1.00 per 1,000 output rows, and the CSV/JSONL file is included at no extra charge. Rows you filter out or that are duplicates cost nothing.
• Built for big inputs. Files stream to disk and are read one row at a time, so a 200 MB file doesn't need 200 MB of memory. Up to 1,000,000 rows per run.
• Nested data made simple. address.city or offers.0.price work in every option, with or without flattening.
• Tells you what happened. The run's status says how many rows were read, filtered out, duplicates and written, and warns about field names that no row has (the most common mistake is a typo in a field name).
• Polite and safe with files. It identifies itself honestly (User-Agent HumbleEchidnaApify), follows each site's robots.txt, and only requests public web addresses on the standard ports (80 and 443).

Limits

• One source per run: one dataset, one file, or one list of pasted rows. Pasted rows travel inside the run's input, so keep them to small data; for big data use a dataset or a file link.
• Files: up to 200 MB (after decompression), public web addresses on ports 80 and 443, CSV/TSV, JSON, JSON Lines, Excel (.xlsx, .xlsm).
• Excel: one sheet per run. Old .xls and password-protected workbooks aren't read (save them as .xlsx or CSV first). A workbook may unzip to at most 1 GB, with at most 150 MB of text.
• Up to 1,000,000 rows read per run; the status says so if a source has more.
• Sorting without a row limit holds at most 100 MB of matching rows; beyond that the run stops before writing anything and says so. Set Max rows per run (it then keeps only the top rows) or filter first.
• A single output row can be up to 5 MB; larger ones are skipped and counted.
• The CSV/JSONL file is saved up to 100 MB; beyond that it's skipped (the dataset still has every row).

FAQ

Which duplicate is kept?

The first one in the source's order, before sorting. Rows where every duplicate field is empty are always kept (so rows without an email aren't collapsed into one). Text that looks the same counts as the same: "é" saved as one character or as "e" plus an accent is one value. A field whose name has a comma goes in double quotes: "Revenue, USD".

Why does my JSON file fail with "set itemsPath"?

The file is one object with the rows inside it, like {"data": {"items": [...]}}. Put the path to the list, here data.items, in JSON: path to the list of rows. The error lists the paths it found.

Why are CSV values all text?

On purpose: leading zeros (zip codes, ids) survive. Filters and sorting still treat numbers as numbers. Excel cells keep their type (see the next question).

Can it convert Excel to JSON or CSV?

Yes. Upload an .xlsx file or paste its link in Or: file URL or upload; it's detected from the file itself, so a download link without .xlsx in it works too. You get one row per spreadsheet row, keyed by the sheet's first non-empty row (the header). When a title or notes sit above the table, put the number of the row with the column names in Excel: header row, as Excel numbers it, and the rows above are skipped. Numbers and TRUE/FALSE stay numbers and booleans, dates become 2025-12-31 (or 2025-12-31T14:30:00 with a time), and formulas give the value Excel last saved. Empty rows are skipped. Pick a sheet other than the first in Excel: sheet name, and choose Also save as a file: CSV for Excel to CSV. Filters, duplicates, sorting and renaming work as for any other source.

Can I use a file on my computer or a Google Sheet?

Yes. Click the upload button in Or: file URL or upload and pick the file; a path like C:\Users\... typed into the field can't be reached by the run. For a Google Sheet, paste its link as it is (the /edit link is fine) after setting Share > General access to Anyone with the link: the tab in the link is read, or the one you name in Excel: sheet name. Google Drive and Dropbox file links can't be read (their robots.txt disallows it), so upload those files instead.

Can I paste JSON instead of giving a link?

Yes. Put the rows in Or: paste rows (JSON array) (items in the API), as a list of objects: [{"name": "Ada", "city": "London"}, {"name": "Bo", "city": "Paris"}]. Filters, duplicates, sorting, renaming, flattening and the CSV or JSONL file work as for any other source, so this is JSON to CSV with no file to host. An element that isn't an object becomes {"value": ...}. An empty list counts as nothing pasted, so the file link is used.

Why does the status say a field wasn't found in any row?

Field names are case-sensitive and must match your data exactly; use dots for nested fields.

Can it read someone else's dataset?

Only datasets your own Apify account can access. The run reads the dataset you pick with read-only access.

How do I turn nested scraper JSON into flat CSV columns?

Turn on Flatten nested objects ("flatten": true): {"address": {"city": "Paris"}} becomes an address.city column, and a list becomes numbered columns (categories.0, categories.1). To keep only the nested values you need, name them in Fields to keep (and rename), flattened or not: location.lat -> lat,

offers.0.price -> price

How do I keep only Google Maps leads with a website, a phone and enough reviews?

Pick the Maps scraper's dataset in Apify dataset and filter on the scraper's own field names, for example:

website is not empty
phone is not empty
totalScore >= 4.5
reviewsCount >= 50
permanentlyClosed != true

Add placeId under Remove duplicates by (two searches over the same area return the same place twice) and reviewsCount desc under Sort by. Field names differ between scrapers, so check one row of your dataset first; the status warns about a field that no row has. The worked example shows the whole input and its CSV.

Why not just export the dataset as CSV from Apify?

Apify's export (and its dataset API) gives you every row: it can pick, omit or flatten fields and reverse the order, but it can't keep only the rows that match a condition, drop duplicates by a key, or sort by a field. Dataset Transformer does those in one step and writes a new dataset (plus an optional CSV file) that you export the usual way or pass to the next integration. You pay only for the rows it keeps.

Can it remove duplicates across several runs of my scraper?

Each run reads one source, so duplicates are removed within that dataset or file. To get only the rows that are new or changed since your scraper's previous run, use Dataset Diff first, then clean its output here.

Can it change values, split a column or combine two datasets?

No. Dataset Transformer keeps or drops rows and fields, sorts, renames and flattens, but doesn't rewrite values (no lowercasing, splitting or formulas). To add fields from another dataset or file matched by a key, use Join Datasets; to have an LLM classify or summarize each row, use AI for Every Row.

Is it legal to transform datasets and files with it?

It only processes data you choose: your own Apify datasets, and files at addresses you supply, fetched politely (robots.txt honoured, honest User-Agent). Make sure you're allowed to use the files you point it at. The example data is the US Treasury's Fiscal Data API, which is "offered free, without restriction" for commercial and non-commercial use.

Something that used to work now fails. Why?

The run log and the status say what went wrong. Please open an issue with the input you used.

Related actors

It works on any dataset; some of ours whose output it's handy for:

Feedback and support

Found a bug, or need an operation that isn't here? Open an issue on the Issues tab with the input you used.

Versions

Current version: 1.2. See the Changelog tab for what changed in each version.