Merge, Dedupe & Transform Datasets: Filter, JSON to CSV

Dataset Transformer cleans and reshapes tabular data in one step: filter rows, remove duplicates, flatten nested JSON, pick or rename fields, sort, and export CSV. Feed it one or several Apify dataset IDs (for example the output of other scrapers, merged into one), a pasted JSON array, or a public JSON / JSONL / CSV file URL. It runs with limited permissions and only reads the datasets you pick.

Store page: apify.com/chris_elder/dataset-transformer

✅ Pay per result: $0.50 per 1,000 output rows · 🆓 Dropped / filtered rows are never charged · 🔑 No API key

▶ Try it with one click · 📄 Real example output · 💲 Pricing · 🔌 API & integrations · ❓ FAQ · 🛟 Support

What does Dataset Transformer do?

• 🔗 Merge datasets: add up to 50 dataset IDs in datasetIds and get one combined output, optionally tagged with a sourceField so you can see where each row came from
• 🧹 Filter rows with 14 operators (eq, ne, gt, gte, lt, lte, contains, notContains, startsWith, endsWith, exists, notExists, in, regex) combined with AND or OR
• 🔁 Remove duplicates by one or more fields (dot paths like meta.city); the first occurrence is kept
• 📦 Flatten nested JSON into dotted keys ({ "meta": { "city": "London" } } → meta.city)
• ✏️ Pick, omit or rename fields
• ↕️ Sort ascending or descending by any field
• 📄 JSON to CSV – optional OUTPUT.csv in the Key-Value store; CSV files you load come back as JSON dataset rows (CSV to JSON)
• 🆓 Dropped rows free – only rows written to the output dataset are charged

Use cases

• Teams running several scrapers: merge the runs of different Actors (or yesterday's and today's run) into one deduplicated dataset
• Scraper users – clean and dedupe a noisy dataset before it goes to Google Sheets, Airtable or a warehouse
• Agents & n8n/Make – “filter + dedupe this run’s dataset” as one step after another Actor
• Developers – flatten nested JSON and convert JSON to CSV without writing a script
• Ops / QA – keep only rows that pass rules (e.g. price exists, rating ≥ 4) after a crawl
• Document pipelines – after a PDF to Markdown run, pass its datasetId, pick the url, title, wordCount and markdown fields and export a CSV

How to use it

1. Click Try for free – the prefill filters score ≥ 5, removes the duplicate Ada/London row, flattens and renames meta.city → city, sorts by score and exports CSV (2 output rows ≈ $0.0015 including the start fee).
2. To clean real data, replace items with datasetId (any dataset in your account, or a public one) or fileUrl (JSON array, JSONL or CSV).
3. Download the cleaned dataset (JSON, CSV, Excel from the Storage tab) or OUTPUT.csv from the run’s Key-Value store.

Input example

{
  "items": [
    { "id": 1, "name": "Ada", "meta": { "city": "London" }, "score": 9 },
    { "id": 2, "name": "Ada", "meta": { "city": "London" }, "score": 9 },
    { "id": 3, "name": "Bob", "meta": { "city": "Berlin" }, "score": 4 },
    { "id": 4, "name": "Cara", "meta": { "city": "Paris" }, "score": 7 }
  ],
  "filters": [{ "field": "score", "op": "gte", "value": 5 }],
  "dedupeBy": ["name", "meta.city"],
  "flatten": true,
  "rename": { "meta.city": "city" },
  "sortBy": "score",
  "sortDir": "desc",
  "exportCsv": true,
  "maxRows": 10
}

Other options: filterMode (and/or), pick, omit, csvKey.

Example: 4 rows in → 2 clean rows out

Step	Rows	What happened
Input	4	Ada (9), Ada (9, duplicate), Bob (4), Cara (7)
Filter score ≥ 5	3	Bob dropped (free)
Dedupe by name + meta.city	2	second Ada/London dropped (free)
Flatten + rename + sort desc	2	meta.city → city, highest score first

Output example (from a real public run)

Public run data (no login needed): dataset JSON · dataset CSV · OUTPUT.csv · RUN_SUMMARY

Dataset rows:

[
  { "id": 1, "name": "Ada", "score": 9, "city": "London" },
  { "id": 4, "name": "Cara", "score": 7, "city": "Paris" }
]

OUTPUT.csv:

id,name,score,city
1,Ada,9,London
4,Cara,7,Paris

RUN_SUMMARY:

{ "inputCount": 4, "outputCount": 2, "droppedFilter": 1, "droppedDedupe": 1, "skippedBudget": 0, "budgetHit": false, "exportCsv": true, "csvKey": "OUTPUT.csv", "source": "items" }

📊 Want these results in Google Sheets? Send them straight to a Sheet with Google Sheets Sync – run it after this Actor, or add it on this Actor's Integrations tab so every run exports automatically.

How much does it cost to clean a dataset?

Event	Free / Bronze	Silver	Gold+
Actor start (× memory GB, min 1)	$0.0005	$0.0005	$0.0005
Row output	$0.0005	$0.0004	$0.0003

Filtered, deduped and invalid rows are free. Only rows written to the output dataset are charged.

Job	Approx. cost
Prefill (2 output rows, 512 MB)	2 × $0.0005 + $0.0005 start = $0.0015
1,000 output rows	$0.50 + start
10,000 output rows (Gold)	10,000 × $0.0003 = $3.00 + start

Use Max cost per run to cap spend; the Actor stops writing rows when the cap is reached (budgetHit: true in RUN_SUMMARY).

Integrations (API, AI agents, Make, Zapier)

Clean another Actor’s output: pass that run’s defaultDatasetId as datasetId.

curl -X POST "https://api.apify.com/v2/acts/chris_elder~dataset-transformer/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"datasetId":"YOUR_DATASET_ID","dedupeBy":["url"],"flatten":true,"exportCsv":true}'

Python (pip install apify-client):

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("chris_elder/dataset-transformer").call(run_input={
    "datasetId": "YOUR_DATASET_ID",
    "filters": [{"field": "price", "op": "exists"}],
    "dedupeBy": ["url"],
    "exportCsv": True,
})
rows = client.dataset(run["defaultDatasetId"]).list_items().items

MCP (AI agents): https://mcp.apify.com/?tools=chris_elder/dataset-transformer

Make / Zapier / n8n: add the Apify Run Actor step after your scraper, map its dataset ID into datasetId, and read the cleaned rows. Works with Apify schedules and webhooks.

Limits and good to know

• Sources are merged in this order: pasted items, datasetId, each of datasetIds, then fileUrl. Up to 50 datasets and 250,000 input rows per run.
• If you add a dataset or file and leave the 4 sample rows in items, the sample rows (and the sample filter, dedupe, rename and sort) are ignored automatically.
• maxRows default 10,000 (max 100,000). For very large datasets, filter first or raise memory.
• CSV input is parsed with a header row and values arrive as text; gt/gte/lt/lte convert values to numbers, eq compares loosely ("9" equals 9).
• Excel (XLSX) files are not supported as input yet.

FAQ

How do I convert JSON to CSV? Load the JSON (pasted, dataset or file URL), turn on flatten for nested fields and exportCsv, then download OUTPUT.csv.

How do I remove duplicates from an Apify dataset? Set datasetId and list the fields that identify a duplicate in dedupeBy (e.g. ["url"]).

How do I merge two or more Apify datasets? Put their dataset IDs in datasetIds (the defaultDatasetId of each run), add dedupeBy (for example ["url"]) to drop rows that appear in more than one, and set sourceField (for example "source") if you want to know which dataset each row came from. Real run, 23 Oct 2026: 2 datasets with 2 and 30 rows, deduped by contentType, gave 3 output rows (29 duplicates dropped free), for about $0.002 including the start fee.

Does it modify my source dataset? No. It only reads, and writes a new dataset on this run.

Nested fields? Use dot paths (meta.city) and/or turn on flatten.

Why was I charged less than the input size? Filtered and deduped rows are free; you pay only for output rows.

Support, reliability and feature requests

• Dropped rows are free.
• Every release is tested on real inputs; the one-click example works.
• Issues answered within 2 business days via the Actor Issues tab.
• Breaking input or price changes are announced in this README first.

Built by GPT Shop Ltd with AI-assisted development; every release is reviewed and tested.

More tools from GPT Shop Ltd

Google Sheets Sync · Bulk File Downloader · HTTP Request Sender · Sitemap URL Extractor · PDF to Markdown · all tools

More use cases (FAQ)

• How do I convert an Apify dataset or JSON to CSV/Excel? Set exportCsv: true.
• How do I remove duplicate rows from scraped data? Use dedupeBy with one or more fields (nested paths such as meta.city work).
• Can I flatten nested JSON for Google Sheets? Yes. Set flatten: true, then rename columns with rename.

Done-for-you

Need this run for you or customised? Done-for-you data from £49: https://tools.gptshopltd.com/done-for-you-scraper/