Dataset Merge, Dedupe & Transform

Combine several Apify datasets and JSON / CSV / JSONL files into one clean dataset: merge, remove duplicates by the fields you choose, filter rows, pick / drop / rename fields — in one run, with no code.

What it does

1. Reads any mix of Apify dataset IDs (or names) and public http(s) URLs to JSON, JSONL or CSV files (CSV delimiter auto-detected: , ; tab |).
2. Filters rows with simple rules (eq, contains, gt, regex, in, exists, …), combined with AND or OR.
3. Dedupes by one or more fields (dot paths like address.city work), or by the whole item. Keeps the first or last occurrence. Values can be normalised (trim + lower-case) so Ana@Example.com = ana@example.com.
4. Transforms: pick fields, omit fields, rename fields (nested → flat supported), optionally add _source.
5. Writes the result to the run's default dataset (or a named dataset in your account, created if missing and appended to if it exists) and a SUMMARY record in the key-value store.

Typical uses: combine runs of the same scraper, dedupe leads across sources, clean a CSV export before import, trim a big dataset to the columns you need.

Input

Field	Type	Default	Notes
datasetIds	dataset picker (array)	–	Pick datasets from your account, or paste IDs / username~dataset-name. The run gets read-only access to exactly these datasets
sourceUrls	request list	–	Public JSON / JSONL / CSV URLs
csvDelimiter	string	auto	Force a CSV delimiter
dedupeMode	fields / fullItem / none	fields
dedupeFields	array	–	Required when mode = fields
keep	first / last	first	last holds unique rows in memory
normalizeDedupeValues	boolean	true	Trim + lower-case before comparing
filters	array of {field, op, value, caseSensitive}	–	Ops: eq, neq, contains, not_contains, starts_with, ends_with, gt, gte, lt, lte, exists, not_exists, empty, not_empty, regex, in, not_in
filterMode	all / any	all	AND / OR
pickFields / omitFields	array	–	Dot paths allowed
renameFields	object	–	e.g. {"address.city": "city"}
addSourceField	boolean	false	Adds _source
maxItems	integer	0 (no limit)	Stop after N output rows (reading stops too, so no extra scan charges; with keep: last all rows are read first)
outputDatasetName	string	–	Write to a named dataset instead of the default
maxSourceMb	integer	200	Per-URL download limit

Example:

{
  "datasetIds": ["crm-export"],
  "sourceUrls": [{"url": "https://example.com/leads.csv"}, {"url": "https://example.com/more.json"}],
  "dedupeFields": ["email"],
  "filters": [{"field": "age", "op": "gte", "value": 18}],
  "pickFields": ["name", "email", "age", "address.city"],
  "renameFields": {"address.city": "city"},
  "addSourceField": true
}

Output example

[
  {"name": "Ivy Tan", "email": "ivy@example.com", "age": 45, "city": "Montreal", "_source": "crm-export"},
  {"name": "Cleo Park", "email": "cleo@example.com", "age": 29, "city": "Lyon", "_source": "https://example.com/more.json"}
]

SUMMARY (key-value store): itemsRead, filteredOut, duplicatesRemoved, itemsOutput, sourceErrors, stoppedEarly, outputDataset, chargedEvents, and a per-source status list.

Why pay instead of the free dedup-datasets?

lukaskrivka/dedup-datasets is free (you pay only platform usage) and excellent for merging and deduping Apify datasets at very large scale. If that is all you need, use it. Pay for this Actor only if you need one of these:

• CSV, JSON and JSONL files from URLs, not just Apify datasets. Mix both in one run.
• No-code filtering and reshaping: 17 filter operators, plus pick, drop and rename of fields (dot paths work). The free Actor needs JavaScript transform functions for this.
• Forgiving match rules out of the box: trims spaces, ignores case and treats 1 and 1.0 as equal when comparing dedupe fields. The free Actor compares exact values (normalising needs your own JavaScript).
• Keep the last duplicate instead of the first (keep: last), e.g. to keep the newest version of each URL. The free Actor keeps the first.

Neither Actor does fuzzy (similarity) matching. The free Actor also has things this one does not: loading all runs of an Actor or Task, output to key-value store records, outputting only the duplicates, and an "only new items" filter dataset.

Pricing (pay per event)

Event	Price	When
Row output	$0.0004 per row ($0.40 per 1,000)	Each row written to the output dataset
Rows scanned	$0.001 per started 1,000 rows read	Covers reading large inputs that produce few rows
Actor start	$0.00005	Once per run (up to 1 GB memory; +$0.00005 per extra GB)

Cost examples

Exact charges at the prices above (Actor start $0.00005 per run at up to 1 GB memory; +$0.00005 per extra GB):

Job	Rows written	Started 1,000-row blocks read	Total
Scan 100,000 rows, write 10,000 unique	10,000 × $0.0004 = $4.00	100 × $0.001 = $0.10	$4.10005 (incl. $0.00005 start)
Scan 100,000 rows, write 100,000 (no duplicates)	100,000 × $0.0004 = $40.00	100 × $0.001 = $0.10	$40.10005
Merge 10,000 rows into 6,000 unique	$2.40	10 × $0.001 = $0.01	$2.41005
Scan 1,000,000 rows, keep 5,000	$2.00	1,000 × $0.001 = $1.00	$3.00005
Scan 250 rows, write 45	$0.018	1 × $0.001	$0.01905
Scan 250 rows, write all 250	$0.10	1 × $0.001	$0.10105

Run compute is included in these prices (no separate platform-usage charge). Set Max total charge on the run to cap spend; the Actor stops cleanly when the cap is reached.

Limits

• keep: first streams rows out and keeps only a 32-character hash per unique key in memory (roughly 100–150 MB per 1M unique keys).
• keep: last must hold every unique row in memory until the end; for millions of rows use keep: first or raise memory (max 4 GB).
• Remote files are downloaded whole (default limit 200 MB each); use more memory for very large files.
• Only http(s) URLs. Private URLs that need auth headers are not supported; load those into an Apify dataset first.
• CSV values stay strings; numeric filters (gt, lt, …) compare numeric strings as numbers.
• One failing source does not fail the run; it is listed in SUMMARY.sources with the reason (e.g. dataset not found or not accessible, HTTP Error 404). The run fails only if every source fails.