Dataset Toolkit: Dedupe, Merge, Filter, Convert

Clean up scraped data in one run. Merge several Apify datasets and CSV, JSON, JSON Lines, Excel or Parquet files, remove duplicates (exact, case/whitespace-insensitive or fuzzy), filter with simple rules, flatten nested JSON, sort, pick and rename columns, and export to CSV, Excel (XLSX), JSON, JSONL or Parquet with download links. Large datasets are streamed, not loaded into memory.

Why this one

• Finishes the job. Every input is read independently: a missing dataset or a broken file gets a clear error in the run summary while everything else is still processed. Data is staged on disk, so a million rows don't run out of memory.
• Dedupe that matches how data really looks. Exact, normalized (Ann@X.com = ann@x.com) or fuzzy (Acme Corporation Ltd = ACME Corporaton Ltd.), on any fields or the whole row. Keep the first, the last or the most complete row, and optionally count how many duplicates each row had.
• Filter without code. price >= 10 and country in ["CA", "US"] and not email empty. Parsed safely, never executed.
• Any format in, any format out. CSV/TSV (delimiter and encoding detected), JSON arrays, JSON Lines, XLSX, Parquet, gzip. Out: the run's dataset plus CSV, XLSX, JSON, JSONL and Parquet files.
• Pay per 1,000 rows, not per row.

Input

{
  "datasetIds": ["aBcDeFgHiJkLmNoPq", "nXyZ0123456789abc"],
  "fileUrls": ["https://example.com/leads.csv"],
  "sourceField": "_source",
  "flatten": "objects",
  "filter": "country in [\"CA\", \"US\"] and not email empty",
  "dedupe": "normalized",
  "dedupeFields": ["email"],
  "keep": "most-complete",
  "duplicateCountField": "duplicates",
  "sortBy": ["-rating", "name"],
  "fields": ["name", "email", "phone", "address.city", "rating", "duplicates"],
  "rename": {"address.city": "city"},
  "exportFormats": ["csv", "xlsx"]
}

Processing order: read and merge (inline items, then datasets, then files) → flatten → add the source column → filter → dedupe → sort → limit → select / omit / rename → output. So filter, dedupe and sort use the original (flattened) field names, and rename comes last.

Field	What it does
datasetIds	Apify dataset IDs (or username~dataset-name). Yours, shared with you, or public.
fileUrls	Public file URLs: CSV, TSV, JSON, JSONL/NDJSON, XLSX (first sheet), Parquet, optionally .gz.
items	Rows inline, as a JSON array (handy for agents).
fileFormat, csvDelimiter	Override detection.
sourceField	Column naming each row's input (inline, dataset:<id> or the URL).
flatten, flattenSeparator	objects: {"address": {"city": "X"}} → address.city. all: arrays too (tags.0).
filter	Rule language below.
dedupe, dedupeFields, fuzzyThreshold, keep, duplicateCountField	Duplicate removal.
sortBy, limit	-price or price:desc for descending; :number / :text force a type.
fields, omitFields, rename	Output columns.
exportFormats, exportFileName	Files to write (output.csv, ...).
datasetOutput	rows (default), summary (one item with counts and links) or none.

Filter language

price >= 10 and (country = "CA" or country in ["US", "MX"])
title matches "(?i)^(senior|lead)" and not email empty
tags contains "vip" or `Company Name` startswith "acme"
address.city exists

• Comparisons: = != < <= > >=. With a number (price > 10) the field is compared numerically, also when it is text like "12.50". With text (date >= "2024-01-01") it's compared as text, so ISO dates work.
• contains (substring, or list membership), startswith, endswith: ignore case.
• matches: Python regular expression; add (?i) to ignore case.
• in [..], exists (present and not null), empty (missing, null, blank text, empty list/object).
• Combine with and, or, not (or &&, ||, !) and parentheses. Keywords are case-insensitive.
• A condition on a missing field is false, except !=, which is true.
• Field names with spaces or that are keywords go in backticks. Nested fields: address.city, items.0.sku.

Dedupe modes

• exact: values must be identical (1 and "1" differ).
• normalized: ignores letter case, repeated or surrounding whitespace and Unicode width (1 = "1").
• fuzzy: normalized text compared with RapidFuzz token sort ratio; a row joins the first earlier group scoring at least fuzzyThreshold (default 90). Catches typos, punctuation and word order. To stay fast on large data, rows are only compared with groups whose text starts with the same character, so a typo in the very first letter is not caught.

With several dedupeFields, their values are combined into one key. Without dedupeFields, the whole row is compared (ignoring the source column). Rows whose dedupe fields are all empty are never removed.

Output

• Dataset: the final rows (or one summary item with datasetOutput: "summary").
• Key-value store:
  • SUMMARY: counts and per-input status, for example:

    {"rowsRead": 12000, "rowsFiltered": 1500, "duplicatesRemoved": 830, "rowsCutByLimit": 0, "rowsOut": 9670,
     "sources": [{"source": "dataset:aBcDeFgHiJkLmNoPq", "kind": "dataset", "status": "ok", "rowsRead": 10000},
                 {"source": "https://example.com/leads.csv", "kind": "url", "status": "error", "rowsRead": 0,
                  "error": {"code": "HTTP_ERROR", "message": "The server answered HTTP 403 ..."}}],
     "exports": [{"format": "csv", "key": "output.csv", "status": "ok", "rows": 9670, "bytes": 1203311,
                  "url": "https://api.apify.com/v2/key-value-stores/.../records/output.csv"}],
     "stoppedReason": null, "charged": {"rows-processed-1000": 12, "file-exported": 1}}
    

  • The export files. CSV, XLSX and Parquet have every column seen, in first-seen order; nested values are written as JSON text. Parquet column types are inferred (integer, float, boolean, otherwise text).

Input statuses: ok, partial (rows before the error are kept), error, skipped (spending limit). Error codes: DATASET_NOT_FOUND, ACCESS_DENIED, DATASET_READ_FAILED, HTTP_ERROR, DOWNLOAD_FAILED, BLOCKED_ADDRESS, INVALID_URL, FILE_TOO_LARGE, PARSE_ERROR, UNSUPPORTED_FORMAT, SPENDING_LIMIT_REACHED; exports: EXPORT_FAILED, TOO_MANY_COLUMNS. The run fails only if the input itself is invalid (e.g. no sources); if every input fails, the run still succeeds and its status message says why. NaN/Infinity numbers are output as null, and a row over Apify's 9 MB item limit is skipped from the dataset (counted in rowsNotPushed) but kept in export files.

Pricing (pay per event)

Event	Price	When
rows-processed-1000	$0.002	Each started block of 1,000 input rows read (rows later filtered out or removed as duplicates count)
file-exported	$0.005	Each export file written
apify-actor-start	$0.001 per started GB of run memory (from 2026-10-30)	Once per run, charged by Apify. The default 256 MB run counts as 1; Apify covers the first 5 seconds of compute in return

Example: dedupe 50,000 rows from two datasets and export a CSV and an XLSX = 50 × $0.002 + 2 × $0.005 = $0.11. Inputs that fail to load and failed exports are not charged. With a maximum charge per run, the Actor checks the budget before each block of 1,000 rows (keeping enough back for the exports you asked for), stops reading when it runs out and still delivers what it read, with stoppedReason: "SPENDING_LIMIT_REACHED".

Limits

• Rows are staged on the run's local disk, so the row count is limited by disk, not memory. Measured: about 15,000 to 25,000 rows/s; 1,000,000 rows with dedupe, sort and CSV + XLSX + Parquet exports in about 70 s with ~300 MB peak memory at 1 GB.
• Memory: the default is 256 MB, which suits the typical job (up to a few hundred thousand rows; measured: 860,000 rows with normalized dedupe, sort and a CSV export finished at 256 MB in about 10 minutes, at 512 MB in 5 minutes, for the same compute). For big jobs raise Run options > Memory: 1 GB for fuzzy dedupe or XLSX export of large inputs, 2 to 4 GB for multi-million-row runs (disk scales with memory). Compute per row costs about the same at any size; more memory mostly makes it faster. If memory gets nearly full, reading stops before the run could crash, the rows read so far are delivered and the summary says stoppedReason: "MEMORY_LOW". Below 1 GB the fuzzy group cap is reduced in proportion (minimum 50,000).
• Files: up to 1 GB each (after download). JSON files whose top level is an object (not an array) are loaded whole and limited to 1/16 of the run's memory (16 MB at 256 MB, up to 100 MB); use JSON arrays or JSON Lines for bigger files.
• Fuzzy dedupe keeps up to 300,000 groups in memory (at 1 GB or more) and compares each row with up to 20,000 groups that start with the same character; beyond that, later rows are matched only against known groups (the summary warns). Fuzzy runs at about 10,000 rows/s on typical data. For millions of rows prefer normalized.
• XLSX holds at most 1,048,575 rows and 16,384 columns (the file stops there; other formats have everything). CSV, XLSX and Parquet exports allow up to 10,000 columns. Cells longer than 32,767 characters are cut in XLSX.
• Only the first sheet of an XLSX workbook is read; old .xls is not supported. Datasets are read with hidden (#-prefixed) fields and empty items skipped.
• At most 100 inputs per run. File URLs must be public (private and reserved network addresses are refused).

Use from an AI agent (MCP)

Through the Apify MCP server, an agent can pass rows as items or point at datasets from other Actor runs, and set datasetOutput: "summary" to get counts and download links instead of every row.

Privacy

Your data is processed only inside your run and stored only in your run's storages.

Open-source licenses

Built on permissively licensed software: Apify SDK (Apache-2.0), HTTPX (BSD-3-Clause), RapidFuzz (MIT), Apache Arrow / pyarrow (Apache-2.0), openpyxl (MIT) and SQLite (public domain).