# Dataset Toolkit: Dedupe, Merge, Filter, Convert (`tinplatelabs/dataset-toolkit`) Actor Merge Apify datasets and CSV, JSON, JSONL, XLSX or Parquet files, remove duplicates (exact, normalized or fuzzy), filter with simple rules, flatten, sort, select and rename fields, then export to CSV, Excel, JSON or Parquet. Streams large data. Priced per 1,000 rows. - **URL**: https://apify.com/tinplatelabs/dataset-toolkit.md - **Developed by:** [tinplatelabs](https://apify.com/tinplatelabs) (community) - **Categories:** Developer tools, Automation, AI - **Stats:** 6 total users, 5 monthly users, 99.2% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Dataset Toolkit: Dedupe, Merge, Filter, Convert Clean up scraped data in one run. Merge several **Apify datasets** and **CSV, JSON, JSON Lines, Excel or Parquet files**, **remove duplicates** (exact, case/whitespace-insensitive or **fuzzy**), **filter** with simple rules, **flatten** nested JSON, **sort**, pick and rename columns, and **export** to CSV, Excel (XLSX), JSON, JSONL or Parquet with download links. Large datasets are streamed, not loaded into memory. ### Why this one - **Finishes the job.** Every input is read independently: a missing dataset or a broken file gets a clear error in the run summary while everything else is still processed. Data is staged on disk, so a million rows don't run out of memory. - **Dedupe that matches how data really looks.** Exact, normalized (`Ann@X.com ` = `ann@x.com`) or fuzzy (`Acme Corporation Ltd` = `ACME Corporaton Ltd.`), on any fields or the whole row. Keep the first, the last or the **most complete** row, and optionally count how many duplicates each row had. - **Filter without code.** `price >= 10 and country in ["CA", "US"] and not email empty`. Parsed safely, never executed. - **Any format in, any format out.** CSV/TSV (delimiter and encoding detected), JSON arrays, JSON Lines, XLSX, Parquet, gzip. Out: the run's dataset plus CSV, XLSX, JSON, JSONL and Parquet files. - **Pay per 1,000 rows**, not per row. ### Input ```json { "datasetIds": ["aBcDeFgHiJkLmNoPq", "nXyZ0123456789abc"], "fileUrls": ["https://example.com/leads.csv"], "sourceField": "_source", "flatten": "objects", "filter": "country in [\"CA\", \"US\"] and not email empty", "dedupe": "normalized", "dedupeFields": ["email"], "keep": "most-complete", "duplicateCountField": "duplicates", "sortBy": ["-rating", "name"], "fields": ["name", "email", "phone", "address.city", "rating", "duplicates"], "rename": {"address.city": "city"}, "exportFormats": ["csv", "xlsx"] } ``` Processing order: **read and merge** (inline `items`, then datasets, then files) → **flatten** → add the source column → **filter** → **dedupe** → **sort** → **limit** → **select / omit / rename** → output. So filter, dedupe and sort use the original (flattened) field names, and `rename` comes last. | Field | What it does | |---|---| | `datasetIds` | Apify dataset IDs (or `username~dataset-name`). Yours, shared with you, or public. | | `fileUrls` | Public file URLs: CSV, TSV, JSON, JSONL/NDJSON, XLSX (first sheet), Parquet, optionally `.gz`. | | `items` | Rows inline, as a JSON array (handy for agents). | | `fileFormat`, `csvDelimiter` | Override detection. | | `sourceField` | Column naming each row's input (`inline`, `dataset:` or the URL). | | `flatten`, `flattenSeparator` | `objects`: `{"address": {"city": "X"}}` → `address.city`. `all`: arrays too (`tags.0`). | | `filter` | Rule language below. | | `dedupe`, `dedupeFields`, `fuzzyThreshold`, `keep`, `duplicateCountField` | Duplicate removal. | | `sortBy`, `limit` | `-price` or `price:desc` for descending; `:number` / `:text` force a type. | | `fields`, `omitFields`, `rename` | Output columns. | | `exportFormats`, `exportFileName` | Files to write (`output.csv`, ...). | | `datasetOutput` | `rows` (default), `summary` (one item with counts and links) or `none`. | #### Filter language ``` price >= 10 and (country = "CA" or country in ["US", "MX"]) title matches "(?i)^(senior|lead)" and not email empty tags contains "vip" or `Company Name` startswith "acme" address.city exists ``` - Comparisons: `=` `!=` `<` `<=` `>` `>=`. With a number (`price > 10`) the field is compared numerically, also when it is text like `"12.50"`. With text (`date >= "2024-01-01"`) it's compared as text, so ISO dates work. - `contains` (substring, or list membership), `startswith`, `endswith`: ignore case. - `matches`: Python regular expression; add `(?i)` to ignore case. - `in [..]`, `exists` (present and not null), `empty` (missing, null, blank text, empty list/object). - Combine with `and`, `or`, `not` (or `&&`, `||`, `!`) and parentheses. Keywords are case-insensitive. - A condition on a missing field is false, except `!=`, which is true. - Field names with spaces or that are keywords go in `backticks`. Nested fields: `address.city`, `items.0.sku`. #### Dedupe modes - **exact**: values must be identical (`1` and `"1"` differ). - **normalized**: ignores letter case, repeated or surrounding whitespace and Unicode width (`1` = `"1"`). - **fuzzy**: normalized text compared with [RapidFuzz](https://github.com/rapidfuzz/RapidFuzz) token sort ratio; a row joins the first earlier group scoring at least `fuzzyThreshold` (default 90). Catches typos, punctuation and word order. To stay fast on large data, rows are only compared with groups whose text starts with the same character, so a typo in the very first letter is not caught. With several `dedupeFields`, their values are combined into one key. Without `dedupeFields`, the whole row is compared (ignoring the source column). Rows whose dedupe fields are all empty are never removed. ### Output - **Dataset**: the final rows (or one summary item with `datasetOutput: "summary"`). - **Key-value store**: - `SUMMARY`: counts and per-input status, for example: ```json {"rowsRead": 12000, "rowsFiltered": 1500, "duplicatesRemoved": 830, "rowsCutByLimit": 0, "rowsOut": 9670, "sources": [{"source": "dataset:aBcDeFgHiJkLmNoPq", "kind": "dataset", "status": "ok", "rowsRead": 10000}, {"source": "https://example.com/leads.csv", "kind": "url", "status": "error", "rowsRead": 0, "error": {"code": "HTTP_ERROR", "message": "The server answered HTTP 403 ..."}}], "exports": [{"format": "csv", "key": "output.csv", "status": "ok", "rows": 9670, "bytes": 1203311, "url": "https://api.apify.com/v2/key-value-stores/.../records/output.csv"}], "stoppedReason": null, "charged": {"rows-processed-1000": 12, "file-exported": 1}} ``` - The export files. CSV, XLSX and Parquet have every column seen, in first-seen order; nested values are written as JSON text. Parquet column types are inferred (integer, float, boolean, otherwise text). Input statuses: `ok`, `partial` (rows before the error are kept), `error`, `skipped` (spending limit). Error codes: `DATASET_NOT_FOUND`, `ACCESS_DENIED`, `DATASET_READ_FAILED`, `HTTP_ERROR`, `DOWNLOAD_FAILED`, `BLOCKED_ADDRESS`, `INVALID_URL`, `FILE_TOO_LARGE`, `PARSE_ERROR`, `UNSUPPORTED_FORMAT`, `SPENDING_LIMIT_REACHED`; exports: `EXPORT_FAILED`, `TOO_MANY_COLUMNS`. The run fails only if the input itself is invalid (e.g. no sources); if every input fails, the run still succeeds and its status message says why. NaN/Infinity numbers are output as null, and a row over Apify's 9 MB item limit is skipped from the dataset (counted in `rowsNotPushed`) but kept in export files. ### Pricing (pay per event) | Event | Price | When | |---|---|---| | `rows-processed-1000` | $0.002 | Each started block of 1,000 input rows read (rows later filtered out or removed as duplicates count) | | `file-exported` | $0.005 | Each export file written | | `apify-actor-start` | $0.001 per started GB of run memory (from 2026-10-30) | Once per run, charged by Apify. The default 256 MB run counts as 1; Apify covers the first 5 seconds of compute in return | Example: dedupe 50,000 rows from two datasets and export a CSV and an XLSX = 50 × $0.002 + 2 × $0.005 = **$0.11**. Inputs that fail to load and failed exports are not charged. With a maximum charge per run, the Actor checks the budget before each block of 1,000 rows (keeping enough back for the exports you asked for), stops reading when it runs out and still delivers what it read, with `stoppedReason: "SPENDING_LIMIT_REACHED"`. ### Limits - Rows are staged on the run's local disk, so the row count is limited by disk, not memory. Measured: about 15,000 to 25,000 rows/s; 1,000,000 rows with dedupe, sort and CSV + XLSX + Parquet exports in about 70 s with ~300 MB peak memory at 1 GB. - **Memory:** the default is 256 MB, which suits the typical job (up to a few hundred thousand rows; measured: 860,000 rows with normalized dedupe, sort and a CSV export finished at 256 MB in about 10 minutes, at 512 MB in 5 minutes, for the same compute). For big jobs raise **Run options > Memory**: 1 GB for fuzzy dedupe or XLSX export of large inputs, 2 to 4 GB for multi-million-row runs (disk scales with memory). Compute per row costs about the same at any size; more memory mostly makes it faster. If memory gets nearly full, reading stops before the run could crash, the rows read so far are delivered and the summary says `stoppedReason: "MEMORY_LOW"`. Below 1 GB the fuzzy group cap is reduced in proportion (minimum 50,000). - Files: up to 1 GB each (after download). JSON files whose top level is an object (not an array) are loaded whole and limited to 1/16 of the run's memory (16 MB at 256 MB, up to 100 MB); use JSON arrays or JSON Lines for bigger files. - Fuzzy dedupe keeps up to 300,000 groups in memory (at 1 GB or more) and compares each row with up to 20,000 groups that start with the same character; beyond that, later rows are matched only against known groups (the summary warns). Fuzzy runs at about 10,000 rows/s on typical data. For millions of rows prefer `normalized`. - XLSX holds at most 1,048,575 rows and 16,384 columns (the file stops there; other formats have everything). CSV, XLSX and Parquet exports allow up to 10,000 columns. Cells longer than 32,767 characters are cut in XLSX. - Only the first sheet of an XLSX workbook is read; old `.xls` is not supported. Datasets are read with hidden (`#`-prefixed) fields and empty items skipped. - At most 100 inputs per run. File URLs must be public (private and reserved network addresses are refused). ### Use from an AI agent (MCP) Through the [Apify MCP server](https://mcp.apify.com), an agent can pass rows as `items` or point at datasets from other Actor runs, and set `datasetOutput: "summary"` to get counts and download links instead of every row. ### Privacy Your data is processed only inside your run and stored only in your run's storages. ### Open-source licenses Built on permissively licensed software: [Apify SDK](https://github.com/apify/apify-sdk-python) (Apache-2.0), [HTTPX](https://github.com/encode/httpx) (BSD-3-Clause), [RapidFuzz](https://github.com/rapidfuzz/RapidFuzz) (MIT), [Apache Arrow / pyarrow](https://arrow.apache.org/) (Apache-2.0), [openpyxl](https://openpyxl.readthedocs.io/) (MIT) and SQLite (public domain). # Actor input Schema ## `datasetIds` (type: `array`): Datasets to read, e.g. the output of a scraper run: pick them or paste dataset IDs (or username~dataset-name). Another user's dataset must be shared with you or public. Read page by page, so size is not limited by memory. ## `fileUrls` (type: `array`): Public URLs of CSV, TSV, JSON (array), JSON Lines, XLSX (first sheet) or Parquet files, optionally gzip-compressed. Format is detected from the extension, Content-Type or content. Up to 1 GB each. ## `items` (type: `array`): Rows given directly as a JSON array of objects, e.g. from an AI agent or an API call. Read before datasets and files. ## `fileFormat` (type: `string`): Force the format of every file URL. Auto detects it per file. ## `csvDelimiter` (type: `string`): Delimiter of CSV files. Auto picks the most frequent of , ; tab | in the header line. ## `sourceField` (type: `string`): Name of a column recording where each row came from (inline, dataset:\ or the file URL), e.g. \_source. Empty = no column. Ignored by whole-row dedupe. ## `flatten` (type: `string`): Turn nested JSON into columns before filtering: objects makes address.city columns; all also splits arrays (tags.0). Without flattening, nested values become JSON text in CSV/XLSX. ## `flattenSeparator` (type: `string`): Joins nested names when flattening, e.g. . or /. ## `filter` (type: `string`): Keep only rows matching this rule, e.g. price >= 10 and country in \["CA", "US"] and not email empty. Operators: = != < <= > >= contains startswith endswith matches (regex) in exists empty, combined with and / or / not and parentheses. Nested fields: address.city; names with spaces in `backticks`. See the README. ## `dedupe` (type: `string`): exact: identical values. normalized: ignores case, extra spaces and Unicode width. fuzzy: near matches such as typos (rapidfuzz similarity at least the threshold). ## `dedupeFields` (type: `array`): Fields that identify a duplicate, e.g. email or name + city. Empty = the whole row. Rows with all these fields empty are always kept. ## `fuzzyThreshold` (type: `integer`): Fuzzy mode: similarity (0-100, token sort ratio) needed to count as a duplicate. 90 catches typos and word order; lower values merge more aggressively. ## `keep` (type: `string`): Keep the first row seen, the last one, or the one with the most non-empty fields. ## `duplicateCountField` (type: `string`): Optional column name that receives how many rows were in each kept row's duplicate group, e.g. duplicates. ## `sortBy` (type: `array`): Fields to sort by, in order. Prefix with - or add :desc for descending; add :number or :text to force a type. Numbers sort numerically (also in text), text ignores case, missing values go last. You can also sort by the duplicate count column. ## `limit` (type: `integer`): Keep at most this many rows after filtering, dedupe and sorting. 0 = all. ## `fields` (type: `array`): Output exactly these columns, in this order (nested paths like address.city allowed). Empty = all fields. ## `omitFields` (type: `array`): Columns to drop (when 'Keep only these fields' is empty). ## `rename` (type: `object`): Old name to new name, e.g. {"Email Address": "email"}. Applied last. ## `exportFormats` (type: `array`): Files written to the run's key-value store, with download links in the SUMMARY record. CSV, XLSX and Parquet use every column seen; nested values become JSON text. ## `exportFileName` (type: `string`): Base name of the export files (extension added). ## `datasetOutput` (type: `string`): rows: the processed rows go to the run's dataset. summary: one item with counts and export links (handy for agents and huge files). none: only the files and the SUMMARY record. ## Actor input object example ```json { "fileUrls": [ "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv" ], "fileFormat": "auto", "csvDelimiter": "auto", "flatten": "none", "flattenSeparator": ".", "dedupe": "none", "fuzzyThreshold": 90, "keep": "first", "limit": 0, "exportFormats": [ "csv" ], "exportFileName": "output", "datasetOutput": "rows" } ``` # Actor output Schema ## `rows` (type: `string`): No description ## `summary` (type: `string`): No description ## `files` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "fileUrls": [ "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv" ], "exportFormats": [ "csv" ] }; // Run the Actor and wait for it to finish const run = await client.actor("tinplatelabs/dataset-toolkit").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "fileUrls": ["https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv"], "exportFormats": ["csv"], } # Run the Actor and wait for it to finish run = client.actor("tinplatelabs/dataset-toolkit").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "fileUrls": [ "https://raw.githubusercontent.com/datasets/country-codes/main/data/country-codes.csv" ], "exportFormats": [ "csv" ] }' | apify call tinplatelabs/dataset-toolkit --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,tinplatelabs/dataset-toolkit" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/LcchOpI2hSqQAWdRc/builds/4zsjVfWU8vmqEhyp0/openapi.json