Merge, Dedupe & Transform Datasets: CSV, JSON, Excel
Pricing
from $0.40 / 1,000 result rows
Merge, join, dedupe, filter and reshape Apify datasets and CSV/JSON/Excel files. Export CSV, JSON, XLSX. $0.40 per 1,000 rows; dupes free.
You can access the Merge, Dedupe & Transform Datasets: CSV, JSON, Excel programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.
{ "openapi": "3.0.1", "info": { "version": "0.1", "x-build-id": "8zHdRmq1Tp8bMcErw" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/ataism~dataset-tools/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-ataism-dataset-tools", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/ataism~dataset-tools/runs": { "post": { "operationId": "runs-sync-ataism-dataset-tools", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/ataism~dataset-tools/run-sync": { "post": { "operationId": "run-sync-ataism-dataset-tools", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "datasetIds": { "title": "Apify datasets", "type": "array", "description": "Datasets in your own account. Pick them with the dataset picker: choosing a dataset gives this run read access to it, so the Actor works with Limited permissions and never asks for or stores an API token. You can also pass dataset IDs through the API. Datasets are read before file URLs; hidden fields (names starting with #) are dropped." }, "fileUrls": { "title": "File URLs (CSV, JSON, JSONL, XLSX)", "type": "array", "description": "Direct http(s) links to data files, one per line (max 200 MB each; 50 MB for .xlsx). Private, local and internal addresses are blocked. The format is detected from the content. A .json file may be an array of objects, one object, or one object per line. The default is a small public CSV (World Bank GDP by country and year, CC-BY 4.0).", "items": { "type": "string" } }, "files": { "title": "Uploaded files", "type": "array", "description": "Upload CSV, JSON, JSONL or XLSX files from your computer (the editor offers this), or paste a key-value store record URL. Each file is read once and not kept." }, "xlsxSheet": { "title": "Excel sheet", "type": "string", "description": "Sheet number (1 = first) or exact sheet name for .xlsx files. The first row of the sheet is the header.", "default": "1" }, "encoding": { "title": "Text encoding", "enum": [ "utf-8", "windows-1252", "utf-16le" ], "type": "string", "description": "Encoding of CSV, JSON and JSONL files. UTF-8 is the default; if accented letters look wrong (a warning is logged and listed in RUN_SUMMARY when a file has invalid UTF-8), use windows-1252 (Excel's usual CSV export in Western Europe) or utf-16le.", "default": "utf-8" }, "onSourceError": { "title": "If a source cannot be read", "enum": [ "fail", "skip" ], "type": "string", "description": "\"fail\" stops before any output (nothing is charged) and lists the problem. \"skip\" continues without that source and lists it in RUN_SUMMARY. Each file is fully read and checked before any output, so a damaged file is caught here and nothing is saved or charged for it. The run also stops after 5 failed sources or 500 MB downloaded for failed files.", "default": "fail" }, "mergeMode": { "title": "Merge mode", "enum": [ "concat", "join-inner", "join-left" ], "type": "string", "description": "\"concat\" appends all sources. A join needs exactly 2 sources: the first is the left side (streamed), the second the right side (held in memory, at most 1,000,000 rows). Matching right rows are added to each left row; a right field with the same name as a left field gets the suffix _right. If several right rows match, each gives one output row.", "default": "concat" }, "joinKey": { "title": "Join key (left field)", "type": "string", "description": "Field to match on, for example \"id\" or \"owner.id\". Rows with an empty key never match." }, "joinRightKey": { "title": "Join key on the right side", "type": "string", "description": "Only if the right source names the field differently. Empty = same as the join key." }, "sourceField": { "title": "Add source field", "type": "string", "description": "If set, each row gets this field holding where it came from (\"dataset:<id>\" or the file URL without its query string). Useful when concatenating." }, "dedupeFields": { "title": "Dedupe on fields", "type": "array", "description": "Rows with the same values in all these fields count as duplicates; only one is kept. Dot paths work for nested fields (\"owner.id\"). Use a single entry \"*\" to compare whole rows. Empty = no dedupe. Dedupe runs before filtering.", "items": { "type": "string" } }, "dedupeKeep": { "title": "Keep", "enum": [ "first", "last" ], "type": "string", "description": "Which row of a duplicate group is kept. \"last\" reads the sources twice and keeps one hash per distinct key in memory.", "default": "first" }, "normalizeKeys": { "title": "Normalise keys (case and whitespace)", "type": "boolean", "description": "Dedupe and join keys are compared after trimming, collapsing inner whitespace and lower-casing, so \"ACME Inc\" equals \"acme inc \". Off = exact match. Output values are never changed.", "default": false }, "filters": { "title": "Filters (all must match)", "type": "array", "description": "A JSON list of conditions, combined with AND: [{\"field\": \"price\", \"op\": \"gt\", \"value\": 10}]. ops: equals, notEquals, contains, notContains, regex, exists, notExists, gt, gte, lt, lte. gt/gte/lt/lte compare numbers (also numbers in CSV text) or else text order, which works for ISO dates. Optional \"ignoreCase\": true (not for regex). Regex runs on a linear-time engine: case-sensitive, at most 200 characters, no backreferences or lookahead." }, "flatten": { "title": "Flatten nested objects", "type": "boolean", "description": "Turns {\"owner\": {\"name\": \"A\"}} into {\"owner.name\": \"A\"}. Arrays stay as they are (CSV and Excel files show them as JSON text). Done after filtering, before select/rename.", "default": false }, "dropFields": { "title": "Drop fields", "type": "array", "description": "Fields to remove (after flattening, so use dot keys like \"owner.name\" when flatten is on).", "items": { "type": "string" } }, "selectFields": { "title": "Keep only these fields", "type": "array", "description": "If set, output rows hold only these fields, in this order (original names, applied before renaming).", "items": { "type": "string" } }, "renameFields": { "title": "Rename fields", "type": "object", "description": "{\"old name\": \"new name\"} pairs, applied last among the field operations." }, "onKeyCollision": { "title": "On field name clash", "enum": [ "skip", "overwrite" ], "type": "string", "description": "If flattening or renaming gives two fields of a row the same name: \"skip\" leaves that row out (counted in RUN_SUMMARY.keyCollisions, not charged); \"overwrite\" keeps the row and the later field wins.", "default": "skip" }, "sortField": { "title": "Sort by field", "type": "string", "description": "Output field name (after renaming). Sorting holds all rows in memory: at most 1,000,000 rows and about 40% of the run memory; above that the run stops with an error and charges nothing. Numbers sort numerically; empty values go last." }, "sortOrder": { "title": "Sort order", "enum": [ "asc", "desc" ], "type": "string", "description": "Ascending or descending.", "default": "asc" }, "maxOutputRows": { "title": "Max output rows", "minimum": 0, "type": "integer", "description": "Stop after this many output rows (applied after sorting). 0 = no limit. Each saved row costs $0.0004, so 20 rows cost $0.008. The prefilled 20 is for the quick start: set 0 to output everything.", "default": 0 }, "maxInputRows": { "title": "Max input rows", "minimum": 1, "maximum": 5000000, "type": "integer", "description": "Stop reading after this many input rows in total (hard maximum 5,000,000 per run). Input rows are free to read; this only bounds run time and memory.", "default": 5000000 }, "outputDataset": { "title": "Save rows to the dataset", "type": "boolean", "description": "Rows go to the run's default dataset. Turn off to get only a file.", "default": true }, "outputFile": { "title": "Also save a file", "enum": [ "none", "csv", "json", "jsonl", "xlsx" ], "type": "string", "description": "Writes all output rows to the run's key-value store as output.csv / .json / .jsonl / .xlsx and puts the link in RUN_SUMMARY. Rows are charged once, whether they are in the dataset, the file or both. Excel files use extra sheets after 1,000,000 rows.", "default": "none" }, "neutralizeFormulas": { "title": "Neutralise spreadsheet formulas in CSV/Excel files", "type": "boolean", "description": "Prefixes text cells that start with = + - @ with an apostrophe so Excel or Sheets cannot run them. Changes those values, so it is off by default.", "default": false }, "maxFileMb": { "title": "Max size per file (MB)", "minimum": 1, "maximum": 200, "type": "integer", "description": "Larger files are not read. Hard cap 200 MB (50 MB for .xlsx).", "default": 200 }, "downloadTimeoutSecs": { "title": "Download timeout (seconds)", "minimum": 10, "maximum": 1800, "type": "integer", "description": "Per file, including redirects.", "default": 300 }, "mode": { "title": "Mode", "type": "string", "description": "Internal: \"selfTest\" runs the maintainer health check and saves no items.", "default": "run" } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 }, "PROXY_UNBLOCKER_UNITS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 }, "PROXY_UNBLOCKER_UNITS": { "type": "integer", "example": 0 } } } } } } } } }}OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.
OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.
By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.
You can download the OpenAPI definitions for Merge, Dedupe & Transform Datasets: CSV, JSON, Excel from the options below:
If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.
You can also check out our other API clients: