Dataset Merge & Dedup Datasets — Filter, Join CSV & Excel
Pricing
from $0.50 / 1,000 output rows
Merge Apify datasets, CSV, JSON and Excel files or Google Sheets into one table. Clean fields, normalise URLs, extract domains, filter, remove duplicates and join (VLOOKUP) in one run. Works with limited permissions. CSV and Excel exports free. $0.50 per 1,000 output rows.
You can access the Dataset Merge & Dedup Datasets — Filter, Join CSV & Excel programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.
{ "openapi": "3.0.1", "info": { "version": "0.0", "x-build-id": "pglEz0WAxFfN5dVjs" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/saliga~dataset-merge-dedupe/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-saliga-dataset-merge-dedupe", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/saliga~dataset-merge-dedupe/runs": { "post": { "operationId": "runs-sync-saliga-dataset-merge-dedupe", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/saliga~dataset-merge-dedupe/run-sync": { "post": { "operationId": "run-sync-saliga-dataset-merge-dedupe", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "datasetIds": { "title": "Apify datasets", "type": "array", "description": "Datasets to merge, for example the output of other Actor runs. Pick them here (or paste their IDs) so the run gets permission to read them, also when this Actor runs with limited permissions. A named dataset can also be given as username~dataset-name. Datasets are read in pages of 5,000 rows." }, "fileUrls": { "title": "File links", "type": "array", "description": "Public links to CSV, TSV, JSON, JSONL / NDJSON or Excel (.xlsx) files, up to 100 MB each. Google Sheets links work when the sheet is shared as \"Anyone with the link\" (the tab in the link is used); Google Drive and Dropbox share links work too.", "items": { "type": "string" } }, "rows": { "title": "Rows (JSON)", "type": "array", "description": "Rows given directly as a JSON array of objects. Handy for tests and AI agents. The example rows are fictional. When you also give datasets or file links, they are ignored, and so is every example setting of this form still at its example value (clear the example rows to keep those settings)." }, "fileFormat": { "title": "File format", "enum": [ "auto", "csv", "tsv", "json", "jsonl", "xlsx" ], "type": "string", "description": "Format of the file links. Detected from the link, the server's content type and the first bytes when left on automatic.", "default": "auto" }, "excelSheet": { "title": "Excel sheet name", "type": "string", "description": "Sheet to read in Excel files. Empty: the first sheet." }, "addSourceColumn": { "title": "Add a _source column", "type": "boolean", "description": "Tag each row with where it came from: dataset:<name or ID>, the file link, or rows.", "default": false }, "fields": { "title": "Keep only these fields", "type": "array", "description": "Empty: keep all fields. Names after renaming; company.website pulls a nested value into its own column. The domain and _source columns are kept automatically when you create them.", "items": { "type": "string" } }, "renameFields": { "title": "Rename fields", "type": "object", "description": "Old name -> new name, e.g. {\"Company Name\": \"company\", \"URL\": \"website\"}. Use it to line up columns from different sources before merging." }, "trimStrings": { "title": "Trim spaces", "type": "boolean", "description": "Remove spaces at the start and end of every text value.", "default": true }, "lowercaseFields": { "title": "Lowercase fields", "type": "array", "description": "Text in these fields becomes lowercase, e.g. email domains or country codes.", "items": { "type": "string" } }, "normalizeUrlFields": { "title": "Normalise URL fields", "type": "array", "description": "URLs in these fields get a lowercase host, no utm_* or click-ID parameters (gclid, fbclid, msclkid ...), no #fragment and no trailing slash; https:// is added to bare domains. Good before deduplicating by URL.", "items": { "type": "string" } }, "extractDomainFrom": { "title": "Extract domain from", "type": "string", "description": "A URL or website field. Adds a domain column (www. removed, lowercase): https://www.Example.com/about -> example.com. Dedupe or join on it." }, "filters": { "title": "Filter conditions", "type": "array", "description": "List of {\"field\", \"op\", \"value\"}. Ops: eq, neq, contains, not_contains, regex, gt, gte, lt, lte, empty, not_empty, in (value is a list). Text comparisons ignore case unless the condition has \"caseSensitive\": true. gt/gte/lt/lte compare numbers when both sides are numbers (1,200 counts), text otherwise (ISO dates work). The example keeps companies with 10 or more employees; clear it for your own data." }, "filterMode": { "title": "Rows must match", "enum": [ "all", "any" ], "type": "string", "description": "How the conditions combine.", "default": "all" }, "dedupe": { "title": "Remove duplicates", "type": "boolean", "description": "Remove duplicate rows. With no dedupe fields, rows count as duplicates only when every field is equal (the _source column is ignored).", "default": false }, "dedupeFields": { "title": "Dedupe by fields", "type": "array", "description": "Rows with the same values in these fields are duplicates, e.g. domain, or name + city. Rows where all of these fields are empty are never removed.", "items": { "type": "string" } }, "keep": { "title": "Keep", "enum": [ "first", "last" ], "type": "string", "description": "Which row of each duplicate group stays. Keeping the last row holds all rows in memory.", "default": "first" }, "dedupeIgnoreCase": { "title": "Ignore case and extra spaces", "type": "boolean", "description": "\"Acme Inc\" and \" acme inc\" count as the same value.", "default": true }, "outputDuplicates": { "title": "Save removed duplicates", "type": "boolean", "description": "Write the removed duplicates to a separate dataset of this run (linked in the OUTPUT record) instead of dropping them. Free.", "default": false }, "joinDatasetId": { "title": "Join with dataset", "type": "string", "description": "A dataset to look values up in." }, "joinFileUrl": { "title": "Join with file link", "type": "string", "description": "Public CSV, TSV, JSON, JSONL or Excel link, or a Google Sheet." }, "joinRows": { "title": "Join with rows (JSON)", "type": "array", "description": "A JSON array of objects to look values up in." }, "joinOn": { "title": "Match your field", "type": "string", "description": "Field of your rows to match on, e.g. domain." }, "joinRightOn": { "title": "...with this field of the join table", "type": "string", "description": "Empty: the same name as above." }, "joinType": { "title": "Rows without a match", "enum": [ "left", "inner" ], "type": "string", "description": "Left join keeps every row; inner join keeps only rows with a match. Only rows written to the output are charged.", "default": "left" }, "joinFields": { "title": "Columns to bring over", "type": "array", "description": "Empty: every column of the join table except its key.", "items": { "type": "string" } }, "joinPrefix": { "title": "Prefix for clashing columns", "type": "string", "description": "When your rows already have a column of the same name, the joined value is added as <prefix><name>, so nothing is overwritten.", "default": "right_" }, "joinIgnoreCase": { "title": "Match keys ignoring case", "type": "boolean", "description": "\"ACME.example\" matches \"acme.example\". The first matching row of the join table is used.", "default": true }, "sortBy": { "title": "Sort by field", "type": "string", "description": "Numbers sort as numbers, text alphabetically; empty values go last. Sorting holds all rows in memory." }, "sortDirection": { "title": "Sort direction", "enum": [ "asc", "desc" ], "type": "string", "description": "Ascending (A to Z, small to large) or descending.", "default": "asc" }, "maxRows": { "title": "Max output rows", "minimum": 0, "type": "integer", "description": "Stop after this many output rows. 0: no limit. Also a cap on the price.", "default": 0 }, "exportFormats": { "title": "Also save as files", "type": "array", "description": "Save the output rows as files in the key-value store, with download links in the OUTPUT record. Free.", "items": { "type": "string", "enum": [ "csv", "xlsx", "json" ], "enumTitles": [ "CSV", "Excel (.xlsx)", "JSON" ] } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 }, "PROXY_UNBLOCKER_UNITS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 }, "PROXY_UNBLOCKER_UNITS": { "type": "integer", "example": 0 } } } } } } } } }}OpenAPI is a standard for designing and describing RESTful APIs, allowing developers to define API structure, endpoints, and data formats in a machine-readable way. It simplifies API development, integration, and documentation.
OpenAPI is effective when used with AI agents and GPTs by standardizing how these systems interact with various APIs, for reliable integrations and efficient communication.
By defining machine-readable API specifications, OpenAPI allows AI models like GPTs to understand and use varied data sources, improving accuracy. This accelerates development, reduces errors, and provides context-aware responses, making OpenAPI a core component for AI applications.
You can download the OpenAPI definitions for Dataset Merge & Dedup Datasets — Filter, Join CSV & Excel from the options below:
If you’d like to learn more about how OpenAPI powers GPTs, read our blog post.
You can also check out our other API clients: