# Dataset Toolkit — Dedupe, Merge, Clean, Diff (`glueworks/dataset-toolkit`) Actor Post-process any Apify dataset: remove duplicates, merge multiple datasets, clean and normalize fields, or diff two crawls. Outputs a new dataset plus a summary record. - **URL**: https://apify.com/glueworks/dataset-toolkit.md - **Developed by:** [KAHA Chen](https://apify.com/glueworks) (community) - **Categories:** Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Dataset Toolkit — Dedupe, Merge, Clean, Diff > **Tool card (for AI agents & quick evaluation)** > > **What it does:** Post-processes existing Apify datasets — removes duplicates (`dedupe`), combines up to 10 datasets (`merge`), normalizes fields (`clean`), or compares two crawls (`diff`) — and writes the result to a new dataset plus a `__summary` record. > > **When to use:** > - Remove duplicate products/listings/profiles from a scraper's output before export. > - Merge outputs of several scrapers (or runs) into one dataset with uniform field names. > - Clean scraped data for a database, BI tool, or LLM: trim whitespace, drop empty fields, convert `"$1,299.00"` to `1299`, strip HTML, keep only selected fields. > - Diff yesterday's crawl against today's to get only `added` / `removed` / `changed` records. > > **Inputs:** `operation` (`dedupe`|`merge`|`clean`|`diff`), `datasetIds` (1–10 Apify dataset IDs; exactly 2 for diff, old then new; empty = demo mode), plus per-operation options (`dedupeKeys`, `fieldMapping`, `cleanRules`, `diffKeys`). **Outputs:** processed records in the run's default dataset (or a named one), with a final `{"__summary": true, itemsIn, itemsOut, ...}` record. > > **Key caveat:** it never modifies source datasets; downstream consumers should skip the record where `__summary` is true. Post-process any Apify dataset without writing a single line of code. Point this Actor at the output of your scraper (or several scrapers), pick an operation, and get a new, tidy dataset back — plus a summary record telling you exactly what happened. - **Dedupe** — remove duplicate records, by any field combination or the whole record - **Merge** — combine up to 10 datasets into one, with optional field renaming and source tagging - **Clean** — trim whitespace, drop empty fields, convert types, rename, whitelist/blacklist fields, regex-replace - **Diff** — compare two crawls and get `added` / `removed` / `changed` records It streams data in batches of 1,000 items, so it handles large datasets without loading everything into memory, and it works with **any** dataset produced by **any** Actor on the Apify platform. ### Why you need this Scrapers produce raw data. Before that data is useful, you almost always need one of these steps: | Use case | Operation | |---|---| | Your crawler visited some pages twice and you have duplicate products / listings / profiles | `dedupe` | | You scraped the same kind of data from several sources (or several runs) and want one combined dataset with consistent field names | `merge` | | Whitespace, `null` fields, prices as strings (`"$1,299.00"`), HTML fragments in text — and you want to feed the data to a BI tool, a database, or an LLM | `clean` | | You crawl a site daily and only care about **what changed** since yesterday — new listings, removed items, price changes | `diff` | | Reducing dataset size before export (drop 40 noisy fields, keep the 5 you use) | `clean` with `pick` | ### Quick start 1. Run any scraper on Apify — note its dataset ID (or use a named dataset). 2. Run **Dataset Toolkit** with that ID in `datasetIds` and pick an operation. 3. Read the results from this run's default dataset, or set `outputDatasetName` to write to a named dataset your other tools can consume. **No dataset yet? Just press Run.** With `datasetIds` empty, the Actor runs in **demo mode** on a small built-in sample of messy e-commerce records so you can inspect the exact output format of every operation. The default input (clean + trim + removeEmpty, demo mode) always succeeds — no setup required. ### Operations and examples #### 1. Dedupe Removes duplicates. Identity is defined by `dedupeKeys` (dot-notation supported for nested fields); leave it empty to deduplicate on the entire record. `keepStrategy` decides whether the first or the last occurrence wins (use `last` when later items are fresher). Memory-efficient: only a 32-byte hash is kept per unique record (`keepStrategy: "first"`). **Input** ```json { "operation": "dedupe", "datasetIds": ["aBcDeFgHiJkLmNoP"], "dedupeKeys": ["sku", "seller.name"], "keepStrategy": "last" } ```` **Output** — the surviving records, unchanged, plus a summary: ```json { "__summary": true, "operation": "dedupe", "itemsIn": 12840, "itemsOut": 11213, "itemsRemoved": 1627, "durationSeconds": 8.4 } ``` #### 2. Merge Concatenates 1–10 datasets into one output dataset. Optionally rename fields on the way (so differently-shaped sources end up with a uniform schema) and tag each record with the dataset it came from. **Input** ```json { "operation": "merge", "datasetIds": ["datasetFromScraperA", "datasetFromScraperB"], "fieldMapping": { "datasetFromScraperA": { "name": "title", "cost": "price" }, "datasetFromScraperB": { "productTitle": "title" } }, "addSourceField": "source" } ``` `fieldMapping` also accepts a flat form `{ "oldField": "newField" }` applied to every dataset. Both sides support dot-notation (`"seller.name": "sellerName"`). **Output** — all records from A then B, renamed, each with `"source": ""`. > Tip: run `merge` first, then `dedupe` on the merged dataset to build a clean combined catalogue from multiple scrapers. #### 3. Clean Applies an **ordered** array of rules to every record. Supported rules: | Rule | Parameters | What it does | |---|---|---| | `trim` | optional `fields` | Strip whitespace from all string values (recursively), or only the listed fields | | `removeEmpty` | optional `removeEmptyContainers` | Drop fields that are `null` or `""` (and `[]` / `{}` if the flag is true). `0` and `false` are kept | | `convert` | `field`, `to`: `"number"` | `"string"` | Type conversion. `"$1,299.00"` → `1299`. If a value cannot be converted it is left unchanged (the record is never dropped) | | `pick` | `fields` | Whitelist — keep only these fields (dot-notation reconstructs nesting) | | `omit` | `fields` | Blacklist — remove these fields | | `regexReplace` | `pattern`, `replacement`, optional `field`/`fields` | Regex substitution on one field, several, or every string field (e.g. strip HTML tags) | | `rename` | `from`, `to` | Move a field, including into/out of nested objects | **Input** ```json { "operation": "clean", "datasetIds": ["aBcDeFgHiJkLmNoP"], "cleanRules": [ { "type": "trim" }, { "type": "removeEmpty" }, { "type": "regexReplace", "field": "description", "pattern": "<[^>]+>", "replacement": "" }, { "type": "convert", "field": "price", "to": "number" }, { "type": "rename", "from": "seller.name", "to": "sellerName" }, { "type": "pick", "fields": ["title", "price", "sellerName", "url"] } ] } ``` **Output** — one cleaned record per input record: trimmed, HTML stripped, `price` numeric, only the four picked fields. #### 4. Diff Compares exactly two datasets: the **old** one first, the **new** one second. Records are matched by `diffKeys`. Unchanged records are not emitted. **Input** ```json { "operation": "diff", "datasetIds": ["yesterdaysCrawlDatasetId", "todaysCrawlDatasetId"], "diffKeys": ["url"] } ``` **Output** ```json { "diff": "added", "key": { "url": "https://shop.example/p/999" }, "new": { "...": "..." } } { "diff": "removed", "key": { "url": "https://shop.example/p/123" }, "old": { "...": "..." } } { "diff": "changed", "key": { "url": "https://shop.example/p/456" }, "changedFields": ["price", "stock"], "old": { "...": "..." }, "new": { "...": "..." } } { "__summary": true, "operation": "diff", "diffStats": { "added": 12, "removed": 3, "changed": 41 }, "...": "..." } ``` This is the building block for **change monitoring**: price drops, new job postings, delisted products — without writing any comparison code. ### Output - Results go to the run's **default dataset**, or to a **named dataset** if you set `outputDatasetName` (note: pushing to an existing named dataset appends). - The **last record is always a summary**: `{"__summary": true, operation, itemsIn, itemsOut, itemsRemoved, durationSeconds, sourceDatasets, demoMode}` (plus `diffStats` for diff). Downstream, simply skip records where `__summary` is set. - The run's status message shows the same counts at a glance. ### Use via MCP / AI Agents Any AI agent connected to Apify's hosted MCP server at **[mcp.apify.com](https://mcp.apify.com)** (Claude, ChatGPT, Cursor, VS Code, or any MCP-compatible client) can discover and run this Actor — no extra setup required: 1. **Discover** it with the `search-actors` tool (e.g. query "dataset dedupe merge clean diff"). 2. **Inspect** the input schema with `fetch-actor-details`. 3. **Run** it with `call-actor` using the Actor ID `glueworks/dataset-toolkit`. To pin this Actor as a dedicated tool in your MCP client, connect to: ``` https://mcp.apify.com/?tools=glueworks/dataset-toolkit ``` Minimal input for an agent call (dedupe a scraper's output by URL): ```json { "operation": "dedupe", "datasetIds": [""], "dedupeKeys": ["url"] } ``` **Notes for agents:** - This Actor operates on **existing Apify datasets** — pass the `defaultDatasetId` of a previous Actor run (e.g. from `call-actor` results). It is the natural second step after any scraping Actor. - Safe to try with no data: an empty `datasetIds` runs a built-in **demo mode**, useful for previewing the output shape. - `diff` needs **exactly 2** dataset IDs — OLD first, NEW second — and `diffKeys` identifying records (e.g. `["url"]`). - The last output record has `"__summary": true` with `itemsIn`/`itemsOut` counts — use it to report results, and exclude it when passing data downstream. - Chain operations (e.g. merge → dedupe → clean) by feeding each run's output dataset ID into the next run. ### Using it in an Apify workflow The typical setup is scraper ➜ toolkit, automated: - **Actor task + webhook**: create a task for your scraper, add a webhook on `ACTOR.RUN.SUCCEEDED` that calls `Run Actor` on Dataset Toolkit, passing `{"datasetIds": ["{{resource.defaultDatasetId}}"], "operation": "dedupe", "dedupeKeys": ["url"]}` as input. - **From your own code** (JS): ```js const scraperRun = await client.actor('apify/web-scraper').call(scraperInput); await client.actor('glueworks/dataset-toolkit').call({ operation: 'clean', datasetIds: [scraperRun.defaultDatasetId], outputDatasetName: 'products-clean', cleanRules: [{ type: 'trim' }, { type: 'removeEmpty' }, { type: 'convert', field: 'price', to: 'number' }], }); ``` - **Daily diff monitoring**: schedule your scraper to write to a named dataset per day (or keep the two latest run dataset IDs), then schedule Dataset Toolkit with `operation: "diff"` and alert on `diffStats` in the summary record. ### Pricing Pay-per-event, fully usage-based: | Event | Price | |---|---| | Actor start | $0.005 per run | | Items processed | $0.02 per 1,000 input items (rounded up, min. 1) | Example: deduplicating a 50,000-item dataset costs $0.005 + 50 × $0.02 = **$1.005**. A 5,000-item clean costs **$0.105**. ### Limits and notes - Up to **10 input datasets**, read in batches of 1,000 items (constant memory for `clean`, `merge`, and `dedupe` with `keepStrategy: "first"`). - `dedupe` with `keepStrategy: "last"` buffers one record per unique key; `diff` indexes the whole **old** dataset in memory. For multi-million-item datasets, raise the run's memory limit. - Field paths use **dot-notation** (`seller.name`). Array elements are treated as atomic values (no `items.0.price` indexing). - Key matching is **type-sensitive**: `1` and `"1"` are different keys. Use a `convert` clean rule first if your sources disagree on types. - `diff` expects keys to be unique within each dataset; with duplicate keys, a key matches at most once. ### FAQ **What happens if I run it with the default input?** It runs in demo mode: cleans a small built-in sample dataset and writes the results plus a summary, so you can see the output shape. No external data is touched. **Does it modify my source datasets?** Never. Sources are read-only; results always go to a new (or the default) dataset. **Can I chain operations, e.g. merge + dedupe + clean?** Run the Actor once per step, feeding each run's output dataset ID (or a named output dataset) into the next. Each step costs its own items-processed events. **How are nested fields handled?** All key/field parameters accept dot-notation. `pick` rebuilds the nested structure; `rename` and `fieldMapping` can move values between nesting levels. **What about records that fail a conversion?** `convert` never drops or nulls a record — unconvertible values are passed through unchanged, so you never lose data silently. **Which scrapers does it work with?** Any Actor that writes to a dataset — Web Scraper, Cheerio Scraper, all Store scrapers, or your own Actors. # Actor input Schema ## `operation` (type: `string`): What to do with the input dataset(s). dedupe = remove duplicate records (see dedupeKeys, keepStrategy); merge = concatenate 1-10 datasets into one (see fieldMapping, addSourceField); clean = apply an ordered list of cleaning rules to every record (see cleanRules); diff = compare exactly 2 datasets and output added/removed/changed records (see diffKeys). Default: clean. ## `datasetIds` (type: `array`): 1–10 Apify dataset IDs (e.g. "aBcDeFgHiJkLmNoP", typically the defaultDatasetId of a previous Actor run) or named datasets to read. For diff, provide exactly 2: the OLD dataset first, the NEW one second. Leave EMPTY to run in demo mode on built-in sample data (useful to preview the output format). Source datasets are never modified. ## `outputDatasetName` (type: `string`): Optional name for the output (named) dataset. If empty, results go to the run's default dataset. Note: pushing to an existing named dataset appends to it. ## `dedupeKeys` (type: `array`): Only used when operation=dedupe. Field names whose combination defines a duplicate, e.g. \["sku", "seller.name"] (dot-notation supported for nested fields). Leave empty to deduplicate on the ENTIRE record. Matching is type-sensitive: 1 and "1" are different keys. ## `keepStrategy` (type: `string`): Only used when operation=dedupe. Which occurrence of a duplicate to keep: "first" (default, streams with minimal memory) or "last" (keeps the freshest data, buffers one record per unique key). Use "last" when later records are more up to date. ## `fieldMapping` (type: `object`): Only used when operation=merge. Optional rename map applied while merging. Flat form {"oldField": "newField"} applies to all datasets; nested form {"": {"old": "new"}} applies per dataset. Dot-notation supported on both sides, e.g. {"seller.name": "sellerName"}. Example: {"datasetA": {"cost": "price"}, "datasetB": {"productTitle": "title"}}. ## `addSourceField` (type: `string`): Only used when operation=merge. If set to a field name (e.g. "source"), each merged record gets this field containing the ID of the dataset it came from. Leave empty to disable. ## `cleanRules` (type: `array`): Only used when operation=clean. Ordered array of rule objects, applied to every record in sequence. Supported types: {"type": "trim"} (strip whitespace, optional fields array), {"type": "removeEmpty"} (drop null/"" fields; 0 and false are kept), {"type": "convert", "field": "price", "to": "number"} (e.g. "$1,299.00" -> 1299; unconvertible values pass through unchanged), {"type": "pick", "fields": \[...]} (whitelist), {"type": "omit", "fields": \[...]} (blacklist), {"type": "regexReplace", "pattern": "<\[^>]+>", "replacement": "", "field": "description"} (regex substitution; omit field to apply to all string fields), {"type": "rename", "from": "seller.name", "to": "sellerName"}. Dot-notation supported. See README for full examples. ## `diffKeys` (type: `array`): Only used (and required) when operation=diff. Field names identifying the same logical record across the two datasets, e.g. \["url"] or \["sku", "seller.name"] (dot-notation supported). Keys should be unique within each dataset; matching is type-sensitive. Default: \["id"]. ## Actor input object example ```json { "operation": "clean", "datasetIds": [], "outputDatasetName": "", "dedupeKeys": [], "keepStrategy": "first", "addSourceField": "", "cleanRules": [ { "type": "trim" }, { "type": "removeEmpty" } ], "diffKeys": [ "id" ] } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "operation": "clean", "datasetIds": [], "cleanRules": [ { "type": "trim" }, { "type": "removeEmpty" } ], "diffKeys": [ "id" ] }; // Run the Actor and wait for it to finish const run = await client.actor("glueworks/dataset-toolkit").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "operation": "clean", "datasetIds": [], "cleanRules": [ { "type": "trim" }, { "type": "removeEmpty" }, ], "diffKeys": ["id"], } # Run the Actor and wait for it to finish run = client.actor("glueworks/dataset-toolkit").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "operation": "clean", "datasetIds": [], "cleanRules": [ { "type": "trim" }, { "type": "removeEmpty" } ], "diffKeys": [ "id" ] }' | apify call glueworks/dataset-toolkit --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=glueworks/dataset-toolkit", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Dataset Toolkit — Dedupe, Merge, Clean, Diff", "description": "Post-process any Apify dataset: remove duplicates, merge multiple datasets, clean and normalize fields, or diff two crawls. Outputs a new dataset plus a summary record.", "version": "0.1", "x-build-id": "NCnb6ttttmJkJmZB8" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/glueworks~dataset-toolkit/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-glueworks-dataset-toolkit", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/glueworks~dataset-toolkit/runs": { "post": { "operationId": "runs-sync-glueworks-dataset-toolkit", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/glueworks~dataset-toolkit/run-sync": { "post": { "operationId": "run-sync-glueworks-dataset-toolkit", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "operation": { "title": "Operation", "enum": [ "dedupe", "merge", "clean", "diff" ], "type": "string", "description": "What to do with the input dataset(s). dedupe = remove duplicate records (see dedupeKeys, keepStrategy); merge = concatenate 1-10 datasets into one (see fieldMapping, addSourceField); clean = apply an ordered list of cleaning rules to every record (see cleanRules); diff = compare exactly 2 datasets and output added/removed/changed records (see diffKeys). Default: clean.", "default": "clean" }, "datasetIds": { "title": "Dataset IDs", "maxItems": 10, "type": "array", "description": "1–10 Apify dataset IDs (e.g. \"aBcDeFgHiJkLmNoP\", typically the defaultDatasetId of a previous Actor run) or named datasets to read. For diff, provide exactly 2: the OLD dataset first, the NEW one second. Leave EMPTY to run in demo mode on built-in sample data (useful to preview the output format). Source datasets are never modified.", "default": [], "items": { "type": "string" } }, "outputDatasetName": { "title": "Output dataset name", "type": "string", "description": "Optional name for the output (named) dataset. If empty, results go to the run's default dataset. Note: pushing to an existing named dataset appends to it.", "default": "" }, "dedupeKeys": { "title": "Dedupe: key fields", "type": "array", "description": "Only used when operation=dedupe. Field names whose combination defines a duplicate, e.g. [\"sku\", \"seller.name\"] (dot-notation supported for nested fields). Leave empty to deduplicate on the ENTIRE record. Matching is type-sensitive: 1 and \"1\" are different keys.", "default": [], "items": { "type": "string" } }, "keepStrategy": { "title": "Dedupe: keep strategy", "enum": [ "first", "last" ], "type": "string", "description": "Only used when operation=dedupe. Which occurrence of a duplicate to keep: \"first\" (default, streams with minimal memory) or \"last\" (keeps the freshest data, buffers one record per unique key). Use \"last\" when later records are more up to date.", "default": "first" }, "fieldMapping": { "title": "Merge: field mapping", "type": "object", "description": "Only used when operation=merge. Optional rename map applied while merging. Flat form {\"oldField\": \"newField\"} applies to all datasets; nested form {\"\": {\"old\": \"new\"}} applies per dataset. Dot-notation supported on both sides, e.g. {\"seller.name\": \"sellerName\"}. Example: {\"datasetA\": {\"cost\": \"price\"}, \"datasetB\": {\"productTitle\": \"title\"}}." }, "addSourceField": { "title": "Merge: add source field", "type": "string", "description": "Only used when operation=merge. If set to a field name (e.g. \"source\"), each merged record gets this field containing the ID of the dataset it came from. Leave empty to disable.", "default": "" }, "cleanRules": { "title": "Clean: rules", "type": "array", "description": "Only used when operation=clean. Ordered array of rule objects, applied to every record in sequence. Supported types: {\"type\": \"trim\"} (strip whitespace, optional fields array), {\"type\": \"removeEmpty\"} (drop null/\"\" fields; 0 and false are kept), {\"type\": \"convert\", \"field\": \"price\", \"to\": \"number\"} (e.g. \"$1,299.00\" -> 1299; unconvertible values pass through unchanged), {\"type\": \"pick\", \"fields\": [...]} (whitelist), {\"type\": \"omit\", \"fields\": [...]} (blacklist), {\"type\": \"regexReplace\", \"pattern\": \"<[^>]+>\", \"replacement\": \"\", \"field\": \"description\"} (regex substitution; omit field to apply to all string fields), {\"type\": \"rename\", \"from\": \"seller.name\", \"to\": \"sellerName\"}. Dot-notation supported. See README for full examples.", "default": [ { "type": "trim" }, { "type": "removeEmpty" } ] }, "diffKeys": { "title": "Diff: key fields", "type": "array", "description": "Only used (and required) when operation=diff. Field names identifying the same logical record across the two datasets, e.g. [\"url\"] or [\"sku\", \"seller.name\"] (dot-notation supported). Keys should be unique within each dataset; matching is type-sensitive. Default: [\"id\"].", "default": [ "id" ], "items": { "type": "string" } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```