Dataset Toolkit — Dedupe, Merge, Clean, Diff

Tool card (for AI agents & quick evaluation)

What it does: Post-processes existing Apify datasets — removes duplicates (dedupe), combines up to 10 datasets (merge), normalizes fields (clean), or compares two crawls (diff) — and writes the result to a new dataset plus a __summary record.

When to use:

• Remove duplicate products/listings/profiles from a scraper's output before export.
• Merge outputs of several scrapers (or runs) into one dataset with uniform field names.
• Clean scraped data for a database, BI tool, or LLM: trim whitespace, drop empty fields, convert "$1,299.00" to 1299, strip HTML, keep only selected fields.
• Diff yesterday's crawl against today's to get only added / removed / changed records.

Inputs: operation (dedupe|merge|clean|diff), datasetIds (1–10 Apify dataset IDs; exactly 2 for diff, old then new; empty = demo mode), plus per-operation options (dedupeKeys, fieldMapping, cleanRules, diffKeys). Outputs: processed records in the run's default dataset (or a named one), with a final {"__summary": true, itemsIn, itemsOut, ...} record.

Key caveat: it never modifies source datasets; downstream consumers should skip the record where __summary is true.

Post-process any Apify dataset without writing a single line of code. Point this Actor at the output of your scraper (or several scrapers), pick an operation, and get a new, tidy dataset back — plus a summary record telling you exactly what happened.

• Dedupe — remove duplicate records, by any field combination or the whole record
• Merge — combine up to 10 datasets into one, with optional field renaming and source tagging
• Clean — trim whitespace, drop empty fields, convert types, rename, whitelist/blacklist fields, regex-replace
• Diff — compare two crawls and get added / removed / changed records

It streams data in batches of 1,000 items, so it handles large datasets without loading everything into memory, and it works with any dataset produced by any Actor on the Apify platform.

Why you need this

Scrapers produce raw data. Before that data is useful, you almost always need one of these steps:

Use case	Operation
Your crawler visited some pages twice and you have duplicate products / listings / profiles	dedupe
You scraped the same kind of data from several sources (or several runs) and want one combined dataset with consistent field names	merge
Whitespace, null fields, prices as strings ("$1,299.00"), HTML fragments in text — and you want to feed the data to a BI tool, a database, or an LLM	clean
You crawl a site daily and only care about what changed since yesterday — new listings, removed items, price changes	diff
Reducing dataset size before export (drop 40 noisy fields, keep the 5 you use)	clean with pick

Quick start

1. Run any scraper on Apify — note its dataset ID (or use a named dataset).
2. Run Dataset Toolkit with that ID in datasetIds and pick an operation.
3. Read the results from this run's default dataset, or set outputDatasetName to write to a named dataset your other tools can consume.

No dataset yet? Just press Run. With datasetIds empty, the Actor runs in demo mode on a small built-in sample of messy e-commerce records so you can inspect the exact output format of every operation. The default input (clean + trim + removeEmpty, demo mode) always succeeds — no setup required.

Operations and examples

1. Dedupe

Removes duplicates. Identity is defined by dedupeKeys (dot-notation supported for nested fields); leave it empty to deduplicate on the entire record. keepStrategy decides whether the first or the last occurrence wins (use last when later items are fresher). Memory-efficient: only a 32-byte hash is kept per unique record (keepStrategy: "first").

Input

{
  "operation": "dedupe",
  "datasetIds": ["aBcDeFgHiJkLmNoP"],
  "dedupeKeys": ["sku", "seller.name"],
  "keepStrategy": "last"
}

Output — the surviving records, unchanged, plus a summary:

{ "__summary": true, "operation": "dedupe", "itemsIn": 12840, "itemsOut": 11213, "itemsRemoved": 1627, "durationSeconds": 8.4 }

2. Merge

Concatenates 1–10 datasets into one output dataset. Optionally rename fields on the way (so differently-shaped sources end up with a uniform schema) and tag each record with the dataset it came from.

Input

{
  "operation": "merge",
  "datasetIds": ["datasetFromScraperA", "datasetFromScraperB"],
  "fieldMapping": {
    "datasetFromScraperA": { "name": "title", "cost": "price" },
    "datasetFromScraperB": { "productTitle": "title" }
  },
  "addSourceField": "source"
}

fieldMapping also accepts a flat form { "oldField": "newField" } applied to every dataset. Both sides support dot-notation ("seller.name": "sellerName").

Output — all records from A then B, renamed, each with "source": "<datasetId>".

Tip: run merge first, then dedupe on the merged dataset to build a clean combined catalogue from multiple scrapers.

3. Clean

Applies an ordered array of rules to every record. Supported rules:

Rule	Parameters	What it does
trim	optional fields	Strip whitespace from all string values (recursively), or only the listed fields
removeEmpty	optional removeEmptyContainers	Drop fields that are null or "" (and [] / {} if the flag is true). 0 and false are kept
convert	field, to: "number" | "string"	Type conversion. "$1,299.00" → 1299. If a value cannot be converted it is left unchanged (the record is never dropped)
pick	fields	Whitelist — keep only these fields (dot-notation reconstructs nesting)
omit	fields	Blacklist — remove these fields
regexReplace	pattern, replacement, optional field/fields	Regex substitution on one field, several, or every string field (e.g. strip HTML tags)
rename	from, to	Move a field, including into/out of nested objects

Input

{
  "operation": "clean",
  "datasetIds": ["aBcDeFgHiJkLmNoP"],
  "cleanRules": [
    { "type": "trim" },
    { "type": "removeEmpty" },
    { "type": "regexReplace", "field": "description", "pattern": "<[^>]+>", "replacement": "" },
    { "type": "convert", "field": "price", "to": "number" },
    { "type": "rename", "from": "seller.name", "to": "sellerName" },
    { "type": "pick", "fields": ["title", "price", "sellerName", "url"] }
  ]
}

Output — one cleaned record per input record: trimmed, HTML stripped, price numeric, only the four picked fields.

4. Diff

Compares exactly two datasets: the old one first, the new one second. Records are matched by diffKeys. Unchanged records are not emitted.

Input

{
  "operation": "diff",
  "datasetIds": ["yesterdaysCrawlDatasetId", "todaysCrawlDatasetId"],
  "diffKeys": ["url"]
}

Output

{ "diff": "added",   "key": { "url": "https://shop.example/p/999" }, "new": { "...": "..." } }
{ "diff": "removed", "key": { "url": "https://shop.example/p/123" }, "old": { "...": "..." } }
{ "diff": "changed", "key": { "url": "https://shop.example/p/456" },
  "changedFields": ["price", "stock"], "old": { "...": "..." }, "new": { "...": "..." } }
{ "__summary": true, "operation": "diff", "diffStats": { "added": 12, "removed": 3, "changed": 41 }, "...": "..." }

This is the building block for change monitoring: price drops, new job postings, delisted products — without writing any comparison code.

Output

• Results go to the run's default dataset, or to a named dataset if you set outputDatasetName (note: pushing to an existing named dataset appends).
• The last record is always a summary: {"__summary": true, operation, itemsIn, itemsOut, itemsRemoved, durationSeconds, sourceDatasets, demoMode} (plus diffStats for diff). Downstream, simply skip records where __summary is set.
• The run's status message shows the same counts at a glance.

Use via MCP / AI Agents

Any AI agent connected to Apify's hosted MCP server at mcp.apify.com (Claude, ChatGPT, Cursor, VS Code, or any MCP-compatible client) can discover and run this Actor — no extra setup required:

1. Discover it with the search-actors tool (e.g. query "dataset dedupe merge clean diff").
2. Inspect the input schema with fetch-actor-details.
3. Run it with call-actor using the Actor ID glueworks/dataset-toolkit.

To pin this Actor as a dedicated tool in your MCP client, connect to:

https://mcp.apify.com/?tools=glueworks/dataset-toolkit

Minimal input for an agent call (dedupe a scraper's output by URL):

{
    "operation": "dedupe",
    "datasetIds": ["<DATASET_ID_FROM_A_PREVIOUS_ACTOR_RUN>"],
    "dedupeKeys": ["url"]
}

Notes for agents:

• This Actor operates on existing Apify datasets — pass the defaultDatasetId of a previous Actor run (e.g. from call-actor results). It is the natural second step after any scraping Actor.
• Safe to try with no data: an empty datasetIds runs a built-in demo mode, useful for previewing the output shape.
• diff needs exactly 2 dataset IDs — OLD first, NEW second — and diffKeys identifying records (e.g. ["url"]).
• The last output record has "__summary": true with itemsIn/itemsOut counts — use it to report results, and exclude it when passing data downstream.
• Chain operations (e.g. merge → dedupe → clean) by feeding each run's output dataset ID into the next run.

Using it in an Apify workflow

The typical setup is scraper ➜ toolkit, automated:

• Actor task + webhook: create a task for your scraper, add a webhook on ACTOR.RUN.SUCCEEDED that calls Run Actor on Dataset Toolkit, passing {"datasetIds": ["{{resource.defaultDatasetId}}"], "operation": "dedupe", "dedupeKeys": ["url"]} as input.
• From your own code (JS):

const scraperRun = await client.actor('apify/web-scraper').call(scraperInput);
await client.actor('glueworks/dataset-toolkit').call({
    operation: 'clean',
    datasetIds: [scraperRun.defaultDatasetId],
    outputDatasetName: 'products-clean',
    cleanRules: [{ type: 'trim' }, { type: 'removeEmpty' }, { type: 'convert', field: 'price', to: 'number' }],
});

• Daily diff monitoring: schedule your scraper to write to a named dataset per day (or keep the two latest run dataset IDs), then schedule Dataset Toolkit with operation: "diff" and alert on diffStats in the summary record.

Pricing

Pay-per-event, fully usage-based:

Event	Price
Actor start	$0.005 per run
Items processed	$0.02 per 1,000 input items (rounded up, min. 1)

Example: deduplicating a 50,000-item dataset costs $0.005 + 50 × $0.02 = $1.005. A 5,000-item clean costs $0.105.

Limits and notes

• Up to 10 input datasets, read in batches of 1,000 items (constant memory for clean, merge, and dedupe with keepStrategy: "first").
• dedupe with keepStrategy: "last" buffers one record per unique key; diff indexes the whole old dataset in memory. For multi-million-item datasets, raise the run's memory limit.
• Field paths use dot-notation (seller.name). Array elements are treated as atomic values (no items.0.price indexing).
• Key matching is type-sensitive: 1 and "1" are different keys. Use a convert clean rule first if your sources disagree on types.
• diff expects keys to be unique within each dataset; with duplicate keys, a key matches at most once.

FAQ

What happens if I run it with the default input? It runs in demo mode: cleans a small built-in sample dataset and writes the results plus a summary, so you can see the output shape. No external data is touched.

Does it modify my source datasets? Never. Sources are read-only; results always go to a new (or the default) dataset.

Can I chain operations, e.g. merge + dedupe + clean? Run the Actor once per step, feeding each run's output dataset ID (or a named output dataset) into the next. Each step costs its own items-processed events.

How are nested fields handled? All key/field parameters accept dot-notation. pick rebuilds the nested structure; rename and fieldMapping can move values between nesting levels.

What about records that fail a conversion? convert never drops or nulls a record — unconvertible values are passed through unchanged, so you never lose data silently.

Which scrapers does it work with? Any Actor that writes to a dataset — Web Scraper, Cheerio Scraper, all Store scrapers, or your own Actors.