# Merge, Dedup & Transform Datasets - Limited permissions (`lukaskrivka/dedup-datasets-limited-permissions`) Actor

Limited permission version of the popular Actor. The ultimate dataset processor. Extremely fast merging, deduplications & transformations all in a single run.

- **URL**: https://apify.com/lukaskrivka/dedup-datasets-limited-permissions.md
- **Developed by:** [Lukáš Křivka](https://apify.com/lukaskrivka) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Dataset processing actor (limited permissions) - merge, dedup & transform

> **Looking for the other version of this Actor?**
> This is the **limited permissions** build. It only works with dataset IDs you pass in the input, so it doesn't require the full permission approval when you run it.
> If you need to merge **all runs of an Actor or Task** (optionally within a date range), use the full [lukaskrivka/dedup-datasets](https://apify.com/lukaskrivka/dedup-datasets) Actor instead - reading your runs is not possible with limited permission access, so that version asks for full permission approval.

Refined and optimized dataset processing actor for large scale merging, deduplications and transformation

### Why to use this actor

- Extremely fast data processing thanks for parallelizing workloads (easily 20x faster than default loading/pushing datasets)
- Allows reading from multiple datasets silmutanesously, ideal for merging after scraping with many runs
- Actor migration proof - All steps that can be persisted are persisted => work is not repeated and no duplicated data pushed
- `Dedup as loading` mode allows for near constant memory processing even for huge datasets (think 10M+)
- Deduplication allows for combination of many fields and even nested objects/arrays (those are JSON.stringified for deep equality check)
- Allows for storing into KV store records
- Allows super fast blank runs that count duplicates

### Merging

You can provide more than one dataset. In that case all items are merged into single dataset or key value store output. If you use the `Dedup after load` mode, the order of items will retain the order of datasets provided.

### Deduplication

If you optionally provide deduplication `fields`, this actor will deduplicate the dataset items. The deduplication process check the values of each field for equality and only return the first unique one (the first item that has a unique value for that field).

You can provide more than one field. In that case a combined string of that fields is checked, e.g. `"name": "Adidas Shoes, "id": "12345"` gets converted into `"Adidas Shoes12345"` for the checking purpose. So only items that have both fields the same are considered duplicates. This means the more fields you add, the less duplicates will be found.

Fields that are objects or arrays are also deeply compared via `JSON.stringify`. Just be aware that doing this for very large structures might have performance implications.

### Transformation

This actor enables you to do arbitrary data transformations before and after deduplication via `preDedupTransformFunction` and `postDedupTransformFunction`.

These functions simply take the array of items and should return array of items. You don't need to necessarily return the same amount of items (can filter some out or add new ones).

You can access an object with helper variables, currently containing the [Apify SDK reference](https://sdk.apify.com/docs/api/apify)

The default transformation does nothing with the items:

```javascript
(items, { Apify, customInputData }) => {
    return items;
}
```

In case of `dedup-as-loading` mode, you only have access to the items of the specific batch.
But you can also access `datasetId` and `datasetOffset` parameters as each batch is only from one dataset.

```javascript
(items, { Apify, datasetId, datasetOffset, customInputData }) => {
    return items;
}
```

### Input

Detailed INPUT table with description can be found on the [actor's public page](https://apify.com/lukaskrivka/dedup-datasets-limited-permissions/input-schema).

### Changelog

Check the list of past updates [here](https://github.com/metalwarrior665/actor-dedup-datasets/blob/master/CHANGELOG.md)

# Actor input Schema

## `datasetIds` (type: `array`):

Datasets that should be deduplicated and merged

## `fields` (type: `array`):

Fields whose combination should be unique for the item to be considered unique. If none are provided, the actor does not perform deduplication.

## `output` (type: `string`):

What will be pushed to the dataset from this actor

## `mode` (type: `string`):

How the loading and deduplication process will work.

## `outputDatasetId` (type: `string`):

Optionally can push into dataset of your choice. If you provide a dataset name that doesn't exist, a new named dataset will be created.

## `fieldsToLoad` (type: `array`):

You can choose which fields to load only. Useful to speed up the loading and reduce memory needs.

## `preDedupTransformFunction` (type: `string`):

Function to transform items before deduplication is applied. For 'dedup-after-load' mode this is done for all items at once. For 'dedup-as-loading' this is applied to each batch separately.

## `postDedupTransformFunction` (type: `string`):

Function to transform items after deduplication is applied. For 'dedup-after-load' mode this is done for all items at once. For 'dedup-as-loading' this is applied to each batch separately.

## `outputTo` (type: `string`):

Either can output to a single dataset or to split data into KV records depending on upload batch size. KV is upload is much faster but data end up in many files.

## `parallelLoads` (type: `integer`):

Datasets can be loaded in parallel batches to speed things up if needed.

## `parallelPushes` (type: `integer`):

Deduped data can be pushed in parallel batches to speed things up if needed. If you want the data to be in the exact same order, you need to set this to 1.

## `uploadBatchSize` (type: `integer`):

How many items it should upload in one pushData call. Useful to not overload Apify API. Only important for dataset upload.

## `batchSizeLoad` (type: `integer`):

How many items it will load in a single batch.

## `offset` (type: `integer`):

By default we don't skip any items which is the same as setting offset to 0. For multiple datasets, it takes offset into the sum of their item counts but that is not very useful.

## `limit` (type: `integer`):

By default we don't limit the number loaded items

## `verboseLog` (type: `boolean`):

Good for smaller runs. Large runs might run out of log space.

## `nullAsUnique` (type: `boolean`):

If you want to treat null (or missing) fields as always unique items.

## `datasetIdsOfFilterItems` (type: `array`):

The items from these datasets will be just used as a dedup filter for the main datasets. These items are loaded first and then the main datasets are compared for uniqueness and pushed.

## `customInputData` (type: `object`):

You can pass custom data as a JSON object to be accessible in the transform functions as part of the 2nd parameter object.

## `appendDatasetIds` (type: `boolean`):

Useful for transform functions. Each item will get a field `__datasetId__` with the dataset ID it came from.

## Actor input object example

```json
{
  "output": "unique-items",
  "mode": "dedup-after-load",
  "preDedupTransformFunction": "async (items, { Apify }) => {\n    return items;\n}",
  "postDedupTransformFunction": "async (items, { Apify }) => {\n    return items;\n}",
  "outputTo": "dataset",
  "parallelLoads": 10,
  "parallelPushes": 5,
  "uploadBatchSize": 500,
  "batchSizeLoad": 50000,
  "verboseLog": false,
  "nullAsUnique": false,
  "customInputData": {},
  "appendDatasetIds": false
}
```

# Actor output Schema

## `dataset` (type: `string`):

Merged and deduplicated items. This is the output for `outputTo: dataset` when no `outputDatasetId` is provided.

## `keyValueStore` (type: `string`):

The `OUTPUT-*` records holding the merged and deduplicated items. This is the output for `outputTo: key-value-store`.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "preDedupTransformFunction": async (items, { Apify }) => {
        return items;
    },
    "postDedupTransformFunction": async (items, { Apify }) => {
        return items;
    },
    "customInputData": {}
};

// Run the Actor and wait for it to finish
const run = await client.actor("lukaskrivka/dedup-datasets-limited-permissions").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "preDedupTransformFunction": """async (items, { Apify }) => {
    return items;
}""",
    "postDedupTransformFunction": """async (items, { Apify }) => {
    return items;
}""",
    "customInputData": {},
}

# Run the Actor and wait for it to finish
run = client.actor("lukaskrivka/dedup-datasets-limited-permissions").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "preDedupTransformFunction": "async (items, { Apify }) => {\\n    return items;\\n}",
  "postDedupTransformFunction": "async (items, { Apify }) => {\\n    return items;\\n}",
  "customInputData": {}
}' |
apify call lukaskrivka/dedup-datasets-limited-permissions --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,lukaskrivka/dedup-datasets-limited-permissions"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/QXGNR9uobVenxNVNV/builds/1u7rA5G914c4eYTbZ/openapi.json