# JSON Dataset Deduplication — Composite Keys & Provenance (`vermilion_beauty/my-actor`) Actor

Remove duplicate JSON records by one field, composite keys or whole-record equality. Keep numeric and string IDs distinct and trace each retained or duplicate row to its source. $0.05 buys one completed nonempty batch of up to 1,000 records and 128 KiB. Structured output for automated workflows.

- **URL**: https://apify.com/vermilion\_beauty/my-actor.md
- **Developed by:** [Mike](https://apify.com/vermilion_beauty) (community)
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$50.00 / 1,000 completed 1,000-record blocks

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Remove duplicate records and keep their source history

Turn repeated JSON rows into a clean, reproducible dataset. Match on one field or several fields together; see exactly which source row was kept and which duplicate it replaced. The first matching row wins.

**$0.05 per completed nonempty batch of up to 1,000 records** (also limited to 128 KiB of compact input). No subscription to this Actor. Empty input has no event charge.

### Try it with sample data — no dataset preparation

1. Open [the ready-to-run email + company example](https://apify.com/vermilion_beauty/my-actor/examples/remove-duplicate-leads-by-email-and-company).
2. Use its included synthetic input. Sign in to Apify if prompted, review the charge limit, then start the run.
3. Select **Cleaned records**, then **JSON**, to see the **two retained records**. Select **Provenance and billing**, then open **RESULT** to see the duplicate-to-original references and source positions. Require `status: "complete"` before using results.

A fresh repeat run is a separate execution and may incur another $0.05 charge. Reuse existing output when you do not need another run. Repeating the same ordered input, options and version produces identical deduplication results.

### Before and after: remove duplicate leads by email + company

These are synthetic examples, not real leads. All indexes below start at zero.

**INPUT / matching key: email + company**

```json
{
  "authorizedData": true,
  "sources": [
    [
      {
        "email": "demo@example.com",
        "company": "North"
      },
      {
        "email": "demo@example.com",
        "company": "South"
      },
      {
        "email": "demo@example.com",
        "company": "North"
      }
    ]
  ],
  "fields": [
    "/email",
    "/company"
  ]
}
```

**OUTPUT / excerpt from RESULT**

```json
{
  "items": [
    {
      "email": "demo@example.com",
      "company": "North"
    },
    {
      "email": "demo@example.com",
      "company": "South"
    }
  ],
  "provenance": [
    {
      "source_index": 0,
      "row_index": 0
    },
    {
      "source_index": 0,
      "row_index": 1
    }
  ],
  "duplicates": [
    {
      "duplicate": {
        "source_index": 0,
        "row_index": 2
      },
      "retained": {
        "source_index": 0,
        "row_index": 0
      }
    }
  ],
  "summary": {
    "input_count": 3,
    "unique_count": 2,
    "duplicate_count": 1,
    "report_truncated": false
  }
}
```

The North duplicate at row 2 points back to North at row 0. South remains because the company differs. `provenance` follows the order of retained `items`; `duplicates` links each removed row to its retained original. No hidden merging or fuzzy matching.

### Price, limits and charge protection

One completed nonempty batch costs $0.05, whether it contains 3 or 1,000 input records. Both the 1,000-record and 128-KiB limits apply. Runtime usage is included in the Actor event price. Set the run's maximum total charge to $0.05 for this single-batch example; a lower budget rejects nonempty input before output. Empty input produces no billed event. Input or processing failures before the charge step produce no completed-batch event. A later interruption can leave billing pending: inspect BILLING before retrying; do not assume every failed run is uncharged. Apify account storage/retention rules still apply.

### Automation contract

- **INPUT:** authorized inline JSON source arrays; optional exact-match field pointers. Multiple pointers form one composite key.
- **OUTPUT:** cleaned dataset plus authoritative RESULT with items, provenance, duplicates, summary, schema\_version, input\_fingerprint and status.
- **PRICE:** $0.05 per completed nonempty batch; zero event charge for empty input.
- **LIMITS:** 1,000 rows, 128 KiB compact input, ten source arrays and 60-second processing deadline.
- **USE CASES:** duplicate leads, product catalogs, repeated API results and reproducible agent workflows.
- **ERROR BEHAVIOR:** invalid/over-limit input or an unverified privacy/output condition fails closed; consume only complete RESULT. See error and billing guidance below.
- **EXAMPLE CALL:** submit the complete JSON INPUT above in the Actor or saved task's input form. The Apify API button supplies authenticated API/client examples; keep your token private.
- **EXAMPLE RESPONSE:** the OUTPUT excerpt above shows deduplication data; the actual RESULT also includes completion status and version/fingerprint metadata.

**CSV limitation:** to deduplicate CSV by email and company, first convert the CSV into JSON objects, then use the example above. This release has no CSV-file upload, dataset-ID import or API-URL fetching. Exact matching does not lowercase, trim or validate contact details. Numeric and string IDs remain distinct.

### See the result first

Single field: `[{"id":1},{"id":1},{"id":2}]` with `fields:["/id"]` becomes `[{"id":1},{"id":2}]`.

Composite key: SKU A + supplier North is distinct from SKU A + supplier South. A second A/North row is removed; the first row wins.

Typed matching: `[{"id":1},{"id":"1"},{"id":1},{"id":true}]` becomes `[{"id":1},{"id":"1"},{"id":true}]`. JSON numbers 1 and 1.0 compare equal.

Provenance for the single-field example:

```json
{"provenance":[{"source_index":0,"row_index":0},{"source_index":0,"row_index":2}],"duplicates":[{"duplicate":{"source_index":0,"row_index":1},"retained":{"source_index":0,"row_index":0}}]}
```

Run the same ordered input and options again with the same product version: retained records, order, provenance and duplicate counts are identical. Changing input order can change which first record is retained. Run IDs and billing receipts are separate operational metadata.

### Copy a use case

Each JSON block is complete input; replace synthetic rows with your authorized data. Exact matching does not lowercase, trim, verify contacts or select the newest record.

#### Remove duplicate JSON records with composite keys

Match both SKU and supplier; identical SKUs from different suppliers stay separate.

```json
{"authorizedData":true,"sources":[[{"sku":"A","supplier":"North"},{"sku":"A","supplier":"South"},{"sku":"A","supplier":"North"}]],"fields":["/sku","/supplier"]}
```

Expected: 2 retained rows from 3 input rows.

#### Deduplicate data while preserving source provenance

Combine two inline sources and trace retained and removed rows to source and row indexes.

```json
{"authorizedData":true,"sources":[[{"id":1,"value":"first"}],[{"id":1,"value":"later"},{"id":2,"value":"new"}]],"fields":["/id"]}
```

Expected: 2 retained rows from 3 input rows.

#### Find duplicate records using typed field matching

Numeric 1, string 1 and boolean true stay distinct; the repeated numeric ID is removed.

```json
{"authorizedData":true,"sources":[[{"id":1},{"id":"1"},{"id":1},{"id":true}]],"fields":["/id"]}
```

Expected: 3 retained rows from 4 input rows.

#### Clean duplicate API results for AI workflows

Clean an inline JSON API response before the next automation step. No API fetching or AI model calls.

```json
{"authorizedData":true,"sources":[[{"id":"event-1","value":10},{"id":"event-1","value":10},{"id":"event-2","value":20}]],"fields":["/id"]}
```

Expected: 2 retained rows from 3 input rows.

#### Remove duplicate leads by email and company

Exact email-and-company matching on authorized JSON data. No fuzzy matching, email verification or case normalization.

```json
{"authorizedData":true,"sources":[[{"email":"demo@example.com","company":"North"},{"email":"demo@example.com","company":"South"},{"email":"demo@example.com","company":"North"}]],"fields":["/email","/company"]}
```

Expected: 2 retained rows from 3 input rows.

#### Deduplicate product catalogs by SKU and supplier

Keep the first catalog row for each SKU-and-supplier pair. Does not select the cheapest or most recent price.

```json
{"authorizedData":true,"sources":[[{"sku":"P-10","supplier":"North","price":12},{"sku":"P-10","supplier":"North","price":15},{"sku":"P-10","supplier":"South","price":14}]],"fields":["/sku","/supplier"]}
```

Expected: 2 retained rows from 3 input rows.

### Controlled V1 limits

Inline JSON only, at most ten source arrays; at most 1,000 records and 128 KiB of compact UTF-8 input, including options. Processing deadline 60 seconds. No scraping, fuzzy matching, source-accuracy guarantee or external dataset fetching in this release. Larger batches are outside the controlled experiment.

### Price

$0.05 per non-empty successful batch of up to 1,000 input records. Platform event name: completed-1000-record-block. Empty input: no event charge. No automatic startup or dataset-write event charge. Platform runtime usage is included in product event pricing. Customer storage retention follows Apify account settings; no unlimited archival service is offered.

### Input

```json
{"authorizedData":true,"sources":[[{"id":1,"name":"Number"},{"id":"1","name":"String"},{"id":1,"name":"Duplicate"}]],"fields":["/id"]}
```

This produces two records: numeric 1 and string "1" remain distinct. Omit fields for whole-record matching. Multiple fields create a deterministic composite key, not a concatenated string. Nested fields use JSON Pointer, for example /address/city. Missing policy defaults to distinct; equal and error are explicit options. Null, missing, boolean and string values are distinguished. Numeric 1 and 1.0 compare equal. Object property order is ignored; array order matters.

### Output

RESULT in the run's key-value store is authoritative. Require status complete before consuming items. It includes items, provenance (source\_index,row\_index), duplicates (duplicate and retained references), summary, schema\_version and input\_fingerprint. Default dataset provides a convenient copy of retained items. Duplicate detail reporting is capped, with report\_truncated disclosed; counts remain complete. BILLING separately records confirmed/pending event state; a prepared billing intent is not a payment receipt.

### Data handling

Use only data you are entitled to process. Run, default dataset and key-value store are restricted before processing; authenticated owner access is required. Do not redistribute signed export links: they deliberately grant access. No customer records are sent to an LLM, third-party analysis service or developer-owned aggregation database. Input arrives in Apify storage before the Actor starts; use Restricted general resource access in your Apify account for protection from submission onward. Do not submit credentials or sensitive records in this controlled experiment.

Stores remain unnamed and use Apify's automatic plan-dependent expiry. Free-plan most recent ten runs may persist four months; there is no universal seven-day deletion promise. Export required results before expiry. You can delete your run storage through Apify Console. Naming storage or changing account retention extends retention under your own account settings; SafeDedup does neither.

### Errors and support

Over-limit input: split into batches within both row and byte limits. Insufficient budget: allow $0.05 for a non-empty batch. Missing-field error: correct selectors or choose the documented missing policy. Incomplete output: do not consume the partial dataset; retry only in fresh run storage. Pending billing: do not repeatedly restart or recharge the same run; the transaction preserves the existing intent to avoid duplicate charges. Processing fails closed when privacy, pricing or output validation cannot be confirmed.

This is a controlled market experiment. No production uptime or classification precision percentage is claimed. Marketplace issues are the support channel; do not include private records, tokens or signed export URLs in public issues.

# Actor input Schema

## `authorizedData` (type: `boolean`):

Required. Process only data you own or have permission to use.

## `sources` (type: `array`):

Inline JSON datasets: at most 1,000 total rows and 128 KiB compact UTF-8 input including options. No external fetching.

## `fields` (type: `array`):

Omit for entire-row structural equality. Otherwise1–32 RFC6901 pointers, e.g./company/id.

## `missingPolicy` (type: `string`):

distinct preserves each incomplete row; error fails; equal compares missing only with missing. Null is always different.

## `maxDuplicateDetails` (type: `integer`):

Counts remain complete even when detail output is capped.

## Actor input object example

```json
{
  "authorizedData": true,
  "sources": [
    [
      {
        "id": 1,
        "name": "Number"
      },
      {
        "id": "1",
        "name": "String"
      },
      {
        "id": 1,
        "name": "Duplicate"
      }
    ]
  ],
  "fields": [
    "/id"
  ],
  "missingPolicy": "distinct",
  "maxDuplicateDetails": 1000
}
```

# Actor output Schema

## `items` (type: `string`):

Retained original records. Select JSON for arbitrary input fields. Verify RESULT status complete before consumption.

## `provenance` (type: `string`):

Open RESULT for items, source positions, duplicate links, summary and completion status; BILLING for event state. Authenticated storage access.

## `result` (type: `string`):

Machine-readable authoritative RESULT endpoint; for Console use Provenance and billing instead. Require status complete.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "authorizedData": true,
    "sources": [
        [
            {
                "id": 1,
                "name": "Number"
            },
            {
                "id": "1",
                "name": "String"
            },
            {
                "id": 1,
                "name": "Duplicate"
            }
        ]
    ],
    "fields": [
        "/id"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("vermilion_beauty/my-actor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "authorizedData": True,
    "sources": [[
            {
                "id": 1,
                "name": "Number",
            },
            {
                "id": "1",
                "name": "String",
            },
            {
                "id": 1,
                "name": "Duplicate",
            },
        ]],
    "fields": ["/id"],
}

# Run the Actor and wait for it to finish
run = client.actor("vermilion_beauty/my-actor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "authorizedData": true,
  "sources": [
    [
      {
        "id": 1,
        "name": "Number"
      },
      {
        "id": "1",
        "name": "String"
      },
      {
        "id": 1,
        "name": "Duplicate"
      }
    ]
  ],
  "fields": [
    "/id"
  ]
}' |
apify call vermilion_beauty/my-actor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,vermilion_beauty/my-actor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/R1BkONJroMhx9FEtf/builds/MzK036WLjVxLGejfr/openapi.json