Remove duplicate records and keep their source history

Turn repeated JSON rows into a clean, reproducible dataset. Match on one field or several fields together; see exactly which source row was kept and which duplicate it replaced. The first matching row wins.

$0.05 per completed nonempty batch of up to 1,000 records (also limited to 128 KiB of compact input). No subscription to this Actor. Empty input has no event charge.

Try it with sample data — no dataset preparation

1. Open the ready-to-run email + company example.
2. Use its included synthetic input. Sign in to Apify if prompted, review the charge limit, then start the run.
3. Select Cleaned records, then JSON, to see the two retained records. Select Provenance and billing, then open RESULT to see the duplicate-to-original references and source positions. Require status: "complete" before using results.

A fresh repeat run is a separate execution and may incur another $0.05 charge. Reuse existing output when you do not need another run. Repeating the same ordered input, options and version produces identical deduplication results.

Before and after: remove duplicate leads by email + company

These are synthetic examples, not real leads. All indexes below start at zero.

INPUT / matching key: email + company

{
  "authorizedData": true,
  "sources": [
    [
      {
        "email": "demo@example.com",
        "company": "North"
      },
      {
        "email": "demo@example.com",
        "company": "South"
      },
      {
        "email": "demo@example.com",
        "company": "North"
      }
    ]
  ],
  "fields": [
    "/email",
    "/company"
  ]
}

OUTPUT / excerpt from RESULT

{
  "items": [
    {
      "email": "demo@example.com",
      "company": "North"
    },
    {
      "email": "demo@example.com",
      "company": "South"
    }
  ],
  "provenance": [
    {
      "source_index": 0,
      "row_index": 0
    },
    {
      "source_index": 0,
      "row_index": 1
    }
  ],
  "duplicates": [
    {
      "duplicate": {
        "source_index": 0,
        "row_index": 2
      },
      "retained": {
        "source_index": 0,
        "row_index": 0
      }
    }
  ],
  "summary": {
    "input_count": 3,
    "unique_count": 2,
    "duplicate_count": 1,
    "report_truncated": false
  }
}

The North duplicate at row 2 points back to North at row 0. South remains because the company differs. provenance follows the order of retained items; duplicates links each removed row to its retained original. No hidden merging or fuzzy matching.

Price, limits and charge protection

One completed nonempty batch costs $0.05, whether it contains 3 or 1,000 input records. Both the 1,000-record and 128-KiB limits apply. Runtime usage is included in the Actor event price. Set the run's maximum total charge to $0.05 for this single-batch example; a lower budget rejects nonempty input before output. Empty input produces no billed event. Input or processing failures before the charge step produce no completed-batch event. A later interruption can leave billing pending: inspect BILLING before retrying; do not assume every failed run is uncharged. Apify account storage/retention rules still apply.

Automation contract

• INPUT: authorized inline JSON source arrays; optional exact-match field pointers. Multiple pointers form one composite key.
• OUTPUT: cleaned dataset plus authoritative RESULT with items, provenance, duplicates, summary, schema_version, input_fingerprint and status.
• PRICE: $0.05 per completed nonempty batch; zero event charge for empty input.
• LIMITS: 1,000 rows, 128 KiB compact input, ten source arrays and 60-second processing deadline.
• USE CASES: duplicate leads, product catalogs, repeated API results and reproducible agent workflows.
• ERROR BEHAVIOR: invalid/over-limit input or an unverified privacy/output condition fails closed; consume only complete RESULT. See error and billing guidance below.
• EXAMPLE CALL: submit the complete JSON INPUT above in the Actor or saved task's input form. The Apify API button supplies authenticated API/client examples; keep your token private.
• EXAMPLE RESPONSE: the OUTPUT excerpt above shows deduplication data; the actual RESULT also includes completion status and version/fingerprint metadata.

CSV limitation: to deduplicate CSV by email and company, first convert the CSV into JSON objects, then use the example above. This release has no CSV-file upload, dataset-ID import or API-URL fetching. Exact matching does not lowercase, trim or validate contact details. Numeric and string IDs remain distinct.

See the result first

Single field: [{"id":1},{"id":1},{"id":2}] with fields:["/id"] becomes [{"id":1},{"id":2}].

Composite key: SKU A + supplier North is distinct from SKU A + supplier South. A second A/North row is removed; the first row wins.

Typed matching: [{"id":1},{"id":"1"},{"id":1},{"id":true}] becomes [{"id":1},{"id":"1"},{"id":true}]. JSON numbers 1 and 1.0 compare equal.

Provenance for the single-field example:

{"provenance":[{"source_index":0,"row_index":0},{"source_index":0,"row_index":2}],"duplicates":[{"duplicate":{"source_index":0,"row_index":1},"retained":{"source_index":0,"row_index":0}}]}

Run the same ordered input and options again with the same product version: retained records, order, provenance and duplicate counts are identical. Changing input order can change which first record is retained. Run IDs and billing receipts are separate operational metadata.

Copy a use case

Each JSON block is complete input; replace synthetic rows with your authorized data. Exact matching does not lowercase, trim, verify contacts or select the newest record.

Remove duplicate JSON records with composite keys

Match both SKU and supplier; identical SKUs from different suppliers stay separate.

{"authorizedData":true,"sources":[[{"sku":"A","supplier":"North"},{"sku":"A","supplier":"South"},{"sku":"A","supplier":"North"}]],"fields":["/sku","/supplier"]}

Expected: 2 retained rows from 3 input rows.

Deduplicate data while preserving source provenance

Combine two inline sources and trace retained and removed rows to source and row indexes.

{"authorizedData":true,"sources":[[{"id":1,"value":"first"}],[{"id":1,"value":"later"},{"id":2,"value":"new"}]],"fields":["/id"]}

Expected: 2 retained rows from 3 input rows.

Find duplicate records using typed field matching

Numeric 1, string 1 and boolean true stay distinct; the repeated numeric ID is removed.

{"authorizedData":true,"sources":[[{"id":1},{"id":"1"},{"id":1},{"id":true}]],"fields":["/id"]}

Expected: 3 retained rows from 4 input rows.

Clean duplicate API results for AI workflows

Clean an inline JSON API response before the next automation step. No API fetching or AI model calls.

{"authorizedData":true,"sources":[[{"id":"event-1","value":10},{"id":"event-1","value":10},{"id":"event-2","value":20}]],"fields":["/id"]}

Expected: 2 retained rows from 3 input rows.

Remove duplicate leads by email and company

Exact email-and-company matching on authorized JSON data. No fuzzy matching, email verification or case normalization.

{"authorizedData":true,"sources":[[{"email":"demo@example.com","company":"North"},{"email":"demo@example.com","company":"South"},{"email":"demo@example.com","company":"North"}]],"fields":["/email","/company"]}

Expected: 2 retained rows from 3 input rows.

Deduplicate product catalogs by SKU and supplier

Keep the first catalog row for each SKU-and-supplier pair. Does not select the cheapest or most recent price.

{"authorizedData":true,"sources":[[{"sku":"P-10","supplier":"North","price":12},{"sku":"P-10","supplier":"North","price":15},{"sku":"P-10","supplier":"South","price":14}]],"fields":["/sku","/supplier"]}

Expected: 2 retained rows from 3 input rows.

Controlled V1 limits

Inline JSON only, at most ten source arrays; at most 1,000 records and 128 KiB of compact UTF-8 input, including options. Processing deadline 60 seconds. No scraping, fuzzy matching, source-accuracy guarantee or external dataset fetching in this release. Larger batches are outside the controlled experiment.

Price

$0.05 per non-empty successful batch of up to 1,000 input records. Platform event name: completed-1000-record-block. Empty input: no event charge. No automatic startup or dataset-write event charge. Platform runtime usage is included in product event pricing. Customer storage retention follows Apify account settings; no unlimited archival service is offered.

Input

{"authorizedData":true,"sources":[[{"id":1,"name":"Number"},{"id":"1","name":"String"},{"id":1,"name":"Duplicate"}]],"fields":["/id"]}

This produces two records: numeric 1 and string "1" remain distinct. Omit fields for whole-record matching. Multiple fields create a deterministic composite key, not a concatenated string. Nested fields use JSON Pointer, for example /address/city. Missing policy defaults to distinct; equal and error are explicit options. Null, missing, boolean and string values are distinguished. Numeric 1 and 1.0 compare equal. Object property order is ignored; array order matters.

Output

RESULT in the run's key-value store is authoritative. Require status complete before consuming items. It includes items, provenance (source_index,row_index), duplicates (duplicate and retained references), summary, schema_version and input_fingerprint. Default dataset provides a convenient copy of retained items. Duplicate detail reporting is capped, with report_truncated disclosed; counts remain complete. BILLING separately records confirmed/pending event state; a prepared billing intent is not a payment receipt.

Data handling

Use only data you are entitled to process. Run, default dataset and key-value store are restricted before processing; authenticated owner access is required. Do not redistribute signed export links: they deliberately grant access. No customer records are sent to an LLM, third-party analysis service or developer-owned aggregation database. Input arrives in Apify storage before the Actor starts; use Restricted general resource access in your Apify account for protection from submission onward. Do not submit credentials or sensitive records in this controlled experiment.

Stores remain unnamed and use Apify's automatic plan-dependent expiry. Free-plan most recent ten runs may persist four months; there is no universal seven-day deletion promise. Export required results before expiry. You can delete your run storage through Apify Console. Naming storage or changing account retention extends retention under your own account settings; SafeDedup does neither.

Errors and support

Over-limit input: split into batches within both row and byte limits. Insufficient budget: allow $0.05 for a non-empty batch. Missing-field error: correct selectors or choose the documented missing policy. Incomplete output: do not consume the partial dataset; retry only in fresh run storage. Pending billing: do not repeatedly restart or recharge the same run; the transaction preserves the existing intent to avoid duplicate charges. Processing fails closed when privacy, pricing or output validation cannot be confirmed.

This is a controlled market experiment. No production uptime or classification precision percentage is claimed. Marketplace issues are the support channel; do not include private records, tokens or signed export URLs in public issues.