Pricing

from $2.00 / 1,000 dedup runs

Try for free

Go to Apify Store

Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places

Try for free

Scheduled scrapes keep returning the same jobs, leads and places. Feed any Apify dataset, CSV or JSON and get only unique rows: key-field and fuzzy near-duplicate matching, URL tracking cleanup, and optional memory so each run returns only NEW items. Streams 1M rows.

Pricing

from $2.00 / 1,000 dedup runs

Rating

0.0

(0)

Developer

mohamed alaya

Actor stats

Bookmarked

Total users

Monthly active users

16 days ago

Last modified

Categories

Developer tools

Automation

Apify datasets

datasetIds

Optional

Datasets (pick from your storage or pass dataset IDs) to clean, for example the dataset of a job, Google Maps or product scraper run. Several datasets are merged and de-duplicated together, which removes duplicates across runs and sources. Streams page by page, so 1M rows work. Use the picker rather than typing an ID: it gives this Actor permission to read your dataset.

Type:array

Rows (JSON)

records

Optional

Optional: rows pasted directly as an array of objects. Combined with the datasets above. The default is a small job-board sample so an empty run shows real output.

Type:array

Default:

[
  {
    "title": "Senior Data Engineer",
    "company": "Northwind Analytics",
    "location": "Austin, TX",
    "url": "https://www.linkedin.com/jobs/view/senior-data-engineer-at-northwind-analytics-3912345678?refId=abc&trackingId=xyz",
    "salary": "$140k-$170k",
    "postedAt": "2026-10-01",
    "source": "linkedin",
    "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform."
  },
  {
    "title": "Senior Data Engineer",
    "company": "Northwind Analytics",
    "location": "Austin, TX",
    "url": "https://www.linkedin.com/jobs/view/3912345678/?utm_source=daily-scrape",
    "salary": "$140k-$170k",
    "postedAt": "2026-10-02",
    "source": "linkedin",
    "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform."
  },
  {
    "title": "Senior Data Engineer",
    "company": "Northwind Analytics",
    "location": "Austin, TX",
    "url": "https://careers.northwind-analytics.com/jobs/4471",
    "source": "company site",
    "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform. Remote friendly."
  },
  {
    "title": "Product Designer",
    "company": "Lumen Labs",
    "location": "Remote (US)",
    "url": "https://boards.greenhouse.io/lumenlabs/jobs/5501234?gh_src=abc",
    "salary": "$110k-$130k",
    "postedAt": "2026-09-29",
    "source": "greenhouse",
    "description": "Design end-to-end product experiences for our B2B analytics suite, from research to polished UI."
  },
  {
    "title": "Product Designer",
    "company": "Lumen Labs",
    "location": "Remote (US)",
    "url": "https://boards.greenhouse.io/lumenlabs/jobs/5501234",
    "salary": "$110k-$130k",
    "postedAt": "2026-10-02",
    "source": "greenhouse",
    "description": "Design end-to-end product experiences for our B2B analytics suite, from research to polished UI."
  },
  {
    "title": "Customer Success Manager",
    "company": "Brightpath",
    "location": "Denver, CO",
    "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f60718&utm_campaign=x",
    "salary": "$85k-$100k",
    "postedAt": "2026-10-01",
    "source": "indeed",
    "description": "Own onboarding and renewals for a book of mid-market SaaS accounts and be the voice of the customer."
  },
  {
    "title": "Customer Success Manager",
    "company": "Brightpath",
    "location": "Denver, CO",
    "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f60718",
    "salary": "$85k-$100k",
    "postedAt": "2026-10-03",
    "source": "indeed",
    "description": "Own onboarding and renewals for a book of mid-market SaaS accounts and be the voice of the customer."
  },
  {
    "title": "DevOps Engineer",
    "company": "Cobalt Systems",
    "location": "Chicago, IL",
    "url": "https://jobs.lever.co/cobalt/8f2e1c77-aaaa-bbbb-cccc-0123456789ab",
    "salary": "$130k-$150k",
    "postedAt": "2026-09-30",
    "source": "lever",
    "description": "Run our Kubernetes platform, improve CI/CD and own on-call tooling for a 40-engineer organisation."
  },
  {
    "title": "Registered Nurse - ICU",
    "company": "St. Mary Health",
    "location": "Portland, OR",
    "url": "https://www.stmaryhealth.org/careers/rn-icu-2291",
    "salary": "$48-$62/hr",
    "postedAt": "2026-10-02",
    "source": "company site",
    "description": "Provide bedside care in a 24-bed ICU, 12-hour night shifts, sign-on bonus available."
  },
  {
    "title": "Registered Nurse - ICU",
    "company": "St. Mary Health",
    "location": "Portland, OR",
    "url": "https://www.indeed.com/viewjob?jk=ff00112233445566",
    "salary": "$48-$62/hr",
    "postedAt": "2026-10-03",
    "source": "indeed",
    "description": "Provide bedside care in a 24-bed ICU, 12-hour night shifts, sign-on bonus available."
  },
  {
    "title": "Marketing Coordinator",
    "company": "Harbor & Pine",
    "location": "Boston, MA",
    "url": "https://www.linkedin.com/jobs/view/3955550001/",
    "salary": "$62k-$72k",
    "postedAt": "2026-10-01",
    "source": "linkedin",
    "description": "Support campaign execution, manage the content calendar and report weekly on channel performance."
  },
  {
    "title": "Backend Engineer (Go)",
    "company": "Quickstream",
    "location": "Remote",
    "url": "https://jobs.ashbyhq.com/quickstream/9c1d3a52-1111-2222-3333-444455556666",
    "salary": "$150k-$180k",
    "postedAt": "2026-10-02",
    "source": "ashby",
    "description": "Design low-latency payment APIs in Go, with strong ownership of reliability and observability."
  },
  {
    "title": "Backend Engineer (Go)",
    "company": "Quickstream",
    "location": "Remote",
    "url": "https://www.linkedin.com/jobs/view/3955559999/?refId=1",
    "salary": "$150k-$180k",
    "postedAt": "2026-10-03",
    "source": "linkedin",
    "description": "Design low-latency payment APIs in Go, with strong ownership of reliability and observability."
  },
  {
    "title": "Accountant",
    "company": "Fieldstone & Co",
    "location": "Atlanta, GA",
    "url": "https://www.indeed.com/viewjob?jk=1122334455667788",
    "salary": "$70k-$85k",
    "postedAt": "2026-10-01",
    "source": "indeed",
    "description": "Prepare monthly close, reconcile accounts and support audits for a growing professional services firm."
  },
  {
    "title": "Sales Development Representative",
    "company": "Orbit CRM",
    "location": "Miami, FL",
    "url": "https://boards.greenhouse.io/orbitcrm/jobs/6600777",
    "salary": "$55k-$65k + commission",
    "postedAt": "2026-10-03",
    "source": "greenhouse",
    "description": "Prospect, qualify and book meetings for our enterprise account executives."
  }
]

CSV or JSON file URL

sourceUrl

Optional

Optional: a public URL to a CSV, JSON or NDJSON file. Leave empty to skip.

Type:string

Default:

Pasted CSV text

csvText

Optional

Optional: CSV text with a header row (comma, semicolon or tab separated). Leave empty to skip.

Type:string

Default:

How to find duplicates

mode

Optional

keys = rows are duplicates when their key fields match (fast, exact, the default). wholeRecord = only fully identical rows. fuzzy = key matching PLUS near-duplicate text matching (reworded or lightly edited copies).

Type:string

Default:keys

Options:

keyswholeRecordfuzzy

Key preset

preset

Optional

Which columns identify a row when "Key fields" is empty. Auto-detect looks at your columns and picks jobs, places, products, leads or posts keys. For example jobs: same url OR same title + company + location.

Type:string

Default:auto

Options:

autojobsplacesproductsleadsposts

Key fields (custom)

keyFields

Optional

Optional override. One entry per key: a column name ("url"), several alternatives ("url|link"), or a combination joined with + ("title+company+location"). A row is a duplicate if ANY entry matches an earlier row. Column names are case-insensitive and nested paths like address.city work.

Type:array

Default:

[]

Text fields for fuzzy matching

fuzzyFields

Optional

Fuzzy mode only. Columns whose text is compared for near-duplicates. Leave empty to auto-pick the longest text columns (ids, urls, dates and prices are ignored).

Type:array

Default:

[]

Fuzzy similarity threshold (0-100)

similarityThreshold

Optional

Fuzzy mode only. Rows whose text is at least this similar (Jaccard of word or character shingles) are treated as duplicates. 85 catches re-posts and light rewording; raise it to be stricter.

Type:integer

Minimum:30

Maximum:100

Default:85

Which copy to keep

keepPolicy

Optional

first = the earliest row of each duplicate group. last = the most recent row. richest = the row with the most filled-in fields.

Type:string

Default:first

Options:

firstlastrichest

Dataset contents

outputMode

Optional

kept = only the unique rows, unchanged (ready to use). removed = only the removed duplicates with the reason and which row they duplicate. all = both, each tagged with _dedupeStatus. A REMOVED_DUPLICATES summary and run SUMMARY are always saved to the key-value store.

Type:string

Default:kept

Options:

keptremovedall

Return only NEW rows across runs

rememberAcrossRuns

Optional

Turn on for scheduled scrapes: the Actor remembers the keys of rows it already delivered (in a named key-value store) and returns only rows it has not seen before. Use the same "Memory name" on every scheduled run.

Type:boolean

Default:false

Memory name

memoryName

Optional

Name of the remembered key set (letters, digits, dashes). Use a different name per scraper or search so their histories do not mix.

Type:string

Default:default

Case-sensitive keys

caseSensitive

Optional

When off, "Acme Inc" and "ACME INC" produce the same key. Accents and extra spaces are always ignored.

Type:boolean

Default:false

Clean URLs before comparing

normalizeUrls

Optional

Strip tracking parameters (utm_*, fbclid, gclid, refId, trackingId...), "www.", trailing slashes and, for LinkedIn job links, reduce to the job id, so the same posting with different tracking junk matches.

Type:boolean

Default:true

Ignore fields (whole-record mode)

ignoreFields

Optional

Whole-record mode only. Columns left out of the comparison, e.g. scrapedAt or crawlId, which differ on every run. Fields starting with "_" are always ignored.

Type:array

Default:

[]

Shingle size (fuzzy, advanced)

shingleSize

Optional

Fuzzy mode only. Length of the word groups (long text) or character groups (short text) used to compare texts. 3 works well for most data.

Type:integer

Minimum:2

Maximum:8

Default:3

Max rows in removal report

maxRemovedReport

Optional

How many removed rows are listed in the REMOVED_DUPLICATES key-value record (row number, which row it duplicates, why). Does not limit the dataset.

Type:integer

Minimum:0

Maximum:200000

Default:10000

Max remembered keys

maxMemoryKeys

Optional

Cap for the cross-run memory. When exceeded, the oldest remembered keys are dropped.

Type:integer

Minimum:1000

Maximum:20000000

Default:3000000

Max rows to process

maxRecords

Optional

Upper limit of rows read from all sources. Memory use is roughly 100 bytes per row (about 260 with fuzzy mode), so give the run 2-4 GB for a million rows.

Type:integer

Minimum:1

Maximum:10000000

Default:2000000

Image Background Remover - Keamind

stefanie-rink/image-background-remover

Remove image backgrounds instantly! Our tool uses AI to cut out objects & create transparent PNGs within a few seconds. Easy to use.

Steafanie Braid

Instagram LIST / REMOVE Followings

saswave/instagram-list-remove-followings

List all the accounts you are following for data analysis. Then provide a list of following to be removed. Help you clean your instagram feed page by removing bot accounts and not active followers

SASWAVE

Monitoring Checker Duplication

apify/monitoring-checker-duplication

The Monitoring checker duplication is a part of the Apify Monitoring Suite (apify/monitoring). See its readme for more information and how to use this.

Apify

5.0

(5)

background remover

ainz/background-remover

Ai powered background remover, fast with high quality

Elmehdi felaouss

Obsidian Scraping Tool

uasd40/obsidian-scraping-tool

Connects Obsidian vaults with web scraping via MCP. Auto-converts scraped content into formatted notes with tags and templates.

Simon Breidenbach

AI Prompt Keyword Matcher

antonio_espresso/ai-prompt-keyword-matcher

Analyze prompts for fuzzy keyword matches and brand token usage.

Antonio Blago

Image Watermark Remover - Clean Photos in Bulk

seemuapps/image-watermark-remover

Remove text and logo watermarks from images using. Preserves texture and background for natural, artifact-free results.

Seemu Scraping

Opening Hours to JSON - Normalize Google Maps Business Hours

that_red_bird/opening-hours-normalizer

Google Maps and Yelp give you hours as messy text. Pick the scraper's dataset and get weekly JSON, schema.org openingHoursSpecification and an open-now flag per place, with 24/7, closed and split-shift handling and day names in many languages. No re-scraping.

mohamed alaya

PCI DSS 6.4.3 Script Inventory - Payment Page Script Check

that_red_bird/pci-payment-page-scripts

PCI DSS 4.0 asks for an inventory of every script on your payment pages. This Actor loads your checkout URLs, lists each script (origin, SRI, SHA-256, inline or external) and diffs it against the last run as tamper evidence. An inventory and change check, not real-user monitoring.

mohamed alaya

Aggregate Fields

pocesar/aggregate-fields

Create an overview of a dataset by aggregating the possible variations from the selected fields. Useful for checking the consistency of data used together with the Results Checker actor.