Pricing

from $2.00 / 1,000 dedup runs

Try for free

Go to Apify Store

Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places

Try for free

Scheduled scrapes keep returning the same jobs, leads and places. Feed any Apify dataset, CSV or JSON and get only unique rows: key-field and fuzzy near-duplicate matching, URL tracking cleanup, and optional memory so each run returns only NEW items. Streams 1M rows.

Pricing

from $2.00 / 1,000 dedup runs

Rating

0.0

(0)

Developer

mohamed alaya

Actor stats

Bookmarked

Total users

Monthly active users

17 days ago

Last modified

Categories

Developer tools

Automation

You can access the Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in API & Integrations in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

# Set API token
$API_TOKEN=<YOUR_API_TOKEN>

# Prepare Actor input
$cat > input.json << 'EOF'
<{
<  "records": [
<    {
<      "title": "Senior Data Engineer",
<      "company": "Northwind Analytics",
<      "location": "Austin, TX",
<      "url": "https://www.linkedin.com/jobs/view/senior-data-engineer-at-northwind-analytics-3912345678?refId=abc&trackingId=xyz",
<      "salary": "$140k-$170k",
<      "postedAt": "2026-10-01",
<      "source": "linkedin",
<      "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform."
<    },
<    {
<      "title": "Senior Data Engineer",
<      "company": "Northwind Analytics",
<      "location": "Austin, TX",
<      "url": "https://www.linkedin.com/jobs/view/3912345678/?utm_source=daily-scrape",
<      "salary": "$140k-$170k",
<      "postedAt": "2026-10-02",
<      "source": "linkedin",
<      "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform."
<    },
<    {
<      "title": "Senior Data Engineer",
<      "company": "Northwind Analytics",
<      "location": "Austin, TX",
<      "url": "https://careers.northwind-analytics.com/jobs/4471",
<      "source": "company site",
<      "description": "Build and maintain batch and streaming pipelines on Spark and Airflow. Own data quality for the revenue analytics platform. Remote friendly."
<    },
<    {
<      "title": "Product Designer",
<      "company": "Lumen Labs",
<      "location": "Remote (US)",
<      "url": "https://boards.greenhouse.io/lumenlabs/jobs/5501234?gh_src=abc",
<      "salary": "$110k-$130k",
<      "postedAt": "2026-09-29",
<      "source": "greenhouse",
<      "description": "Design end-to-end product experiences for our B2B analytics suite, from research to polished UI."
<    },
<    {
<      "title": "Product Designer",
<      "company": "Lumen Labs",
<      "location": "Remote (US)",
<      "url": "https://boards.greenhouse.io/lumenlabs/jobs/5501234",
<      "salary": "$110k-$130k",
<      "postedAt": "2026-10-02",
<      "source": "greenhouse",
<      "description": "Design end-to-end product experiences for our B2B analytics suite, from research to polished UI."
<    },
<    {
<      "title": "Customer Success Manager",
<      "company": "Brightpath",
<      "location": "Denver, CO",
<      "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f60718&utm_campaign=x",
<      "salary": "$85k-$100k",
<      "postedAt": "2026-10-01",
<      "source": "indeed",
<      "description": "Own onboarding and renewals for a book of mid-market SaaS accounts and be the voice of the customer."
<    },
<    {
<      "title": "Customer Success Manager",
<      "company": "Brightpath",
<      "location": "Denver, CO",
<      "url": "https://www.indeed.com/viewjob?jk=a1b2c3d4e5f60718",
<      "salary": "$85k-$100k",
<      "postedAt": "2026-10-03",
<      "source": "indeed",
<      "description": "Own onboarding and renewals for a book of mid-market SaaS accounts and be the voice of the customer."
<    },
<    {
<      "title": "DevOps Engineer",
<      "company": "Cobalt Systems",
<      "location": "Chicago, IL",
<      "url": "https://jobs.lever.co/cobalt/8f2e1c77-aaaa-bbbb-cccc-0123456789ab",
<      "salary": "$130k-$150k",
<      "postedAt": "2026-09-30",
<      "source": "lever",
<      "description": "Run our Kubernetes platform, improve CI/CD and own on-call tooling for a 40-engineer organisation."
<    },
<    {
<      "title": "Registered Nurse - ICU",
<      "company": "St. Mary Health",
<      "location": "Portland, OR",
<      "url": "https://www.stmaryhealth.org/careers/rn-icu-2291",
<      "salary": "$48-$62/hr",
<      "postedAt": "2026-10-02",
<      "source": "company site",
<      "description": "Provide bedside care in a 24-bed ICU, 12-hour night shifts, sign-on bonus available."
<    },
<    {
<      "title": "Registered Nurse - ICU",
<      "company": "St. Mary Health",
<      "location": "Portland, OR",
<      "url": "https://www.indeed.com/viewjob?jk=ff00112233445566",
<      "salary": "$48-$62/hr",
<      "postedAt": "2026-10-03",
<      "source": "indeed",
<      "description": "Provide bedside care in a 24-bed ICU, 12-hour night shifts, sign-on bonus available."
<    },
<    {
<      "title": "Marketing Coordinator",
<      "company": "Harbor & Pine",
<      "location": "Boston, MA",
<      "url": "https://www.linkedin.com/jobs/view/3955550001/",
<      "salary": "$62k-$72k",
<      "postedAt": "2026-10-01",
<      "source": "linkedin",
<      "description": "Support campaign execution, manage the content calendar and report weekly on channel performance."
<    },
<    {
<      "title": "Backend Engineer (Go)",
<      "company": "Quickstream",
<      "location": "Remote",
<      "url": "https://jobs.ashbyhq.com/quickstream/9c1d3a52-1111-2222-3333-444455556666",
<      "salary": "$150k-$180k",
<      "postedAt": "2026-10-02",
<      "source": "ashby",
<      "description": "Design low-latency payment APIs in Go, with strong ownership of reliability and observability."
<    },
<    {
<      "title": "Backend Engineer (Go)",
<      "company": "Quickstream",
<      "location": "Remote",
<      "url": "https://www.linkedin.com/jobs/view/3955559999/?refId=1",
<      "salary": "$150k-$180k",
<      "postedAt": "2026-10-03",
<      "source": "linkedin",
<      "description": "Design low-latency payment APIs in Go, with strong ownership of reliability and observability."
<    },
<    {
<      "title": "Accountant",
<      "company": "Fieldstone & Co",
<      "location": "Atlanta, GA",
<      "url": "https://www.indeed.com/viewjob?jk=1122334455667788",
<      "salary": "$70k-$85k",
<      "postedAt": "2026-10-01",
<      "source": "indeed",
<      "description": "Prepare monthly close, reconcile accounts and support audits for a growing professional services firm."
<    },
<    {
<      "title": "Sales Development Representative",
<      "company": "Orbit CRM",
<      "location": "Miami, FL",
<      "url": "https://boards.greenhouse.io/orbitcrm/jobs/6600777",
<      "salary": "$55k-$65k + commission",
<      "postedAt": "2026-10-03",
<      "source": "greenhouse",
<      "description": "Prospect, qualify and book meetings for our enterprise account executives."
<    }
<  ],
<  "sourceUrl": "",
<  "csvText": "",
<  "mode": "keys",
<  "preset": "auto",
<  "keyFields": [],
<  "fuzzyFields": [],
<  "similarityThreshold": 85,
<  "keepPolicy": "first",
<  "outputMode": "kept",
<  "rememberAcrossRuns": false,
<  "memoryName": "default",
<  "caseSensitive": false,
<  "normalizeUrls": true,
<  "ignoreFields": [],
<  "shingleSize": 3,
<  "maxRemovedReport": 10000,
<  "maxMemoryKeys": 3000000,
<  "maxRecords": 2000000
<}
<EOF

# Run the Actor using an HTTP API
# See the full API reference at https://docs.apify.com/api/v2
$curl "https://api.apify.com/v2/actors/that_red_bird~remove-duplicate-scraped-data/runs?token=$API_TOKEN" \
<  -X POST \
<  -d @input.json \
<  -H 'Content-Type: application/json'

Remove Duplicates from Any Apify Dataset API

Below, you can find a list of relevant HTTP API endpoints for calling the Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places Actor. For this, you’ll need an Apify account. Replace <YOUR_API_TOKEN> in the URLs with your Apify API token, which you can find under API & Integrations in Apify Console. For details, see the API reference.

Run Actor

POST
https://api.apify.com/v2/actors/that_red_bird~remove-duplicate-scraped-data/runs?token=<YOUR_API_TOKEN>

Note: By adding the method=POST query parameter, this API endpoint can be called using a GET request and thus used in third-party webhooks. Please refer to our Run Actor API documentation.

Run Actor synchronously and get dataset items

POST
https://api.apify.com/v2/actors/that_red_bird~remove-duplicate-scraped-data/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>

Note: This endpoint supports both POST and GET request methods. However, only the POST method allows you to pass input data. For more information, please refer to our Run Actor synchronously and get dataset items API documentation.

Get Actor

GET
https://api.apify.com/v2/actors/that_red_bird~remove-duplicate-scraped-data?token=<YOUR_API_TOKEN>

For more information, please refer to our Get Actor API documentation.

Actors can be used to scrape web pages, extract data, or automate browser tasks. Use the Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places API programmatically via the Apify API.

You can choose from:

Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places API in Python

Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places API in JavaScript

Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places API through CLI

Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places OpenAPI definition

You can start Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places with the Apify API by sending an HTTP POST request to the Run Actorendpoint. An Actor’s input and its content type can be passed as a payload of the POST request, and additional options can be specified using URL query parameters. The Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places is identified within the API by its ID, which is the creator’s username and the name of the Actor.

When the Remove Duplicates from Any Apify Dataset - Jobs, Leads, Places run finishes you can list the data from its default dataset(storage) via the API or you can preview the data directly on Apify Console.

Image Background Remover - Keamind

stefanie-rink/image-background-remover

Remove image backgrounds instantly! Our tool uses AI to cut out objects & create transparent PNGs within a few seconds. Easy to use.

Steafanie Braid

Instagram LIST / REMOVE Followings

saswave/instagram-list-remove-followings

List all the accounts you are following for data analysis. Then provide a list of following to be removed. Help you clean your instagram feed page by removing bot accounts and not active followers

SASWAVE

Monitoring Checker Duplication

apify/monitoring-checker-duplication

The Monitoring checker duplication is a part of the Apify Monitoring Suite (apify/monitoring). See its readme for more information and how to use this.

Apify

5.0

(5)

background remover

ainz/background-remover

Ai powered background remover, fast with high quality

Elmehdi felaouss

Obsidian Scraping Tool

uasd40/obsidian-scraping-tool

Connects Obsidian vaults with web scraping via MCP. Auto-converts scraped content into formatted notes with tags and templates.

Simon Breidenbach

AI Prompt Keyword Matcher

antonio_espresso/ai-prompt-keyword-matcher

Analyze prompts for fuzzy keyword matches and brand token usage.

Antonio Blago

Image Watermark Remover - Clean Photos in Bulk

seemuapps/image-watermark-remover

Remove text and logo watermarks from images using. Preserves texture and background for natural, artifact-free results.

Seemu Scraping

Opening Hours to JSON - Normalize Google Maps Business Hours

that_red_bird/opening-hours-normalizer

Google Maps and Yelp give you hours as messy text. Pick the scraper's dataset and get weekly JSON, schema.org openingHoursSpecification and an open-now flag per place, with 24/7, closed and split-shift handling and day names in many languages. No re-scraping.

mohamed alaya

PCI DSS 6.4.3 Script Inventory - Payment Page Script Check

that_red_bird/pci-payment-page-scripts

PCI DSS 4.0 asks for an inventory of every script on your payment pages. This Actor loads your checkout URLs, lists each script (origin, SRI, SHA-256, inline or external) and diffs it against the last run as tamper evidence. An inventory and change check, not real-user monitoring.

mohamed alaya

Aggregate Fields

pocesar/aggregate-fields

Create an overview of a dataset by aggregating the possible variations from the selected fields. Useful for checking the consistency of data used together with the Results Checker actor.