Under maintenance

Pricing

$5.00/month + usage

Try for free

Go to Store

Deduplicator

Under maintenance

Try for free

Developed by

Mustafa Irshaid

Filters out duplicates from previous runs and outputs only new data. Perfect for scheduled scrapers or chained actors to ensure you process fresh results every time. Seamlessly integrates with any Apify actor using default dataset ID.

0.0 (0)

Pricing

$5.00/month + usage

Total users

Monthly users

Runs succeeded

10%

Last modified

4 days ago

Integrations

Developer tools

You can access the Deduplicator programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

1import { ApifyClient } from 'apify-client';
2
3// Initialize the ApifyClient with your Apify API token
4// Replace the '<YOUR_API_TOKEN>' with your token
5const client = new ApifyClient({
6    token: '<YOUR_API_TOKEN>',
7});
8
9// Prepare Actor input
10const input = {};
11
12// Run the Actor and wait for it to finish
13const run = await client.actor("mustafa.irshaid.113/deduplicator").call(input);
14
15// Fetch and print Actor results from the run's dataset (if any)
16console.log('Results from dataset');
17console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
18const { items } = await client.dataset(run.defaultDatasetId).listItems();
19items.forEach((item) => {
20    console.dir(item);
21});
22
23// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

Deduplicator API in JavaScript

The Apify API client for JavaScript is the official library that allows you to use Deduplicator API in JavaScript or TypeScript, providing convenience functions and automatic retries on errors.

Install the apify-client

$npm install apify-client

Other API clients include:

Deduplicator API in Python

Deduplicator API through CLI

Deduplicator OpenAPI definition

Deduplicator API

Apify Actors

prog-party/apify-actors

This Apify Actors Actor retrieves data from Apify, allowing to filter, and returns a list of actors as a Dataset.

Prog Party

Append to dataset

valek.josef/append-to-dataset

Utility actor that allows you to build a single large dataset from individual default datasets of other actor runs.

Josef Válek

Actor Testing

pocesar/actor-testing

Test your actors with varying inputs and expected outputs, duplicates, bad output fields, or unexpected log messages using Jasmine

Paulo Cesar

Failed Runs Monitor

jannovotny/failed-runs-monitor

This actor will let you know about failed or time outed runs of your actors and tasks via Slack or email. It can also notice you about successful runs with empty dataset, check JSON schema of dataset items, or about runs that are running for too long.

Jan Novotný

Scraper Results Checker

drobnikj/check-crawler-results

This actor checks results from Apify's scrapers or any other actor that stores its result to a dataset, and sends a notification if there are errors. It's designed to run from webhook.

Jakub Drobník

Website Changes Detector

tri_angle/website-changes-detector

This actor uses Apify’s Website Content Crawler to track website changes by comparing new and previous crawl results, highlighting only relevant updates to save time and resources.

Tri⟁angle

Apify Actor Request Collector [not made by Apify]

eloquent_mountain/apify-actor-request-collector

Submit your ideas for new Apify scrapers or actors with the Apify Actor Request Collector. Provide suggestions and an optional email to get updates if your requested actor is built. Created by a community member to enhance your Apify experience!

Paco

LLM Dataset Processor

dusan.vystrcil/llm-dataset-processor

Allows you to process output of other actors or stored dataset with single LLM prompt. It's useful if you need to enrich data, summarize content, extract specific information, or manipulate data in a structured way using AI.

Dušan Vystrčil

Clutch.co Search Results Scraper ( Pay Per Results)

gopalakrishnan/clutch-companies-scraper

This Apify actor swiftly scrapes Clutch.co search results for lead generation. Get essential company data—names, URLs, ratings—for $2/1000 results. Integrates seamlessly with enrichment tools. Fast, efficient, and cost-effective!

Gopalakrishnan

115

Results Checker

lukaskrivka/results-checker

Check the results of your scrapers with this flexible checker. Just supply a dataset or key-value store ID and a few simple rules to get a detailed report.