Under maintenance

Pricing

$5.00/month + usage

Try for free

Go to Store

Deduplicator

Under maintenance

Try for free

Developed by

Mustafa Irshaid

Filters out duplicates from previous runs and outputs only new data. Perfect for scheduled scrapers or chained actors to ensure you process fresh results every time. Seamlessly integrates with any Apify actor using default dataset ID.

0.0 (0)

Pricing

$5.00/month + usage

Total users

Monthly users

Runs succeeded

10%

Last modified

4 days ago

Integrations

Developer tools

You can access the Deduplicator programmatically from your own applications by using the Apify API. You can also choose the language preference from below. To use the Apify API, you’ll need an Apify account and your API token, found in Integrations settings in Apify Console.

Python

JavaScript

CLI

OpenAPI

HTTP

MCP

{
    "mcpServers": {
        "local-actors-mcp-server": {
            "command": "npx",
            "args": [
                "-y",
                "@apify/actors-mcp-server",
                "--actors",
                "mustafa.irshaid.113/deduplicator"
            ],
            "env": {
                "APIFY_TOKEN": "<YOUR_API_TOKEN>"
            }
        }
    }
}

Configure MCP server with Deduplicator

You can interact with the MCP server via standard input/output - stdio (as shown above), which is ideal for local integrations and command-line tools such as the Claude desktop client, or you can interact with the server through Server-Sent Events (SSE) to send messages and receive responses, which looks as follows:

{
    "mcpServers": {
        "remote-actors-mcp-server": {
            "type": "sse",
            "url": "https://mcp.apify.com/sse?actors=mustafa.irshaid.113/deduplicator",
            "headers": {
                "Authorization": "Bearer <YOUR_API_TOKEN>"
            }
        }
    }
}

You can connect to the Apify MCP Server using clients like Tester MCP Client, or any other supported MCP client of your choice.

If you want to learn more about our Apify MCP implementation, check out our MCP documentation. To learn more about the Model Context Protocol in general, refer to the official MCP documentation or read our blog post.

Apify Actors

prog-party/apify-actors

This Apify Actors Actor retrieves data from Apify, allowing to filter, and returns a list of actors as a Dataset.

Prog Party

Append to dataset

valek.josef/append-to-dataset

Utility actor that allows you to build a single large dataset from individual default datasets of other actor runs.

Josef Válek

Actor Testing

pocesar/actor-testing

Test your actors with varying inputs and expected outputs, duplicates, bad output fields, or unexpected log messages using Jasmine

Paulo Cesar

Failed Runs Monitor

jannovotny/failed-runs-monitor

This actor will let you know about failed or time outed runs of your actors and tasks via Slack or email. It can also notice you about successful runs with empty dataset, check JSON schema of dataset items, or about runs that are running for too long.

Jan Novotný

Scraper Results Checker

drobnikj/check-crawler-results

This actor checks results from Apify's scrapers or any other actor that stores its result to a dataset, and sends a notification if there are errors. It's designed to run from webhook.

Jakub Drobník

Website Changes Detector

tri_angle/website-changes-detector

This actor uses Apify’s Website Content Crawler to track website changes by comparing new and previous crawl results, highlighting only relevant updates to save time and resources.

Tri⟁angle

Apify Actor Request Collector [not made by Apify]

eloquent_mountain/apify-actor-request-collector

Submit your ideas for new Apify scrapers or actors with the Apify Actor Request Collector. Provide suggestions and an optional email to get updates if your requested actor is built. Created by a community member to enhance your Apify experience!

Paco

LLM Dataset Processor

dusan.vystrcil/llm-dataset-processor

Allows you to process output of other actors or stored dataset with single LLM prompt. It's useful if you need to enrich data, summarize content, extract specific information, or manipulate data in a structured way using AI.

Dušan Vystrčil

Clutch.co Search Results Scraper ( Pay Per Results)

gopalakrishnan/clutch-companies-scraper

This Apify actor swiftly scrapes Clutch.co search results for lead generation. Get essential company data—names, URLs, ratings—for $2/1000 results. Integrates seamlessly with enrichment tools. Fast, efficient, and cost-effective!

Gopalakrishnan

115

Results Checker

lukaskrivka/results-checker

Check the results of your scrapers with this flexible checker. Just supply a dataset or key-value store ID and a few simple rules to get a detailed report.