Pricing

Pay per usage

Go to Store

Crawler Results Deduplicate

Try for free

Developed by

Petr Cermak

This acts takes crawler execution results and deduplicates them.

0.0 (0)

Pricing

Pay per usage

Monthly users

Last modified

4 years ago

Developer tools

Open source

act-crawler-results-deduplicate

This acts takes crawler execution results and deduplicates them.

Example input:

1{
2    "_id": "EXECUTION_ID",
3    "data": "{
4        \"compareKey\": \"YOUR_COMPARE_KEY\"
5    }"
6}

The optional "data" attribute must be a stringified JSON and can contain a "compareKey" attribute. This key is used to select an object attribute for comparison. If no key is provided, whole object will be used for comparison. It is expected to be called from a crawler webhook.

Pricing

Pricing model

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.

Execution To Xlsx

petr_cermak/execution-to-xlsx

The act takes a JSON input containing a crawler executionID, downloads the results and converts them to XLSX.

Petr Cermak

Microsoft SQL Server Insert

petr_cermak/MSSQL-insert

This actor takes a crawler execution and inserts its results into a remote MSSQL database.

Petr Cermak

Example Process Crawl Results

apify/example-process-crawl-results

Iterates through all results from a crawler run and count them. Needs to be called from the crawler's finish webhook by adding an URL to finish the webhook of your crawler. Use this actor as a starting point to develop custom post-processing of data from the crawler.

Apify

MySQL Insert

petr_cermak/MySQL-insert

This act takes a crawler execution and inserts it's results into a remote MySQL database.

Petr Cermak

Execution To Knack

petr_cermak/execution-to-knack

Act for inserting crawler execution results into Knack database.

Petr Cermak

PostgreSQL Insert

petr_cermak/PostgreSQL-insert

This act takes a crawler execution and inserts it's results into a remote PostgreSQL database.

Petr Cermak

Icon Burglar

yuri/Icon-burglar

This act takes the URLs prepared by a crawler and downloads the images to finally zip them

Yuri Manders

Send Legacy PhantomJS Crawler Results

drobnikj/send-crawler-results

This actor downloads results from Legacy PhantomJS Crawler task and sends them to email as attachments. It is designed to run from finish webhook.

Jakub Drobník

Kijiji Crawler

service-paradis/kijiji-crawler

The Kijiji crawler is a web scraping tool that allows you to extract data from the Kijiji selling platform. With this crawler, you can get information search results and local advertisement details.