Pricing

Pay per usage

Go to Store

Diff Datasets

Try for free

Developed by

Paulo Cesar

Take one dataset on Apify platform, compare it to another, and output the missing ones. This can also be used to output only changed items, using a compound key

0.0 (0)

Pricing

Pay per usage

Total users

Monthly users

Last modified

3 years ago

Automation

Open source

Take one dataset on Apify platform, compare to another, and output the missing ones. This can also be used to output only changed items, using a compound key.

Supports using whole nested objects as value, they are JSON.stringify'd before being turned into a small non-cryptographic space efficient hash

Example

1await Apify.call('pocesar/diff-datasets', {
2    baseDatasetId: 'LdNAlaOY1aKGhwAah', // place the datasets here. The order of "base" and "other" matters
3    otherDatasetId: 'Bzu1pgOjenN43VhPY', // existing items in "base" are not output from "other"
4    uniqueFields: [
5        // simple primitive field value, like string, number, boolean
6        "pageUrl",
7
8        // you can use lodash.get notation to get nested items,
9        // in this case `sub.fields.0` works like `sub.fields[0]` and the object looks like
10        // {
11        //    pageUrl: "https//pageurl",
12        //    sub: {
13        //      fields: [
14        //        {...},
15        //        {...}
16        //      ]
17        //    }
18        //  }
19        "sub.fields.0",
20
21        // you can also use .length to count arrays or string characters, as in
22        "sub.fields.length",
23        "pageUrl.length"
24    ],
25});

Limitations

Every value is kept in memory while reading from the base dataset, more items more memory needed.
The key value store might choke when trying to save the in-memory Set with too many items

License

Apache 2.0

Diff Dataset Fields

lukaskrivka/diff-dataset-fields

Compare changes in text fields between two datasets. Monitor new and updated content on websites.

Lukáš Křivka

Website Screenshot Generator

apify/screenshot-url

Create a screenshot of a website based on a specified URL. The screenshot is stored as the output in a key-value store. It can be used to monitor web changes regularly after setting up the scheduler.

Apify

3.4k

DIFF Eyewear Scraper

mshopik/diff-eyewear-scraper

Scrape DIFF Eyewear and extract data on eyewear from diffeyewear.com. Our DIFF Eyewear API lets you crawl product information and pricing. The saved data can be downloaded as HTML, JSON, CSV, Excel, and XML.

Mark Carter

Validate Dataset(s) with JSON Schema

jaroslavhejlek/validate-dataset-with-json-schema

This Actor validates items in one or more datasets against a provided JSON Schema. Use it if you planning to add a dataset validation schema to your actor and you want test it.

Jaroslav Hejlek

Spawn Workers

pocesar/spawn-workers

This actor lets you spawn tasks or other actors in parallel on the Apify platform that shares a common output dataset, splitting a RequestQueue-like dataset containing request URLs

Paulo Cesar

Google Dataset Items Translator

web.harvester/google-dataset-items-translator

Translate any dataset field(s) to any of the supported languages using the Google Translate website, it goes through all the items in the dataset and translates all of the selected fields

Web Harvester

Yet Another Dataset Translator

mvolfik/yet-another-dataset-translator

Actor to translate datasets with field selection and source language detection. Requires Google Translate API Key.

Matěj Volf

Sort Dataset Items

lukaskrivka/sort-dataset-items

Add this actor as a webhook to your scraper to sort the dataset by index field

Lukáš Křivka

Forward Dataset to Actor or Task

valek.josef/forward-dataset-to-actor-or-task

Forwards contents of specified dataset to a specified field on the input of another Actor or task.

Josef Válek

LLM Dataset Processor

dusan.vystrcil/llm-dataset-processor

Allows you to process output of other actors or stored dataset with single LLM prompt. It's useful if you need to enrich data, summarize content, extract specific information, or manipulate data in a structured way using AI.