Pricing

Pay per usage

Go to Store

Merge, Dedup & Transform Datasets

Try for free

lukaskrivka/dedup-datasets

Developed by

Lukáš Křivka

The ultimate dataset processor. Extremely fast merging, deduplications & transformations all in a single run.

0.0 (0)

Pricing

Pay per usage

Monthly users

138

Runs succeeded

97%

Response time

11 days

Last modified

2 months ago

Automation Developer tools Open source

Back to issues Create new issue

De-duplicate dataset error

Closed

data_explorer opened this issue

While running the task I keep get this error:

2024-05-01T09:22:59.977Z ACTOR: Pulling Docker image of build BzYf2ui1QKTRzHNQI from repository. 2024-05-01T09:23:02.120Z ACTOR: Creating Docker container. 2024-05-01T09:23:02.325Z ACTOR: Starting Docker container. 2024-05-01T09:23:04.501Z INFO System info {"apifyVersion":"2.2.0","apifyClientVersion":"2.0.4","osType":"Linux","nodeVersion":"v16.20.2"} 2024-05-01T09:23:04.604Z My input: 2024-05-01T09:23:04.607Z { 2024-05-01T09:23:04.610Z datasetIds: [ 'Id07DYkDaLOduqIsZ' ], 2024-05-01T09:23:04.612Z fields: [ 2024-05-01T09:23:04.615Z 'categoryName', 2024-05-01T09:23:04.618Z 'title', 2024-05-01T09:23:04.620Z 'address', 2024-05-01T09:23:04.622Z 'phone', 2024-05-01T09:23:04.624Z 'website', 2024-05-01T09:23:04.627Z 'totalScore', 2024-05-01T09:23:04.629Z 'reviewsCount', 2024-05-01T09:23:04.631Z 'openingHours' 2024-05-01T09:23:04.633Z ], 2024-05-01T09:23:04.640Z fieldsToLoad: [ 2024-05-01T09:23:04.642Z 'categoryName', 2024-05-01T09:23:04.645Z 'title', 2024-05-01T09:23:04.650Z 'address', 2024-05-01T09:23:04.652Z 'phone', 2024-05-01T09:23:04.654Z 'website', 2024-05-01T09:23:04.656Z 'totalscore', 2024-05-01T09:23:04.658Z 'reviewsCount', 2024-05-01T09:23:04.665Z 'openingHours' 2024-05-01T09:23:04.667Z ], 2024-05-01T09:23:04.670Z mode: 'dedup-as-loading', 2024-05-01T09:23:04.699Z output: 'unique-items', 2024-05-01T09:23:04.702Z outputDatasetId: 'tenni... [trimmed]

Lukáš Křivka (lukaskrivka)

Hello,

Thanks for the report. This happened because you accidentally passed run ID, instead of dataset ID. But I updated the so that it will handle it as well now.

Add comment

Pricing

Pricing model

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage.

Website Content Crawler

apify/website-content-crawler

Crawl websites and extract text content to feed AI models, LLM applications, vector databases, or RAG pipelines. The Actor supports rich formatting using Markdown, cleans the HTML, downloads files, and integrates well with 🦜🔗 LangChain, LlamaIndex, and the wider LLM ecosystem.

Apify

42.5k

Twitter (X.com) Scraper Unlimited: No Rate-Limits

apidojo/twitter-scraper-lite

Introducing Twitter Scraper Unlimited, the most comprehensive Twitter data extraction solution available. Our enterprise-grade scraper offers unmatched capabilities with a transparent event-based pricing model, making it perfect for both small-scale and large-scale data extraction needs.

API Dojo

2.9k

Parsera

parsera-labs/parsera

Extract data from any website using just a URL and column descriptions

Parsera Labs

Youtube Video Downloader

epctex/youtube-video-downloader

Effortlessly download YouTube videos of your preferred quality with our user-friendly Video Downloader. Try it now!

epctex

934

Cheerio Scraper

apify/cheerio-scraper

Crawls websites using raw HTTP requests, parses the HTML with the Cheerio library, and extracts data from the pages using a Node.js code. Supports both recursive crawling and lists of URLs. This actor is a high-performance alternative to apify/web-scraper for websites that do not require JavaScript.

Apify

6.8k

Web Scraper

apify/web-scraper

Crawls arbitrary websites using the Chrome browser and extracts structured data from web pages using a provided JavaScript function. The Actor supports both recursive crawling and lists of URLs, and automatically manages concurrency for maximum performance.

Apify

79k

🔥 LinkedIn Jobs Scraper

bebity/linkedin-jobs-scraper

ℹ️ Designed for both personal and professional use, simply enter your desired job title and location to receive a tailored list of job opportunities. Try it today!

Bebity

7.5k

Puppeteer Scraper

apify/puppeteer-scraper

Crawls websites with the headless Chrome and Puppeteer library using a provided server-side Node.js code. This crawler is an alternative to apify/web-scraper that gives you finer control over the process. Supports both recursive crawling and list of URLs. Supports login to website.

Apify

5.9k

Fast TikTok API (free-watermark videos)

novi/fast-tiktok-api

All in one TikTok APIs. Fastest TikTok API for Trend, Hashtag, Search, Music, User, Comment. Provides no-watermark download link.

Novi

1.1k

Youtube Transcript Scraper

topaz_sharingan/Youtube-Transcript-Scraper

Are you in search of a robust solution for extracting transcripts from YouTube videos? Look no further 😉, YouTube-Transcript-Scraper will meet your needs. Our software not only efficiently retrieves transcripts but also provides additional valuable information .👍 😀 Scrap away 🕵‍♂️.