Most scrapers offers dedup functionality, when used with a Key/Value store. But it does not seem like your scraper offers that functionality. Unless I am missing it? It would really be beneficial to add some functionality that would help the user to not keep scraping the same content over and over :)

Please let me know what you have available or how I could perhaps achieve the same effect.

cat (Jupri)

Hello. This scraper has internal deduplication functionality. for example, in video searches, it deduplicates by id but I may missed something. let me know if you found any duplicated results.

dawieharmse

Hey Jupri, Thanks for the quick reply. Ah, ok. I could not see any documentation around it. Will keep running my Tasks and see if I pick up on any Duplicated :) Just dont want to scrape hundreds of the same videos every day and dedupe on import to my platform. Wastes a lot of money :)

cat (Jupri)

😺👍

Add comment

Rumble all-inclusive scraper

azzouzana/rumble-all-inclusive-scraper

🚀 Extract video (revenue included), channels, playlists details & top search results from Rumble using our fast scraper. Just provide channel, video, playlist or search URLs, and export data in JSON, CSV, or HTML. 🔥 Quick, easy, and ready to use!

Azzouzana

5.0

Rumble Video Downloader

easyapi/rumble-video-downloader

Extract and download videos from Rumble.com in multiple quality options (240p to 1080p). Get detailed metadata including title, author, duration, and thumbnail. Perfect for content archiving, research, and offline viewing. Supports batch processing of multiple URLs. 🎥✨

EasyApi

Bulk X / Twitter Email Scraper

scraper-mind/twitter-email-scraper

[𝗖𝗵𝗲𝗮𝗽𝗲𝘀𝘁 𝗣𝗿𝗶𝗰𝗲] Mass Twitter Email Scraper or X Email Scraper: It helps you find Twitter profiles with email addresses based on custom keywords. Perfect for influencer marketing, social media outreach, and business development. Fast, accurate, and 50% more affordable than competitors.

Scraper Mind

139

Twitter Email Scraper

easyapi/twitter-email-scraper

🚀 Effortlessly scrape email addresses from Twitter profiles! 🐦 Find Twitter users matching your keywords and specific email domains. Perfect for lead generation, influencer marketing, and targeted outreach campaigns. Powered by stealth technology for reliable data collection. 📧🕵️‍♂️

EasyApi

167

5.0

Twitter/X Email Scraper (⭐ Fast & Affordable) 📧🎵

scrapestorm/twitter-x-email-scraper-fast-affordable

This Twitter/X Email Scraper extracts email addresses from Twitter/X based on your selected keywords and email domains. You can also set the max number of items for results 🔥. Perfect for creator outreach 🤳 and collaborations 💥, it delivers fast and accurate results ⚡.

Storm_Scraper

🎵 TikTok Explorer

jupri/tiktok

💫 All-in-One TiKToK Scraper

cat

Facebook Like Scraper

easyapi/facebook-like-scraper

Effortlessly scrape likes from Facebook posts. Automates the process of extracting valuable engagement data, including user information and reaction types. Perfect for social media analysts, marketers, and researchers looking to gain insights from Facebook post interactions.

EasyApi

109

5.0