YouTube Scraper
Pay $5.00 for 1,000 videos
YouTube Scraper
Pay $5.00 for 1,000 videos
YouTube crawler and video scraper. Alternative YouTube API with no limits or quotas. Extract and download channel name, likes, number of views, and number of subscribers.
How do I deduplicate results so I'm not getting the same channels, videos, etc.
The scraper works perfectly, but I just want to be able to de-duplicate according by channel name.
The same channels pop up again and again, and it is messing up what I am trying to do with this YouTuber scraper.
I have tried to find integrations that may support this, but none pop up.
Wondering if YouTube scraper natively has this, or if it is possible to get this integrated with this scraper.
Hi, thanks for opening this issue!
To confirm, you mean deduplication of the channels in a single run, or multiple? Implementing this in a single run is doable for us, however in multiple ones, you will have to use some outside tools (because ultimately they will have to compare the values with the previous runs via some database).
If you want to do this, then ideally you would create your own solution with a proper database, but you could go full Apify integrations if you want to. You could use the Merge, Dedup & Transform Datasets Actor to deduplicate them. Then use the Append to dataset Actor to put it into one big dataset.
I hope this helps, thanks and happy scraping!
StreamersActor Metrics
2.2k monthly users
-
447 bookmarks
77% runs succeeded
1.5 days response time
Created in Jul 2023
Modified 8 days ago
