Advanced Product Hunt Scraper
Pricing
$15.00/month + usage
Advanced Product Hunt Scraper
Scrape product hunt "Top Products Launching Today" section. Actor crawls products and extracts information about the product: title, description, categories, images, maker info with contact links and website info with raw text and email. Export scraped datasets in JSON, csv, etc. Run via API.
5.0 (2)
Pricing
$15.00/month + usage
18
Monthly users
65
Runs succeeded
>99%
Response time
16 hours
Last modified
11 days ago
Yearly run stops early
Open
I tried running the yearly run for the year 2024, with a very high "topNProducts" limit, and it stopped after only 168 products. Any idea why it terminated so early?
Hi 👋
I’ve checked it, and it seems that sometimes it takes a while for the "loading" to finish (when you scroll down on the yearly page: https://www.producthunt.com/leaderboard/yearly/2024/all, you’ll see the loading issue).
I increased the timeout to 7.5 seconds, so hopefully it works better now. I also fixed another related issue.
I tried scraping 1000 products and it took about 10 minutes just to fetch them — so scraping 100,000 would take quite a while.
If you're planning to scrape that many, I’d really recommend going day by day. That way, you can resume from a specific date if something fails.
Unfortunately, Product Hunt doesn’t have pagination, so if scraping fails midway, there’s no simple way to continue from the "last" item.
P.S. I've also added information to log, where you can see how much items were already scrolled to. It will look like this "Found posts, keep scrolling: 342".
Build is still running and will finish in like 10-15 minutes. New version 2.0.3
Let me know if there’s anything else I can help with!
Thanks, Dan
bilingual_hulusi
Thanks for looking into it. Is there a way to crawl the Weekly or Monthly pages? Those would be a good compromise btw daily and yearly, which I agree could be impractically slow to crawl.
That's a good point! I'll add this functionality to crawl launches by week and let you know once it's ready 😉
Pricing
Pricing model
RentalTo use this Actor, you have to pay a monthly rental fee to the developer. The rent is subtracted from your prepaid usage every month after the free trial period. You also pay for the Apify platform usage.
Free trial
3 days
Price
$15.00
