Hi 👋

I’ve checked it, and it seems that sometimes it takes a while for the "loading" to finish (when you scroll down on the yearly page: https://www.producthunt.com/leaderboard/yearly/2024/all, you’ll see the loading issue).

I increased the timeout to 7.5 seconds, so hopefully it works better now. I also fixed another related issue.

I tried scraping 1000 products and it took about 10 minutes just to fetch them — so scraping 100,000 would take quite a while.

If you're planning to scrape that many, I’d really recommend going day by day. That way, you can resume from a specific date if something fails.

Unfortunately, Product Hunt doesn’t have pagination, so if scraping fails midway, there’s no simple way to continue from the "last" item.

P.S. I've also added information to log, where you can see how much items were already scrolled to. It will look like this "Found posts, keep scrolling: 342".

Build is still running and will finish in like 10-15 minutes. New version 2.0.3

Let me know if there’s anything else I can help with!

Thanks, Dan

Thanks for looking into it. Is there a way to crawl the Weekly or Monthly pages? Those would be a good compromise btw daily and yearly, which I agree could be impractically slow to crawl.

That's a good point! I'll add this functionality to crawl launches by week and let you know once it's ready 😉

Hey Danil - did you get a chance to update the crawler for Weekly pages? Would love to try it out when it's ready, thanks!

Hi 👋

I've just added a new feature that lets you crawl Weekly pages! To use it, please provide both the year and the week number. Also, make sure to set filter.topNProducts — if you don't, we'll default it to 100.

Let me know if everything works smoothly for you!

Thanks, Dan

Thanks! It worked for some runs but seems to be scraping the wrong items for other runs. For ex run YvjGHeJItnJH4FhGC