An useful scraper which can get data from https://www.producthunt.com/.
It scrape page like this
to get product overall data like this
and get detail information for each product
Currently producthunt has strong protect for crawl recursively. To make it more like a person browse the site. Just crawl to detail page of each product in first page.
- Get useful data in homepage
- Ignore ad item
- can crawl detail page
- can take screenshot
- With debug support
- (WIP) download image