3 days trial then $20.00/month - No credit card required now
Product Hunt Scraper
3 days trial then $20.00/month - No credit card required now
An useful scraper which can get data from https://www.producthunt.com/.
Crawl recursively?
Closed
Can there be an option to crawl recursively to a certain depth? I want to be able to scrape all companies in a category, for example by finding the related companies and crawling them.
Currently, I can only get like 37 results per run, which isn't very useful.
By design, this scrapper retrieve data on first page and can control how many page to retrieve.
Crawl recursively will cause crawl to much data in one time and may block by the producthunt.
I'll check this recently. It seems the data logic has change.
I've find the possible issue here. Will try to fix it.
now it can crawl data again. Next to make it crawl recursively.
try with different crawl configuration and found producthunt has strong protect for crawl recursively. To make it more like a person browse the site. Just crawl to detail page of each product in first page.
If you want more data, please use this crawl daily to get what you want and not get block by product hunt.
If no more comments, I'll close this issue three days later.
close it.
kuai ma
