Product Hunt Scraper
Deprecated
Pricing
$20.00/month + usage
Product Hunt Scraper
Deprecated
An useful scraper which can get data from https://www.producthunt.com/.
0.0 (0)
Pricing
$20.00/month + usage
1
Total users
110
Monthly users
1
Last modified
3 months ago
Crawl recursively?
Closed
Can there be an option to crawl recursively to a certain depth? I want to be able to scrape all companies in a category, for example by finding the related companies and crawling them.
Currently, I can only get like 37 results per run, which isn't very useful.
By design, this scrapper retrieve data on first page and can control how many page to retrieve.
Crawl recursively will cause crawl to much data in one time and may block by the producthunt.
I'll check this recently. It seems the data logic has change.
I've find the possible issue here. Will try to fix it.
now it can crawl data again. Next to make it crawl recursively.
try with different crawl configuration and found producthunt has strong protect for crawl recursively. To make it more like a person browse the site. Just crawl to detail page of each product in first page.
If you want more data, please use this crawl daily to get what you want and not get block by product hunt.
If no more comments, I'll close this issue three days later.
close it.
![[Jun25] Product Hunt daily scraper avatar](https://images.apifyusercontent.com/tiLtQRYEBp_bLpACBJcpq6Uy7DPAlF70E6kSLNLHHzo/rs:fill:76:76/cb:1/aHR0cHM6Ly9hcGlmeS1pbWFnZS11cGxvYWRzLXByb2QuczMudXMtZWFzdC0xLmFtYXpvbmF3cy5jb20vZVpsVWg3R29xbU1ucVJoMEctYWN0b3ItOXVGV3ZldG5vbzNCcXVTelctNEdKaldlZXNETy1wcm9kdWN0LWh1bnQtaWNvbi0yNTZ4MjU2LXFrcnY0dzU2LnBuZw.webp){kind=avatar}
