3 days trial then $10.00/month - No credit card required now
Hacker News Data Scraper
3 days trial then $10.00/month - No credit card required now
Extract Y Combinator's Hacker News based on any search criteria. Crawl the front page, Show HN, Ask HN, news, job listings, and historical data. Get links, titles, comments, ratings, and more!
Does it actually scrape the pages?
Closed
I'd like a structured scrape of the HN pages, with the top text as a field, and all comments (with commentator, date, and text) in a hierarchical way.
beremekdar
Hello Joseph,
Thank you for your request to scrape HN pages in a structured manner, including the top text and hierarchical comments. However, the current public actor is not designed to handle this specific case. If you are interested, we can create a custom project tailored to your requirements. Please let us know if you would like to proceed with a custom solution, and we will be glad to assist you further.
Best.
bravura
Berk, what information is this actor supposed to handle?
tugkan_epctex
Hey Joseph,
This is Tugkan. I am here to answer all the possible use cases of the Hacker News actor within all the technical details. As in the description, Hacker News actor is designed to retrieve post items (story), jobs, and all the possible information inside them which also includes comments. However, the generic use of this actor does not support the hierarchical comments. Let me talk with the Engineering Team if this functionality can be integrated by a small feature flag and let you know about it. However, as Berk mentioned, the most robust solution that fulfills all your needs would be possibly a custom integration. Before thinking about that option, I'll talk with the team and let you know about this request.
Thank you for your patience. Best
tugkan_epctex
Hey Mackie,
I just wanted to let you know that we've integrated a new field called "enableCommentHierarchy". If you enable this property, it will retrieve all the comments and build a hierarchy tree on top of it. I hope this will work for you.
Thank you very much and sorry for the inconvenience. Best
epctex
