What does the Hacker News Data Scraper & Activity Monitoring do?

The Hacker News Data Scraper & Activity Monitoring tool is designed to collect detailed information from posts on Hacker News (HN), a Y Combinator platform. It also offers a unique feature: the ability to receive fresh activity updates directly on your Slack Channel. You can scrape all listings on Hacker News or filter by keywords to monitor the buzz around certain topics or even track your competitors' movements on the platform.

This unofficial Hacker News API can help you extract information such as:

• Hacker News Post Url
• Post Associated Link
• Title (Headline)
• Rank
• Author
• Points (Upvotes)
• Date

But the functionality doesn't stop there. Here is a breakdown of its additional features:

• Filter Hacker News posts based on specific filter values defined by the user.
• Monitor new posts by leveraging the full capabilities of the Apify platform.
• Receive activity monitoring updates directly on Slack.

Why use Hacker News Data Scraper & Activity Monitoring?

The data from Hacker News holds immense potential for market research and competitive analysis. By collecting detailed information about HN posts, you can compile a list of trending topics, popular authors, or high-ranking posts. Whether you're a business owner analyzing your competitors' movements on Hacker News, a data analyst tracking tech trends, or a developer seeking to incorporate Hacker News data into your applications, the Hacker News Data Scraper & Activity Monitoring tool can help you get the data you need.

Input example

Hacker News Frontpage and Newest output example:

1{
2  "hnPostUrl": "https://news.ycombinator.com/item?id=39581690",
3  "url": "https://phys.org/news/2024-03-billionaires-bunkers-buying-islands-prepping.html",
4  "title": "Billionaires building bunkers buying islands– prep for apocalypse or feudalism?",
5  "rank": "511",
6  "author": "rolph",
7  "points": "36",
8  "date": "3 Mar 2024, 4:03 PM UTC"
9},

Hacker News Show output example:

1{
2  "hnPostUrl": "https://news.ycombinator.com/item?id=39557852",
3  "url": "https://www.onbloom.app",
4  "title": "Show HN: I made an app to email daily newsletters using your own AI prompt",
5  "rank": "30",
6  "author": "akadeb",
7  "points": "3",
8  "date": "1 Mar 2024, 2:27 AM UTC"
9},

Hacker News Jobs output example:

1{
2  "hnPostUrl": "https://news.ycombinator.com/item?id=39356860",
3  "url": "https://jobs.lever.co/memfault/9fa58d3b-2415-4d80-9e5c-be989f90c2e4",
4  "title": "Memfault (YC W19) Is Hiring a Firmware Engineer in NYC/Boston",
5  "rank": "",
6  "author": "YC Jobs",
7  "points": "Job listing",
8  "date": "13 Feb 2024, 12:00 PM UTC"
9},

Hacker News Ask output example:

1{
2  "hnPostUrl": "https://news.ycombinator.com/item?id=39585694",
3  "url": "https://news.ycombinator.com/item?id=39585694",
4  "title": "Ask HN: What are real costs of an internal developer portal?",
5  "rank": "29",
6  "author": "starmandeluxe",
7  "points": "1 point",
8  "date": "4 Mar 2024, 12:02 AM UTC"
9},

How much will scraping Wolt Restaurants cost you?

This scraper operates on a pay-as-you-use basis, charging 1 Apify Credits per 1000 results, in addition to the standard Apify platform fees.

Requirements

Be aware that to effectively use the monitoring capabilities of this Actor you need to run it in an Apify Task within the platform. For more details, read the guide below.

Hacker News Activity Monitor - Guide and Setup Instructions

The activity monitor is a unique and powerful feature available in select NxNode Actors, designed to enhance your user experience and instantly add value to the data scraped by this Actor. This feature relies on the symbiotic relationship between the Hacker News Actor and special features of the Apify Platform, which can make its initial setup a bit scary. But don't sweat, we will walk you through all the details about how this feature works and what you need to do to set it up.

How does the Activity Monitoring work?

The activity monitoring feature operates by comparing the results from the current and previous datasets of a specific Task run for this Actor. Through this comparison, the Hacker News Data Scraper & Activity Monitoring tool can identify new Hacker News activity since the last Task run. It then stores all newly discovered results in a new file named "NEW_HN_POSTS" in your Actor's Key-Value store.

Please note that this feature relies on the comparison between task runs to function correctly. Therefore, your first Task run will only yield standard results and won't perform any comparison, as there are no previous runs to compare with.

Set Up Guide

Here's a step-by-step guide on how to utilize this feature:

1. Set Up an Apify Task (REQUIRED)

• Navigate to the Actor's input page and click on the "Create task" button located at the top right corner of the page.

2. Customize the Actor Inputs

• Modify the Actor inputs as needed and click "Save".

3. Run the task

• Note that the first task execution will only yield standard results as there are no previous runs for comparison. The comparison functionality will be available from the second task run onwards.

4. Schedule the Task (RECOMMENDED)

• f you plan to monitor Hacker News activity regularly, it's advisable to schedule this Actor to run automatically at set intervals - daily, weekly, monthly, or whatever suits your needs. To set this up, go to the Actor's Task page, click on the three dots at the top right corner, and select "Schedule".

Slack Activity Monitoring Updates

This Actor has a built-in feature that allows you to set up a Slack Bot to send updates about new activities directly to a Slack Channel of your choice. To do this, you'll need to create a Slack App and provide the Actor with your Slack channel name, Slack Bot Token, and Slack Signin Secret. If you're unsure about how to find this information or how to create a Slack App, please refer to this guide.

Slack Activity Monitoring Update example