# Hacker News Data Scraper (`crawl_craft/hacker-news-data-scraper`) Actor

Search any keyword on Hacker News or browse the front page, newest, Ask, Show, Jobs and Best. Get titles, links, authors, points, comment counts and dates, plus optional comments and article descriptions. Export to JSON, CSV or Excel, on a schedule if you like.

- **URL**: https://apify.com/crawl_craft/hacker-news-data-scraper.md
- **Developed by:** [Crawl Craft](https://apify.com/crawl_craft) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### 🚀 What is Hacker News Data Scraper?

Hacker News Data Scraper is an Apify Actor that collects stories from Hacker News by keyword or by browse mode. Each row holds the title, link, author, points, comment count and posting date, with optional comments and article descriptions. Use this Hacker News scraper instead of refreshing the front page by hand.

Tracking Hacker News by refreshing the front page misses the best threads. A search across all stories, or a browse mode on a schedule, keeps every post you care about in one dataset.

- 🔎 **Keyword search:** run a Hacker News search for any word or phrase and sort the results by relevance or by date.
- 🧭 **Six browse modes:** FRONTPAGE, NEWEST, ASK, SHOW, JOBS and BEST work without a keyword.
- 💬 **Comments on demand:** fetch comments for each story, or skip them for a faster run.
- 📰 **Article descriptions:** add a short description of each linked article to the `text` field when one is available.
- 📅 **Date filters:** use 7d, 30d, 6m, 1y or a custom start and end date.

Need a Hacker News web scraper without any setup? Open the Actor, enter a keyword and press **Start**. Looking for a Hacker News extractor that exports to spreadsheets? The dataset downloads as JSON, CSV or Excel.

### 📊 What data can you extract from Hacker News?

Each row in the dataset is one of the Hacker News stories you asked for, and the fields below are ready to export as JSON, CSV or Excel.

| Field | Description | Example |
|---|---|---|
| `id` | Hacker News story ID. | 45751400 |
| `type` | story, ask_hn, show_hn, or job. | story |
| `title` | Story headline. | UV is the best thing to happen to Python |
| `url` | External link (null for self-posts). | https://example.com/article |
| `domain` | Extracted domain from the URL. | example.com |
| `text` | Post body text or article meta description. | A deep dive into... |
| `author` | HN username of the submitter. | todsacerdoti |
| `points` | Upvote score. | 2214 |
| `num_comments` | Total number of comments on HN. | 1324 |
| `created_at` | ISO 8601 submission timestamp. | 2025-10-29T18:57:29Z |
| `age_hours` | Story age in decimal hours from scrape time. | 8808.7 |
| `points_per_hour` | Points divided by `age_hours`. | 0.25 |
| `comments` | Fetched comments (filled when `maxCommentsPerPost` is above 0). | \[] |

Hacker News points and `num_comments` show how much attention a story received. Hacker News comments are fetched only when `maxCommentsPerPost` is above 0, and each comment carries `id`, `author`, `text`, `created_at` and `depth`. The Actor caps the number of comments returned per story.

Hacker News posts without a link, such as Ask posts, have `url` set to null. Use the `domain` column to see Hacker News by domain.

The Overview view in the Output tab lists the main fields. The With Comments view adds the fetched `comments`.

### 💡 Use cases for Hacker News data

- **Watch your product name:** set up Hacker News keyword monitoring with `queries`, `sortBy` set to `date` and `dateRange` set to `7d`, then read `title`, `points` and `num_comments`.
- **Follow a technology:** track Hacker News mentions of a framework or company by searching it on a schedule and comparing `created_at` and `num_comments`.
- **Spot rising stories:** `points_per_hour` and `age_hours` show which stories gain upvotes fastest, which helps when you check Hacker News trending topics.
- **Study a topic over time:** collect a year of stories with `dateRange` set to `1y` for Hacker News trend analysis in your own spreadsheet or notebook.
- **Keep an eye on rivals:** search competitor names for Hacker News competitor monitoring and read `title`, `domain` and `author`.
- **Build a prospect list:** run SHOW or JOBS and collect `author`, `url` and `domain` as a starting point for Hacker News lead generation.
- **Read the mood:** set `maxCommentsPerPost` and export the `comments` text for Hacker News sentiment analysis in your own tools.
- **Curate a digest:** run BEST with `dateRange` set to `7d` to shortlist stories for a Hacker News newsletter.
- **Review hiring posts:** JOBS mode returns Hacker News job postings with `title`, `url` and `text`.
- **Feed a reader app:** combine several `queries` and modes in one run to power a tech news aggregator of your own.
- **Size up a market:** search a product category for Hacker News market research and compare `points` across the results.
- **Learn from launches:** SHOW mode lists Hacker News project launches with `author`, `domain` and `points`.
- **Find what resonates:** read `title` and `text` from a keyword search for Hacker News content research on how people phrase a subject.
- **Follow submitters:** export `author` with `points` and `created_at` to study Hacker News author activity.
- **Build a corpus:** keep `created_at`, `points` and `comments` from many queries as a Hacker News research dataset for analysis.

### 🛠️ How to scrape Hacker News in six steps

1. Create a free Apify account and open Hacker News Data Scraper.
2. Enter one keyword or mode per line in **Search Queries or Modes**, for example `python` or `FRONTPAGE`.
3. Set **Max Results per Query** and keep it small for a first test.
4. Choose **Sort Order**, **Date Range**, **Max Comments per Story** and **Fetch Article Description** if you need them.
5. Click **Start** and wait for the run to finish.
6. Download your Hacker News data as JSON, CSV or Excel.

#### Hacker News date filter for recent stories

Enter a shortcut such as `7d`, `30d`, `6m` or `1y` in `dateRange`, or a custom range such as `2024-01-01,2024-12-31`. Leave it empty to cover all time.

#### Hacker News to Excel

Open the finished dataset, choose Export and pick Excel. The same menu offers CSV and JSON, so Hacker News to CSV takes one more click.

### 💰 How much does it cost to scrape Hacker News?

Hacker News Data Scraper uses pay per event pricing: every result is one billed event, and the price per 1,000 results depends on your Apify plan.

| Apify plan | Price per 1,000 results |
|---|---|
| Free | $1 |
| Bronze | $0.9 |
| Silver | $0.8 |
| Gold, Platinum, Diamond | $0.7 |

For example, 1,000 results cost $1 in the Free row and $0.7 in the Gold, Platinum and Diamond row. Each run start also adds one small Actor Start event of $0.00005. See the Pricing tab for current rates.

### ⬇️ Input

Only `queries` is required: a list of keywords or browse modes, one entry per line. See the Input tab for every option.

![Hacker News scraper input form with search queries and date range](IMAGE_URL_PLACEHOLDER "hacker-news-data-scraper-input-form.png")

Example input:

```json
{
  "queries": ["python"],
  "maxResults": 10,
  "sortBy": "relevance",
  "dateRange": "",
  "maxCommentsPerPost": 0,
  "fetchArticleText": true
}
```

| Field | Type | Default | Description |
|---|---|---|---|
| `queries` | array of strings | `["python"]` (pre-filled) | Keywords to search, or the mode names FRONTPAGE, NEWEST, ASK, SHOW, JOBS, BEST. Required. |
| `maxResults` | integer | `10` | Maximum stories per query or mode, from 0 to 50000. Set 0 for no limit. |
| `sortBy` | string | `relevance` | `relevance` or `date` (newest first). Ignored for browse modes. |
| `dateRange` | string | empty | `7d`, `30d`, `6m`, `1y` or `YYYY-MM-DD,YYYY-MM-DD`. Empty means all time. |
| `maxCommentsPerPost` | integer | `0` | Comments to fetch per story (input range 0 to 5000; the Actor caps how many are returned per story). Set 0 to skip comments. |
| `fetchArticleText` | boolean | `true` | Add a short description from each linked article to `text` when available. |

#### Browse modes

| Mode | What it returns |
|---|---|
| FRONTPAGE | Stories on the Hacker News front page |
| NEWEST | The Hacker News newest submissions |
| ASK | Ask HN posts |
| SHOW | Show HN posts |
| JOBS | Hacker News jobs listings |
| BEST | Stories returned by the BEST mode |

### ⬆️ Output

The scraped results are shown as a dataset in the **Storage** tab. The table view is for convenience and doesn't show **all the fields**.

![Hacker News data scraper output table with points and comments](IMAGE_URL_PLACEHOLDER "hacker-news-data-scraper-output-table.png")

Your Hacker News dataset can be downloaded as JSON, CSV or Excel. Use `points_per_hour` to compare Hacker News engagement rate across stories of different ages.

Example output (values taken from the dataset schema, not from a captured run):

```json
{
  "id": 45751400,
  "type": "story",
  "title": "UV is the best thing to happen to Python",
  "url": "https://example.com/article",
  "domain": "example.com",
  "text": "A deep dive into...",
  "author": "todsacerdoti",
  "points": 2214,
  "num_comments": 1324,
  "created_at": "2025-10-29T18:57:29Z",
  "age_hours": 8808.7,
  "points_per_hour": 0.25,
  "comments": []
}
```

At the end of every run a `RUN_SUMMARY` record in the key-value store lists the total items, the queries, the status and the duration.

### ⚠️ Limitations and error handling

- Each query or mode returns at most the number set in `maxResults`, and 50000 is the highest value the input accepts.
- The Actor caps the number of comments returned per story, so a high `maxCommentsPerPost` does not guarantee that many comments.
- Only public stories and comments are collected. No login is used.
- Many fields can be empty (null), for example `url` on self-posts, so handle missing values in your own tools.
- An article description is added only when the linked page provides one. A story is still returned when it cannot be added.
- If `queries` is empty, the run stops with a message asking for at least one keyword or mode.
- If some queries cannot be completed, the run reports partial results in its status message and in `RUN_SUMMARY`.
- If you set a maximum spend per run, the Actor stops cleanly once that limit is reached.
- A query with no matches returns no rows.

### 🔌 Integrating Hacker News Data Scraper with other apps

Hacker News Data Scraper runs like any Apify Actor, so you can start it from the Apify API, a schedule or a webhook. See the API tab for ready-made code snippets.

- **Schedules:** use [Apify schedules](https://docs.apify.com/platform/schedules) to repeat a run daily or weekly. Set `sortBy` to `date` and `dateRange` to `7d` so each weekly run covers the latest week. Each run is independent and has no "only new stories" option.
- **Webhooks:** attach an [Apify webhook](https://docs.apify.com/platform/integrations/webhooks) to a scheduled run to build Hacker News alerts in the tools you already use.
- **Integrations:** send the dataset to spreadsheets and automation tools through the [Apify integrations](https://docs.apify.com/platform/integrations).
- **API:** if your code needs a Hacker News API, call the Actor through the [Apify API](https://docs.apify.com/api/v2): start a run, then read the dataset items as JSON.

### 🔗 More Apify Actors by crawl_craft

| Actor | What it extracts | When to use it |
|---|---|---|
| [Quora Scraper](https://apify.com/crawl_craft/quora-scraper) | Questions with view and answer counts, answers with text, upvotes and dates, author profiles, spaces and topics | Use it when you also want question-and-answer content on a topic, which Hacker News Data Scraper does not collect |
| [Telegram Scraper](https://apify.com/crawl_craft/telegram-scraper) | Messages, views, reactions, media, links, polls, comments and subscriber counts from public Telegram channels | Use it when your topic tracking also covers public Telegram channels |
| [Facebook Posts Search Scraper](https://apify.com/crawl_craft/facebook-posts-search-scraper) | Public Facebook posts found by keyword or hashtag, with reactions, comments, shares and media | Use it to follow the same keyword on Facebook |

### ⚖️ Is it legal to scrape Hacker News?

Hacker News Data Scraper collects only stories and comments that are publicly visible on Hacker News. It does not log in and does not collect private data such as email addresses. Results can still contain personal data, such as usernames in `author`, so collect them only with a legitimate reason and consider GDPR (Regulation (EU) 2016/679) and the site's terms. Consult a lawyer if you are unsure. For background, read Apify's posts on the [legality of web scraping](https://blog.apify.com/is-web-scraping-legal/) and [ethical web scraping](https://blog.apify.com/what-is-ethical-web-scraping-and-how-do-you-do-it/).

### ❓ FAQ

#### Do I need a Hacker News API key?

No. The Actor needs no key and no login. You only enter keywords or browse modes.

#### How much does Hacker News scraping cost?

Each result is billed as one event, at $1 per 1,000 results in the Free row and down to $0.7 per 1,000 in the top row of the table above. See the Pricing tab for current rates.

#### How many results can I get per query?

Set `maxResults` up to 50000 per query or mode, or enter 0 for no limit. Start with a small number to test your query.

#### How do I get Hacker News top stories?

Enter FRONTPAGE in `queries` for the stories on the front page. BEST returns the stories of the BEST mode, and `dateRange` narrows either one to recent weeks.

#### What is the difference between FRONTPAGE and NEWEST?

FRONTPAGE returns stories on the front page, while NEWEST returns the latest submissions. Both work without a keyword.

#### Can I collect comments?

Yes. Set `maxCommentsPerPost` above 0 and the `comments` field is filled for each story. Leave it at 0 to skip them.

#### Can I search by keyword instead of using a mode?

Yes. Any entry in `queries` that is not a mode name is searched as a keyword across stories, sorted by `sortBy`.

#### Can I scrape private data?

No. Only public stories and comments are collected.

### 💬 Your feedback

Found a bug or missing a field? Open a ticket in the Issues tab of this Actor. For a custom solution, such as extra fields or a tailored workflow, write to hello.crawlcraft@gmail.com.

# Actor input Schema

## `queries` (type: `array`):

Enter keywords to search (e.g. "python", "AI startup") or special mode names: FRONTPAGE, NEWEST, ASK, SHOW, JOBS, BEST. One entry per line.

## `maxResults` (type: `integer`):

Maximum stories to return for each query or mode. Set 0 for no limit (up to 50,000 results).

## `sortBy` (type: `string`):

Sort keyword search results by relevance score or date (newest first). Ignored for special modes.

## `dateRange` (type: `string`):

Filter stories by date. Use shortcuts: 7d, 30d, 6m, 1y — or a custom range: YYYY-MM-DD,YYYY-MM-DD. Leave empty for all time.

## `maxCommentsPerPost` (type: `integer`):

Maximum number of top comments to fetch per story. Set 0 to skip comments entirely (faster run).

## `fetchArticleText` (type: `boolean`):

Fetch a short description from each linked article and add it to the story's text field when one is available.

## `proxyConfiguration` (type: `object`):

Optional connection settings. The default works for most runs.

## Actor input object example

```json
{
  "queries": [
    "python"
  ],
  "maxResults": 10,
  "sortBy": "relevance",
  "dateRange": "",
  "maxCommentsPerPost": 0,
  "fetchArticleText": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `stories` (type: `string`):

All scraped stories with full metadata.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "python"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("crawl_craft/hacker-news-data-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["python"],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("crawl_craft/hacker-news-data-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "python"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call crawl_craft/hacker-news-data-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,crawl_craft/hacker-news-data-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ghpM2W9Xd1dJDelPJ/builds/YzcEi8JFGxysGr5gh/openapi.json