# Hacker News Scraper (`deriverge/hacker-news-scraper`) Actor

Stories from the Hacker News front page lists and full-text search with points, comment count, link and time, plus whole comment threads as rows on request. Filter search by date, points and comments, and get only new stories and comments on each scheduled run.

- **URL**: https://apify.com/deriverge/hacker-news-scraper.md
- **Developed by:** [deriverge s.r.o.](https://apify.com/deriverge) (community)
- **Categories:** News, Social media, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.25 / 1,000 stories

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Hacker News Scraper

Hacker News Scraper returns stories from the Hacker News front page lists (top, new, best, Ask HN, Show HN and jobs) and from full-text search, one row per story with title, link, points, comment count, submitter and time. On request it adds the whole comment thread of each story as rows with depth and parent. Click **Start** with the prefilled input to get the current top 100, which took 3 seconds in October 2026. One search term returns at most 1,000 stories, a limit of the search service. It costs $0.50 per 1,000 stories on the Free plan ($0.25 on Business), and the $5 of monthly credit in Apify's Free plan covers about 10,000 stories.

### What data does it return?

| Field | Description |
|---|---|
| `id`, `hnUrl` | Hacker News item ID and the discussion page, such as `https://news.ycombinator.com/item?id=50024055`. |
| `title` | Story title as submitted, including prefixes such as `Show HN:` or `Ask HN:`. |
| `kind` | `story` for a link, `ask` for Ask HN and Tell HN, `show` for Show HN, `job` or `poll`. |
| `url`, `domain` | The external link and its host without `www`, such as `github.com`. Both `null` for text posts such as most Ask HN. |
| `text` | Body of a text post or job ad as plain text, `null` for plain link stories. |
| `author` | The Hacker News username of the submitter. |
| `points`, `commentsCount` | Score and number of comments at the time of the run. Job ads have no comments, so the count is `null`. |
| `createdAt` | When the story was posted, ISO 8601 UTC. |
| `source`, `rank` | Where the story came from (`top`, `new`, `best`, `ask`, `show`, `jobs`, `search` or `input` for stories you named) and its position there, 1 being first. |
| `matchedQuery` | The search term that found the story, `null` for list and named stories. |
| `commentsRead` | With **Include comments** on, how many comments of the story the run returned, up to **Maximum comments per story**. `commentsCount` stays the total of the thread. |
| `type`, `key` | `story` or `comment`, and the row key such as `story:50024055` that new-only mode compares. |

Comment rows, returned with **Include comments**, have `id`, `hnUrl`, `author`, `createdAt` and `text` like stories, plus `storyId` and `storyTitle` of the thread, `parentId` (the comment it replies to, `null` at the top level) and `depth` (0 for a top-level comment, 1 for a reply and so on). They come right after their story, in the order of the thread.

### How to scrape Hacker News

1. Pick front page lists in **Lists** and set **Maximum stories per list**. Top and new go to 500 stories, best to 200, Ask HN, Show HN and jobs as far as Hacker News lists them.
2. To search, add **Search terms**, choose **What to search** (stories, Ask HN, Show HN, jobs, polls or stories that reached the front page) and an **Order**: relevance, or newest first for monitoring. **Published after**, **Published before**, **Minimum points** and **Minimum comments** narrow the search.
3. To read particular threads, put item links or IDs into **Stories**. With **Include comments** on, a link to a single comment returns that comment and the replies under it.
4. Turn on **Include comments** for the comment trees and set **Maximum comments per story**; replies count towards it.
5. Click **Start**. The **Output** tab shows stories and comments, ready to download as CSV, Excel or JSON. `SUMMARY` in the key-value store counts stories and comments found and rows returned.

An API input that watches for new, well-received stories about Postgres with their discussions:

```json
{
  "lists": [],
  "queries": ["postgres"],
  "sortBy": "date",
  "publishedAfter": "2026-10-01",
  "minPoints": 10,
  "includeComments": true,
  "maxCommentsPerStory": 300,
  "newOnly": true,
  "watchName": "postgres-mentions"
}
```

### Example output

A Show HN story from the top list in a run on 11 October 2026. The submitter's username is replaced with a placeholder here; the actor returns the real one.

```json
{
  "type": "story",
  "key": "story:50024055",
  "id": 50024055,
  "kind": "show",
  "title": "Show HN: The rarest tech books and docs you've probably never read",
  "url": "https://readrare.com/",
  "domain": "readrare.com",
  "text": null,
  "author": "A Hacker News user",
  "points": 175,
  "commentsCount": 65,
  "createdAt": "2026-10-09T17:39:44.000Z",
  "hnUrl": "https://news.ycombinator.com/item?id=50024055",
  "source": "top",
  "rank": 78,
  "matchedQuery": null,
  "commentsRead": null
}
```

### How much does it cost to scrape Hacker News?

| | Free plan | Starter | Scale | Business |
|---|---|---|---|---|
| 1,000 stories | $0.50 | $0.40 | $0.325 | $0.25 |
| Comment, per 1,000 | $0.20 | $0.16 | $0.13 | $0.10 |

You pay only for the events in the table. There is no start fee, and compute time and proxies are included.

1,000 stories cost $0.50 on the Free plan, and 10,000 comments $2.00 more ($0.25 and $1.00 on Business). Stories removed by **Keywords (any of)** and rows already returned in new-only mode are not charged.

### Limits

- The search service returns at most 1,000 results per search term: `python` sorted by relevance gave exactly 1,000 in October 2026, whatever **Maximum results per search term** says. Split long periods with **Published after** and **Published before** to get more.
- Comment threads are read 4 stories at a time. The top 160 stories with 9,906 comments took 38 seconds on 11 October 2026, close to the default 10,000 rows of **Maximum rows**, so raise it for large threads.
- The filters for points, comments and dates apply to search only. **Keywords (any of)** applies to lists and search, matching the title, text and link.
- Deleted and dead stories are skipped, and so are comments without text.
- Points and comment counts change quickly in the first hours; the row shows them at the time of the run.
- Requests that meet a server error or HTTP 429 are repeated twice. If more than half of the requests fail, the run ends as failed.

### Alerts for new stories and comments

Set a **Watch name** or save the input as a task, turn on **Return only rows new since the last run** and schedule it. Each run then returns only stories and comments that no earlier run of the watch returned, so a search sorted by newest first works as a keyword alert. Rows that **Maximum rows** or your spending limit cut off are not counted as seen, and the next run returns them. The `CHANGES` record also lists stories whose points changed since the last run.

### FAQ

#### Is it legal to scrape Hacker News?

The actor reads public data through the two interfaces offered for it: the official Hacker News API on Firebase and the Hacker News search API run by Algolia, without logging in. Rows contain the usernames of submitters and commenters as shown on the site, but no user profiles, karma or contact data. You are responsible for how you use the data, including the privacy of the people whose comments you store.

#### How do I get the monthly "Who is hiring" posts?

Put the link of the month's thread into **Stories**, turn on **Include comments** and set **Maximum comments per story** to a few thousand, because replies count too. The job posts are the comment rows with `depth` 0.

#### Can I search by author or get user profiles?

No. The actor searches titles, links and text, and it does not read user pages.

### Related scrapers

- [Hacker News Search Scraper](https://apify.com/deriverge/hacker-news-search)
- [Hacker News Comments Scraper](https://apify.com/deriverge/hacker-news-comments-scraper)
- [YC Jobs Scraper](https://apify.com/deriverge/yc-jobs-scraper)
- [Substack Scraper](https://apify.com/deriverge/substack-scraper)

### Support

This actor is built and maintained by deriverge s.r.o., a software company based in the Czech Republic. If a run fails or a field you need is missing, please open an issue in the **Issues** tab or write to us at info@deriverge.com. We respond in English and Czech. Runs can be scheduled in Apify Console or started from the **API** tab, which has examples for Python, JavaScript and cURL and works with Make, Zapier, n8n and the Apify MCP server. If the actor saves you time, a short review helps other people find it.

# Changelog

This Actor's version history is a separate document: https://apify.com/deriverge/hacker-news-scraper/changelog.md

# Actor input Schema

## `lists` (type: `array`):

Which front page lists to read: top, new, best, ask, show, jobs. Leave empty when you only search or name stories.

## `maxPerList` (type: `integer`):

Top and new go to 500, best to 200, the others as far as Hacker News lists them.

## `queries` (type: `array`):

Full-text search over titles, links and text. An empty term with a tag or date filter returns everything that matches the filter.

## `searchTags` (type: `string`):

Ask HN and Show HN are subsets of stories; front page means stories that reached the front page.

## `sortBy` (type: `string`):

Relevance ranks by match and popularity; newest first is what a monitor wants. An empty search term always uses newest first.

## `publishedAfter` (type: `string`):

Only items created after this date (YYYY-MM-DD).

## `publishedBefore` (type: `string`):

Only items created before this date (YYYY-MM-DD).

## `minPoints` (type: `integer`):

Only items with at least this many points.

## `minComments` (type: `integer`):

Only items with at least this many comments.

## `maxResultsPerQuery` (type: `integer`):

Results are read a hundred at a time from the top of the ranking.

## `stories` (type: `array`):

Hacker News item links (news.ycombinator.com/item?id=...) or numeric ids. With Include comments on, a link to a single comment returns that comment and the replies under it.

## `includeComments` (type: `boolean`):

Return the comment tree of every story as rows with author, date, text, depth and parent. Charged per comment.

## `maxCommentsPerStory` (type: `integer`):

Replies count towards the cap; the tree is walked from the top.

## `keywords` (type: `array`):

Keep only stories whose title, text or link contain at least one of these words, case-insensitive. Dropped stories are never charged.

## `newOnly` (type: `boolean`):

Keeps a snapshot per watch name (or per saved task) and returns only stories and comments that were not there before. Schedule it hourly or daily and you have a Hacker News alert that bills only for what is new.

## `watchName` (type: `string`):

Name of the snapshot used by the new-only mode, for example "ai-alert". Runs from a saved task get a snapshot automatically even without a name.

## `maxItems` (type: `integer`):

Hard cap on returned rows in the run.

## Actor input object example

```json
{
  "lists": [
    "top"
  ],
  "maxPerList": 100,
  "queries": [],
  "searchTags": "story",
  "sortBy": "relevance",
  "maxResultsPerQuery": 100,
  "includeComments": false,
  "maxCommentsPerStory": 100,
  "newOnly": false,
  "maxItems": 10000
}
```

# Actor output Schema

## `rows` (type: `string`):

One row per story with kind, title, link, domain, points, comment count, author and date; comment rows under each story on request.

## `changes` (type: `string`):

Stories and comments that appeared since the previous snapshot of the same watch name or task, and point changes.

## `summary` (type: `string`):

Counts per list and search term, what filters dropped, what was charged and notes.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "lists": [
        "top"
    ],
    "queries": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("deriverge/hacker-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "lists": ["top"],
    "queries": [],
}

# Run the Actor and wait for it to finish
run = client.actor("deriverge/hacker-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "lists": [
    "top"
  ],
  "queries": []
}' |
apify call deriverge/hacker-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,deriverge/hacker-news-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Z631smXPNxmjgcLmX/builds/RELBcQsa2Iqec24Sq/openapi.json