# Hacker News Scraper: Stories, Comments & Front Page (`arman-bd/hacker-news-scraper`) Actor

Scrape Hacker News through the official Firebase API: stories, scores, authors, comment trees, Ask HN, Show HN and job posts.

- **URL**: https://apify.com/arman-bd/hacker-news-scraper.md
- **Developed by:** [Arman Hossain](https://apify.com/arman-bd) (community)
- **Categories:** Developer tools, Automation, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 item scrapeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Hacker News Scraper: Stories, Comments & Front Page

![Hacker News Scraper: Stories, scores, authors and nested comment trees from six Hacker News feeds, front page to Ask HN and Show HN](https://api.apify.com/v2/key-value-stores/ZQOcNAOHrIgTacAmy/records/hacker-news-scraper.jpg)

**Hacker News Scraper** pulls structured records from the official **Hacker News Firebase API**, front page, newest, best, Ask HN, Show HN and the YC job board, with scores, authors, timestamps and optional nested comment trees.

**Agent skill: [SKILL.md](https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/hacker-news-scraper.md)**

```
https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/hacker-news-scraper.md
```

### What you get

| Output field | Meaning |
|---|---|
| `id`, `type` | HN item ID and its type, `story`, `job`, `poll` or `comment` |
| `title` | Submission title |
| `url` | The link the story points at. `null` for Ask HN and text posts |
| `hnUrl` | Permalink to the discussion on news.ycombinator.com |
| `text` | Self-post body as clean plain text, HTML resolved and entities decoded. Links come through as their full address, and code blocks keep their indentation. `null` for link posts |
| `score` | Points at scrape time |
| `by` | Submitter's HN username |
| `time`, `postedAt` | Unix seconds as HN reports it, plus the same moment as an ISO 8601 string |
| `descendants` | HN's own total comment count for the whole thread |
| `commentCount` | How many comments this run actually fetched into `comments`, `0` when comment fetching is off |
| `feed` | Which feed the item came from |
| `comments` | Nested reply tree: `id`, `by`, `time`, `postedAt`, `text`, `depth`, `replies` |
| `scrapedAt` | Run timestamp |

A `RUN_SUMMARY` record in the key-value store holds per-run counts, the filters used, and any feed or item that failed.

### Common use cases

**Track launches and sentiment for your product.** Watch Show HN and the front page, and read the discussion rather than just the headline.

```json
{
 "feeds": ["showstories", "topstories"],
 "maxItems": 100,
 "includeComments": true,
 "commentDepth": 2
}
```

**Mine Ask HN for market research.** Ask HN threads are long-form problem statements from exactly the audience most B2D companies sell to.

```json
{
 "feeds": ["askstories"],
 "maxItems": 200,
 "includeComments": true,
 "commentDepth": 3,
 "minScore": 20
}
```

**Build a curated tech digest.** Take the highest-scoring stories only, skip comments, and run it on a schedule.

```json
{
 "feeds": ["beststories", "topstories"],
 "maxItems": 50,
 "minScore": 150,
 "includeComments": false
}
```

### Quick start

The front page, nothing else:

```json
{
 "feeds": ["topstories"],
 "maxItems": 30
}
```

Everything HN publishes, in one run:

```json
{
 "feeds": ["topstories", "newstories", "beststories", "askstories", "showstories", "jobstories"],
 "maxItems": 0
}
```

### Input

| Field | Type | Default | Notes |
|---|---|---|---|
| `feeds` | array | `["topstories"]` | Any of `topstories`, `newstories`, `beststories`, `askstories`, `showstories`, `jobstories`. Short forms `top`, `new`, `best`, `ask`, `show`, `job` are accepted and normalised. |
| `maxItems` | integer | `50` | Items per feed. `0` = every ID the feed lists. A negative value, a fraction, or anything that is not a number is rejected with a named error before the run fetches anything. |
| `includeComments` | boolean | `false` | Fetch and nest the reply tree. |
| `commentDepth` | integer | `2` | Reply levels to follow. Values outside `1`-`10` are clamped into it. Only used when `includeComments` is on. |
| `minScore` | integer | `0` | Drop items below this score. A negative value is read as `0`, meaning no floor. |

Combinations that make sense together:

- `newstories` + `minScore`, the firehose is mostly 1-point submissions, so a floor of 10-20 is what makes it usable.
- `includeComments` + a small `maxItems`, comment trees are where the requests go. 30 stories at depth 2 is a normal run; 500 stories at depth 10 is not.
- `jobstories` + `minScore: 0`, job posts carry a score of 1 by convention, so any floor above 1 silently empties the feed.

### Output example

```json
{
 "id": 49168622,
 "type": "story",
 "title": "How to Make a Nintendo 64 Game in 2026",
 "url": "https://phoboslab.org/log/2026/08/xibalba64-making-of",
 "hnUrl": "https://news.ycombinator.com/item?id=49168622",
 "text": null,
 "score": 137,
 "by": "atan2",
 "time": 1785849843,
 "postedAt": "2026-08-06T02:44:03.000Z",
 "descendants": 35,
 "commentCount": 12,
 "feed": "topstories",
 "comments": [
 {
 "id": 49195265,
 "by": "FlavioMacedo",
 "time": 1786015785,
 "postedAt": "2026-08-06T11:29:45.000Z",
 "text": "Great work! I didn't even know it was still possible to publish new games for old consoles like the N64.",
 "depth": 1,
 "replies": []
 }
 ],
 "scrapedAt": "2026-08-06T11:30:00.000Z"
}
```

`descendants` is HN's count for the entire thread; `commentCount` is how many this run actually retrieved at your chosen depth. When they differ, the gap is replies deeper than `commentDepth`.

`RUN_SUMMARY` looks like this:

```json
{
 "feedsRequested": ["topstories", "showstories"],
 "feedsIgnored": [],
 "feedsSucceeded": 2,
 "feedsFailed": 0,
 "perFeed": [
 {
 "feed": "topstories",
 "status": "ok",
 "listed": 500,
 "itemsAttempted": 50,
 "itemsFetched": 50,
 "itemsFailed": 0,
 "itemsEmpty": 0,
 "matched": 50,
 "saved": 50,
 "error": null,
 "lastItemError": null
 },
 {
 "feed": "showstories",
 "status": "ok",
 "listed": 197,
 "itemsAttempted": 93,
 "itemsFetched": 93,
 "itemsFailed": 0,
 "itemsEmpty": 0,
 "matched": 50,
 "saved": 50,
 "error": null,
 "lastItemError": null
 }
 ],
 "failures": [],
 "itemsSaved": 100,
 "commentsSaved": 1843,
 "commentsFailed": 0,
 "duplicatesSkipped": 7,
 "filters": { "maxItems": 50, "minScore": 0, "includeComments": true, "commentDepth": 2 },
 "finishedAt": "2026-08-06T11:34:12.417Z"
}
```

Each `perFeed.status` is one of `ok`, `listing-failed` (the feed list itself could not be read) or `all-items-failed` (the list came back but every item behind it did not). A feed that returned items your filters then removed is `ok`: that is a filter doing its job, not an outage.

### Limits and behaviour

- **Firebase returns one item per request.** A feed endpoint gives you an array of IDs and nothing else, so 500 stories means 500 calls. This Actor applies `maxItems` *before* fetching and runs a bounded pool of 8 concurrent requests, fast without hammering the API.
- **Comment trees fan out fast.** Each level multiplies the call count. `commentDepth` is capped at 10 for that reason, and each level of a tree is fetched concurrently rather than one comment at a time.
- **Duplicates across feeds are dropped.** A story on both `topstories` and `beststories` is saved once, tagged with the first feed it appeared in, and counted in `RUN_SUMMARY.duplicatesSkipped`, so you are never billed twice for the same item.
- **Deleted and dead items are skipped** rather than saved as empty records.
- **One failure never aborts the run.** A failed feed or item is recorded in `RUN_SUMMARY.failures` and the run continues. The Actor throws only when *every* requested feed came back empty because something broke — its listing failed, or every item request behind it did. A feed that returned rows your filters then removed is a success, and a run that saved nothing because every item request failed is not.
- **A cost control that cannot be read wrong.** `maxItems: 0` means the whole feed, so anything that would have to be rounded or reinterpreted to get there — a negative value, a fraction, a value that is not a number at all — is rejected outright rather than falling through to it. The error names the field and arrives before the first request.
- **Transient errors are retried.** 429 and 5xx get three attempts with linear backoff. A not-found response is treated as fatal and not retried.
- **Scores and comment counts are point-in-time.** A front-page story's score changes by the minute; re-run on a schedule if you need the trajectory.

### Finding an item on Hacker News

Every record carries `hnUrl`, which is `https://news.ycombinator.com/item?id=<id>`. The reverse also works: if you have a discussion URL, its `id` query parameter is the same ID this Actor returns, so you can join a record straight onto a link someone sent you.

### API example

```bash
curl -X POST "https://api.apify.com/v2/acts/arman-bd~hacker-news-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "feeds": ["topstories", "showstories"],
 "maxItems": 25,
 "minScore": 50
 }'
```

### JavaScript example

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/hacker-news-scraper').call({
 feeds: ['askstories'],
 maxItems: 50,
 includeComments: true,
 commentDepth: 2,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const item of items) console.log(`${item.score} pts, ${item.title} (${item.commentCount} comments)`);
```

**Defaults:** 1 GB memory, 15 min timeout.

### FAQ

**Do I need a proxy?** No. Proxy configuration is not required to run this Actor.

**Do I need a Hacker News account or API key?** No. You supply no credentials.

**What happens if a feed is unavailable?** It is recorded in `RUN_SUMMARY.failures`, marked `listing-failed` in `RUN_SUMMARY.perFeed`, and the run continues with the remaining feeds. Only a run in which no feed survived ends in failure.

**Can I get comments without the stories?** Not directly, comments are returned nested under the story they belong to, which is how HN models them. Flatten the `comments` tree client-side if you need a flat table.

**Why is `url` null on some items?** Ask HN posts and other self-posts have no external link; their content is in `text` instead.

**Why is `score` null on a comment-like item?** Only stories, jobs and polls carry a score. If you see this, the item was a poll option or a comment surfaced directly.

**Can I schedule it?** Yes, it is designed for scheduled runs. Diff on `id` plus `score` to track how a story moves up the front page.

**Can I integrate it with something else?** Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.

# Actor input Schema

## `feeds` (type: `array`):

Which Hacker News lists to read. 'topstories' is the front page, 'newstories' is the /newest firehose, 'beststories' is the highest-scoring recent posts, 'askstories' is Ask HN, 'showstories' is Show HN and 'jobstories' is the YC job board. Short forms (top, new, best, ask, show, job) work too.

## `maxItems` (type: `integer`):

Stop after this many items from each feed. Firebase serves one item per request, so this is also the main cost control. topstories, newstories and beststories hold up to 500 IDs; Ask HN and Show HN hold about 200; Jobs holds about 30. Set 0 to take everything the feed lists.

## `includeComments` (type: `boolean`):

Fetch replies for each story and nest them under a 'comments' field. Every comment is its own request, so a busy front-page thread can add hundreds of calls. leave this off unless you need discussion text.

## `commentDepth` (type: `integer`):

How many levels of replies to follow when 'Include comment trees' is on. 1 is top-level replies only, 2 adds replies to those replies, and so on. Each extra level multiplies the number of requests.

## `minScore` (type: `integer`):

Drop items scoring below this many points. Useful on 'newstories', where most submissions sit at 1-2 points. Leave at 0 to keep everything, including job posts, which have no meaningful score.

## Actor input object example

```json
{
  "feeds": [
    "topstories",
    "askstories"
  ],
  "maxItems": 50,
  "includeComments": false,
  "commentDepth": 2,
  "minScore": 100
}
```

# Actor output Schema

## `items` (type: `string`):

Every record the run produced.

## `runsummary` (type: `string`):

The RUN\_SUMMARY record from the run's key-value store.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "feeds": [
        "topstories",
        "showstories"
    ],
    "maxItems": 50,
    "commentDepth": 2
};

// Run the Actor and wait for it to finish
const run = await client.actor("arman-bd/hacker-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "feeds": [
        "topstories",
        "showstories",
    ],
    "maxItems": 50,
    "commentDepth": 2,
}

# Run the Actor and wait for it to finish
run = client.actor("arman-bd/hacker-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "feeds": [
    "topstories",
    "showstories"
  ],
  "maxItems": 50,
  "commentDepth": 2
}' |
apify call arman-bd/hacker-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,arman-bd/hacker-news-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/473Ya0Kb9tmTsuhf7/builds/eSI2D4mnTZDl2PFFB/openapi.json