Hacker News Scraper: Stories, Comments & Front Page

Hacker News Scraper pulls structured records from the official Hacker News Firebase API, front page, newest, best, Ask HN, Show HN and the YC job board, with scores, authors, timestamps and optional nested comment trees.

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/hacker-news-scraper.md

What you get

Output field	Meaning
id, type	HN item ID and its type, story, job, poll or comment
title	Submission title
url	The link the story points at. null for Ask HN and text posts
hnUrl	Permalink to the discussion on news.ycombinator.com
text	Self-post body as clean plain text, HTML resolved and entities decoded. Links come through as their full address, and code blocks keep their indentation. null for link posts
score	Points at scrape time
by	Submitter's HN username
time, postedAt	Unix seconds as HN reports it, plus the same moment as an ISO 8601 string
descendants	HN's own total comment count for the whole thread
commentCount	How many comments this run actually fetched into comments, 0 when comment fetching is off
feed	Which feed the item came from
comments	Nested reply tree: id, by, time, postedAt, text, depth, replies
scrapedAt	Run timestamp

A RUN_SUMMARY record in the key-value store holds per-run counts, the filters used, and any feed or item that failed.

Common use cases

Track launches and sentiment for your product. Watch Show HN and the front page, and read the discussion rather than just the headline.

{
 "feeds": ["showstories", "topstories"],
 "maxItems": 100,
 "includeComments": true,
 "commentDepth": 2
}

Mine Ask HN for market research. Ask HN threads are long-form problem statements from exactly the audience most B2D companies sell to.

{
 "feeds": ["askstories"],
 "maxItems": 200,
 "includeComments": true,
 "commentDepth": 3,
 "minScore": 20
}

Build a curated tech digest. Take the highest-scoring stories only, skip comments, and run it on a schedule.

{
 "feeds": ["beststories", "topstories"],
 "maxItems": 50,
 "minScore": 150,
 "includeComments": false
}

Quick start

The front page, nothing else:

{
 "feeds": ["topstories"],
 "maxItems": 30
}

Everything HN publishes, in one run:

{
 "feeds": ["topstories", "newstories", "beststories", "askstories", "showstories", "jobstories"],
 "maxItems": 0
}

Input

Field	Type	Default	Notes
feeds	array	["topstories"]	Any of topstories, newstories, beststories, askstories, showstories, jobstories. Short forms top, new, best, ask, show, job are accepted and normalised.
maxItems	integer	50	Items per feed. 0 = every ID the feed lists. A negative value, a fraction, or anything that is not a number is rejected with a named error before the run fetches anything.
includeComments	boolean	false	Fetch and nest the reply tree.
commentDepth	integer	2	Reply levels to follow. Values outside 1-10 are clamped into it. Only used when includeComments is on.
minScore	integer	0	Drop items below this score. A negative value is read as 0, meaning no floor.

Combinations that make sense together:

• newstories + minScore, the firehose is mostly 1-point submissions, so a floor of 10-20 is what makes it usable.
• includeComments + a small maxItems, comment trees are where the requests go. 30 stories at depth 2 is a normal run; 500 stories at depth 10 is not.
• jobstories + minScore: 0, job posts carry a score of 1 by convention, so any floor above 1 silently empties the feed.

Output example

{
 "id": 49168622,
 "type": "story",
 "title": "How to Make a Nintendo 64 Game in 2026",
 "url": "https://phoboslab.org/log/2026/08/xibalba64-making-of",
 "hnUrl": "https://news.ycombinator.com/item?id=49168622",
 "text": null,
 "score": 137,
 "by": "atan2",
 "time": 1785849843,
 "postedAt": "2026-08-06T02:44:03.000Z",
 "descendants": 35,
 "commentCount": 12,
 "feed": "topstories",
 "comments": [
 {
 "id": 49195265,
 "by": "FlavioMacedo",
 "time": 1786015785,
 "postedAt": "2026-08-06T11:29:45.000Z",
 "text": "Great work! I didn't even know it was still possible to publish new games for old consoles like the N64.",
 "depth": 1,
 "replies": []
 }
 ],
 "scrapedAt": "2026-08-06T11:30:00.000Z"
}

descendants is HN's count for the entire thread; commentCount is how many this run actually retrieved at your chosen depth. When they differ, the gap is replies deeper than commentDepth.

RUN_SUMMARY looks like this:

{
 "feedsRequested": ["topstories", "showstories"],
 "feedsIgnored": [],
 "feedsSucceeded": 2,
 "feedsFailed": 0,
 "perFeed": [
 {
 "feed": "topstories",
 "status": "ok",
 "listed": 500,
 "itemsAttempted": 50,
 "itemsFetched": 50,
 "itemsFailed": 0,
 "itemsEmpty": 0,
 "matched": 50,
 "saved": 50,
 "error": null,
 "lastItemError": null
 },
 {
 "feed": "showstories",
 "status": "ok",
 "listed": 197,
 "itemsAttempted": 93,
 "itemsFetched": 93,
 "itemsFailed": 0,
 "itemsEmpty": 0,
 "matched": 50,
 "saved": 50,
 "error": null,
 "lastItemError": null
 }
 ],
 "failures": [],
 "itemsSaved": 100,
 "commentsSaved": 1843,
 "commentsFailed": 0,
 "duplicatesSkipped": 7,
 "filters": { "maxItems": 50, "minScore": 0, "includeComments": true, "commentDepth": 2 },
 "finishedAt": "2026-08-06T11:34:12.417Z"
}

Each perFeed.status is one of ok, listing-failed (the feed list itself could not be read) or all-items-failed (the list came back but every item behind it did not). A feed that returned items your filters then removed is ok: that is a filter doing its job, not an outage.

Limits and behaviour

• Firebase returns one item per request. A feed endpoint gives you an array of IDs and nothing else, so 500 stories means 500 calls. This Actor applies maxItems before fetching and runs a bounded pool of 8 concurrent requests, fast without hammering the API.
• Comment trees fan out fast. Each level multiplies the call count. commentDepth is capped at 10 for that reason, and each level of a tree is fetched concurrently rather than one comment at a time.
• Duplicates across feeds are dropped. A story on both topstories and beststories is saved once, tagged with the first feed it appeared in, and counted in RUN_SUMMARY.duplicatesSkipped, so you are never billed twice for the same item.
• Deleted and dead items are skipped rather than saved as empty records.
• One failure never aborts the run. A failed feed or item is recorded in RUN_SUMMARY.failures and the run continues. The Actor throws only when every requested feed came back empty because something broke — its listing failed, or every item request behind it did. A feed that returned rows your filters then removed is a success, and a run that saved nothing because every item request failed is not.
• A cost control that cannot be read wrong. maxItems: 0 means the whole feed, so anything that would have to be rounded or reinterpreted to get there — a negative value, a fraction, a value that is not a number at all — is rejected outright rather than falling through to it. The error names the field and arrives before the first request.
• Transient errors are retried. 429 and 5xx get three attempts with linear backoff. A not-found response is treated as fatal and not retried.
• Scores and comment counts are point-in-time. A front-page story's score changes by the minute; re-run on a schedule if you need the trajectory.

Finding an item on Hacker News

Every record carries hnUrl, which is https://news.ycombinator.com/item?id=<id>. The reverse also works: if you have a discussion URL, its id query parameter is the same ID this Actor returns, so you can join a record straight onto a link someone sent you.

API example

curl -X POST "https://api.apify.com/v2/acts/arman-bd~hacker-news-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "feeds": ["topstories", "showstories"],
 "maxItems": 25,
 "minScore": 50
 }'

JavaScript example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/hacker-news-scraper').call({
 feeds: ['askstories'],
 maxItems: 50,
 includeComments: true,
 commentDepth: 2,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const item of items) console.log(`${item.score} pts, ${item.title} (${item.commentCount} comments)`);

Defaults: 1 GB memory, 15 min timeout.

FAQ

Do I need a proxy? No. Proxy configuration is not required to run this Actor.

Do I need a Hacker News account or API key? No. You supply no credentials.

What happens if a feed is unavailable? It is recorded in RUN_SUMMARY.failures, marked listing-failed in RUN_SUMMARY.perFeed, and the run continues with the remaining feeds. Only a run in which no feed survived ends in failure.

Can I get comments without the stories? Not directly, comments are returned nested under the story they belong to, which is how HN models them. Flatten the comments tree client-side if you need a flat table.

Why is url null on some items? Ask HN posts and other self-posts have no external link; their content is in text instead.

Why is score null on a comment-like item? Only stories, jobs and polls carry a score. If you see this, the item was a poll option or a comment surfaced directly.

Can I schedule it? Yes, it is designed for scheduled runs. Diff on id plus score to track how a story moves up the front page.

Can I integrate it with something else? Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.