Hacker News Stories & Comments Scraper

Export Hacker News stories and comments as rows with stable IDs and parent links. Choose Top, New, Best, Ask HN, Show HN or Jobs, or add exact item IDs. No Hacker News API key is needed.

Start with the small example already in the Input form. Open the Dataset to read the results and the run summary to check missing or partial trees. Pricing is $0.05 per start plus $0.00027 per saved useful entity. A comment is a separate entity. Comment limits can make a tree partial.

Table of contents

• What it does
• Data you get
• Quick start
• Input
• Input examples
• Output example
• Output fields
• Pricing
• Use cases
• Repeated exports
• API and export
• Limits and partial results
• Troubleshooting
• FAQ
• Related Actors
• Support

What it does

The Actor reads the public Hacker News API. It saves each root item, then its bounded replies. Stories, jobs, self posts and comments keep their source values. Replies keep their IDs, so you can rebuild a discussion tree.

It fetches only the fixed HN API origin. Article links are returned as data. It does not open those articles. Full-text search, user profiles, votes, posts and a full historical archive are outside this product's scope.

Data you get

• Story title, external link, HN discussion link, author, time, score and source comment count.
• Comment ID, parent ID, root ID and depth.
• Original HTML in textHtml and readable plain text in text.
• Deleted and dead flags, plus clear rows for items that are unavailable.
• A summary of each root tree, skipped edges, requests, retries and run limits.

Quick start

1. Open Input in Apify Console.
2. Keep the new form as it is: Top stories, 3 roots, up to 5 comments per root and depth 2.
3. Keep memory at 256 MB and timeout at 300 seconds. Set a maximum cost of $0.15 for this small example.
4. Click Start.
5. Open Dataset. Look for a story row and comment rows linked by rootId and parentId.
6. Open Key-value store → RUN_SUMMARY. A current busy story can be partial because the example uses small caps.

Repeat the same input to get a fresh export. The Actor has no saved global baseline, so it does not hide unchanged items.

Input

All fields are optional. API input {} uses the runtime defaults below. The new form uses schema prefills with the same values. Restore example input is a separate Console action.

Key	Type	Runtime default / UI prefill	Allowed values	Effect and recommendation
listMode	string	top	top, new, best, ask, show, job	Choose the official list. Use ask for questions or show for projects.
itemIds	integer array	[]	Up to 100 positive safe integers	Exact IDs replace the list when this array is nonempty. Use [8863], not URLs or string IDs. Duplicates are fetched once.
maxStories	integer	3	1–100	Maximum distinct root IDs, in list or input order. This also applies to exact comments and jobs. Start small.
includeComments	boolean	true	true / false	Fetch replies as separate rows. Use false for a story-only export.
maxCommentsPerStory	integer	5	0–200	Cap all descendant observations per root, including deleted, unavailable and error nodes. Zero means zero comments.
maxDepth	integer	2	0–10	Root depth is 0. Direct reply depth is 1. Zero reads roots only. Raise it when nested replies matter.
maxNodes	integer	30	1–1000	Total distinct roots and descendants reserved for fetch. Other time, request and byte caps can stop sooner.

Empty itemIds selects listMode. Unknown fields, null input fields, invalid types and IDs are rejected. IDs must be integers from 1 to 9007199254740991. The value must also exist on HN to return a live entity.

When caps interact, the first reached cap wins. maxStories selects roots before fetching. maxNodes includes roots. A small node cap may leave later roots unread. The summary shows selected, fetched counts and the run reason. It never adds a fake row for an unread item.

Input examples

Current discussions

{
  "listMode": "top",
  "itemIds": [],
  "maxStories": 3,
  "includeComments": true,
  "maxCommentsPerStory": 5,
  "maxDepth": 2,
  "maxNodes": 30
}

Returns up to 18 observed nodes. Busy trees can be partial. List membership and live scores change between runs.

One known story and its replies

{
  "itemIds": [8863],
  "maxStories": 1,
  "includeComments": true,
  "maxCommentsPerStory": 100,
  "maxDepth": 10,
  "maxNodes": 110
}

Returns the historical Dropbox story and its bounded comments. Read treeStatus to see whether all observed children were fetched. The source's descendants can differ from collectedComments because deleted or dead nodes are also observed.

Jobs without replies

{
  "listMode": "job",
  "maxStories": 10,
  "includeComments": false,
  "maxNodes": 10
}

Returns up to 10 job items. Job descendants and text can be null. A direct comment ID is also supported; storyId is null because the Actor does not fetch its ancestors.

Output example

These are selected fields from one real run with the known-story example. The live reply below has ID 8917 and links to the story with parentId. Export the full rows to keep every field.

[
  {
    "entityType": "story",
    "id": 8863,
    "type": "story",
    "rootId": 8863,
    "storyId": 8863,
    "parentId": null,
    "depth": 0,
    "title": "My YC app: Dropbox - Throw away your USB drive",
    "hnUrl": "https://news.ycombinator.com/item?id=8863",
    "url": "http://www.getdropbox.com/u/2/screencast.html",
    "author": "dhouston",
    "time": 1175714200,
    "score": 104,
    "descendants": 71,
    "status": "complete",
    "treeStatus": "complete",
    "collectedComments": 72
  },
  {
    "entityType": "comment",
    "id": 8917,
    "rootId": 8863,
    "storyId": 8863,
    "parentId": 8863,
    "depth": 1,
    "status": "complete"
  }
]

The output keeps Unix seconds in time and adds UTC ISO time in createdAt. Null means the source did not provide that value. Each row has every documented field.

Output fields

Field	Type	Null?	Example	Meaning
entityType	string	No	comment	story includes story/job/poll roots; other values: comment, item, unavailable, error.
id	integer	No	8863	Stable HN item ID.
type	string	Yes	story	Source type: story, job, comment, poll or pollopt. A deleted or unavailable item can have no type.
status	string	No	partial	complete, partial, not_found or error.
reason	string	Yes	comment_cap	Reason for an incomplete or unavailable result. Null for no row-level issue.
rootId	integer	No	8863	Input/list item whose subtree was visited.
storyId	integer	Yes	8863	Story/job/poll root ID. Null for a direct comment or unknown root.
parentId	integer	Yes	8863	Source parent ID. An unavailable child uses its expected edge parent. A missing source parent is flagged.
depth	integer	No	1	Distance from the selected root, which has depth 0.
title	string	Yes	My YC app: Dropbox...	Source title. Comments usually have no title.
hnUrl	string	No	https://news.ycombinator.com/item?id=8863	HN discussion or item link.
url	string	Yes	http://www.getdropbox.com/u/2/screencast.html	Source external link. Self posts and comments may have none. The Actor does not fetch it.
author	string	Yes	dhouston	Public HN username from by.
time	integer	Yes	1175714200	Source Unix seconds, not milliseconds.
createdAt	string	Yes	2007-04-04T19:16:40.000Z	Source time converted to UTC ISO.
score	integer	Yes	104	Source score. It can change during and after a run.
descendants	integer	Yes	71	Source comment count. This is not the number of rows collected.
kids	integer array	No	[9224,8917]	Source child IDs in ranked order. This example is shortened. Empty if not provided.
textHtml	string	Yes	Hello<p>HN	Original source HTML, kept as data.
text	string	Yes	Hello\nHN	Plain text with block boundaries and one entity decode. Script/style content is omitted.
deleted	boolean	No	true	Source deleted flag; false when absent.
dead	boolean	No	false	Source dead flag; false when absent.
retrievedAt	string	No	2026-10-07T03:53:28.715Z	Time this node was observed, in UTC ISO.
treeStatus	string	No	partial	Root fetch completeness: complete, partial, disabled or not_applicable. Non-root rows use not_applicable.
treeReasons	string array	No	["depth_cap"]	Root tree reasons. Empty for no tree issue.
collectedComments	integer	Yes	72	Observed descendants, including status/deleted/dead rows, for a traversed root. Null for non-root/unsupported items.

Replies use breadth-first order within each root: source children first, then their children. Parent rows appear before children in a normal export. IDs are deduplicated across the whole run. A malformed edge or cycle is marked partial rather than fetched in a loop.

Where to find results

Surface	Contents
Default Dataset	Saved entity and status rows. Export JSON or CSV.
KVS OUTPUT	{ "rows": [...], "summary": {...} }; rows match the Dataset.
KVS RUN_SUMMARY	The same summary, without the rows.

The summary has status, reason, sourceCommit, input, requested, selected, duplicatesRemoved, fetched, failed, notFound, savedRows, usefulRows, requests, attempts, retries, responseBytes, internalDeadlineSecs, startedAt, finishedAt, durationSecs, limits, resourceError and billing.

requested is the number of IDs in the selected source list or exact input. selected is the number after deduplication and maxStories. requests counts logical fetch calls; attempts counts actual HTTP attempts, including retries. fetched counts successful item responses, including null. failed counts technical failures. notFound counts null responses. usefulRows excludes dead, deleted, unavailable and error rows.

roots has one entry for each observed root: id, status, treeStatus, reasons, collectedComments, observedNodes, unavailableNodes, failedNodes and skippedEdges. Caps can skip edges without fetching them. billing has entityEvents and chargeLimitReached. resourceError is separate from an input error. Early validation failures have a shorter error summary and keep the original input.

Pricing

The price is $0.05 per Actor start + $0.00027 per useful entity saved ($0.27 per 1,000 useful entities). Platform usage is included in these event prices for standard runs. Memory is fixed at 256 MB.

A story, job or comment is one useful entity if it is live and saved. Deleted, dead, unavailable and error rows do not get an entity fee. The start fee still applies when no useful rows are returned. A result cap does not make source work free. A partial run is charged for its start and useful rows already saved. A failed or cancelled run may also have the start fee and saved entity fees.

Useful entities saved in one run	Start	Entity fee	Total
10	$0.05	$0.00270	$0.05270
100	$0.05	$0.02700	$0.07700
1,000	$0.05	$0.27000	$0.32000
0	$0.05	$0	$0.05

The 1,000-entity line is a price calculation at the node cap. A real 1,000-node export may include uncharged status/deleted/dead rows. Use the actual useful count for its price. There are no content filters or an unchanged-only monitoring mode.

Set Maximum cost per run before starting. The Actor stops saving paid entities when the remaining budget cannot cover one more event. It does not charge for unsaved entities. Fetch work may already have happened before this stop. Dataset writing and event charging are separate platform operations: an interrupted acknowledgement can leave a saved row without a charge; the Actor does not replay the write itself.

Use cases

Goal	Input choice	Fields to use
Track current tech discussions	top, new or best	id, title, score, descendants, retrievedAt
Study project feedback	show or exact itemIds	rootId, parentId, depth, text, treeStatus
Build a public discussion corpus	Exact IDs, bounded comments	id, textHtml, text, deleted, dead, summary
Export HN jobs	job, comments off	title, url, author, createdAt

Repeated exports

Run the same input again and compare exports by id. Every run reads current API values. Use retrievedAt to separate observation time from post time. A partial export cannot prove that a missing row was removed from HN.

If you choose to set an Apify schedule, use the same small input and a cost cap. The Actor creates no schedules and keeps no global baseline. For a complete setup guide, see Apify schedules.

API and export

Use your Apify account token to start the Actor through the Apify API. You do not need a Hacker News token. Do not put your Apify token in the input.

curl -X POST \
  'https://api.apify.com/v2/acts/cliqtomedia~hacker-news-stories-comments-scraper/runs?timeout=300&memory=256&maxTotalChargeUsd=0.15' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"listMode":"top","maxStories":3,"maxCommentsPerStory":5,"maxDepth":2,"maxNodes":30}'

Read the run's defaultDatasetId to download saved rows. Read defaultKeyValueStoreId for OUTPUT and RUN_SUMMARY. Your HTTP client needs an outbound connection to Apify. Use the Apify API reference for run polling and Dataset export.

JSON preserves arrays and nulls. CSV is useful for flat entity fields. Use the same input object in an SDK, MCP Actor tool or your own automation, then join rows by id, rootId and parentId. Setup and permissions depend on that integration; no integration or webhook is created by this Actor.

Limits and partial results

• At most 100 roots, 200 descendant observations per root, depth 10 and 1,000 total reserved nodes. A live Cloud export reached 1,000 observed nodes within the configured limits; this is a cap, not a guaranteed amount.
• Only the six official lists are supported. Lists can be smaller than maxStories and can include jobs. Their order can change while the run is active.
• Source work has a 90-second deadline, at most 1,300 HTTP attempts, concurrency 4, an 8-second request timeout, a 2 MiB response cap and an 8 MiB total response cap. Temporary transport, 429 and 5xx errors are retried up to three attempts within these caps.
• Set platform timeout to 300 seconds. A longer finite timeout also works but does not increase the 90-second source deadline. A smaller timeout reduces the source window to leave time for saved output.
• comment_cap, depth_cap, node_cap, request_cap, byte_cap, response_byte_cap, deadline and charge_cap show a limit. unavailable_child, missing_parent, parent_mismatch, unexpected_child_type and duplicate_or_cycle show an observed source/graph issue.
• treeStatus describes what was fetched. If the charge cap stops saving rows, use run-level status=partial to see that the export is incomplete.
• Original HTML is data. Do not render textHtml, titles or external URLs as trusted HTML in your own app. Use escaped text for display. The Dataset table displays plain text.
• Scores and comments are live. The API is not an atomic snapshot. Dead/deleted/null states may change after a run. A hard platform kill can prevent final KVS writes; rows already saved remain in the Dataset.

Troubleshooting

Symptom	Likely reason	Check	Action
partial tree	A cap stopped children	Root treeReasons, source kids and run summary	Raise the relevant input cap within allowed values, then run again.
Only unavailable rows	HN returned null for the IDs	id, status, notFound	Check the HN item link or use a current list.
Fewer comments than descendants	Caps, dead/deleted items or a changing source	collectedComments, flags and tree reasons	Keep source count and collected count separate.
Run failed before source work	Invalid input or unsupported resources	reason, resourceError	Use numeric IDs, known fields and 256 MB memory.
Source error rows	Request failures or malformed source data	Row reason, failed, retries	Retry a small input. Do not treat this as no matching stories.
Export stopped at budget	Not enough budget for the next entity	billing.chargeLimitReached	Check saved rows and raise the next run's cap if needed.

FAQ

Do I need an API key?

No HN API key, cookie or login is needed. You need your Apify account to run the Actor. API clients use an Apify token in an authorization header, never in this Actor's input.

Does it return every comment?

Only when every reachable child is observed within the caps and treeStatus is complete. The new form intentionally uses small limits. Check the summary for each root. descendants is the source count, not a completeness check.

Why are HTML and plain text different?

textHtml keeps the source. text removes tags, adds block boundaries and decodes entities once. < becomes <, not an HTML tag. Deleted items can have neither value.

Does it fetch the linked article?

No. url is the external link from HN. This product exports HN items and discussions.

What do deleted, dead and null mean?

Deleted and dead are source flags. A null response means the item was unavailable at fetch time. It does not prove why. These observations are kept; they are not silently dropped and do not have an entity fee.

Why is there a start fee?

The scan still makes requests and saves a summary when there are no useful entities. The start fee covers this bounded work. Each fresh run has its own start fee.

Is it a replacement for another HN scraper?

Use a field mapping before switching. This Actor uses numeric IDs, url for the external link and hnUrl for HN. Comments are flat rows with stable IDs, not nested message arrays.

Related Actors

• RSS & Atom Feed Reader / Monitor — collect feed entries alongside HN stories.
• Link Route Audit — Bulk HTTP Status & Redirects — check selected article URLs in a separate, bounded run.

Support

Use this Actor's Issues tab in Apify Store for questions or defects. Include the run ID, input mode and a short error reason. Keep tokens and private account information out of the report.