Hacker News Scraper: Stories, Comments, Who's Hiring

Hacker News scraper returns public story, comment, search, or parsed hiring-post rows, depending on the selected mode.

Hacker News scraper uses the public source named below and writes its results to the Actor dataset.

What you can do

Mode	What it returns	Source
topstories, newstories, beststories, askstories, showstories, jobstories	The current list, in HN rank order (rank field)	Official Firebase API
search	Stories, comments, polls, Ask/Show HN matching a query, with date range, minimum points, minimum comments, type and author filters; relevance or newest-first	Algolia HN Search
comments	The whole comment tree of each story id or link you give, flattened to rows with depth, parent_id, reply_count	Algolia items endpoint (one request per tree)
whoishiring	One parsed row per job post in the latest (or any named) monthly thread, with the raw text kept next to the parsed fields	Algolia items endpoint

Output fields

Fields vary by result type and optional enrichment. The examples below are from sample_output.json.

Output field	Type	Example value or excerpt from saved sample
kind	string	"item"
id	integer	49962012
type	string	"story"
title	string or null	"Denmark Data Breach Exposes 8.8M People's Personal Data"
url	string or null	"https://www.cpr.dk/cpr-nyt/nyhedsarkiv/2026/okt/omfattende-uautoriseret-adgang-til-borgeres-cpr-oplysninger"
domain	string or null	"cpr.dk"
text	string or null	"oh, and a mac port is coming :)"
author	string	"clan"
points	integer or null	198
num_comments	integer or null	139
created_at	string	"2026-10-05T08:09:36Z"
created_at_i	integer	1791187776
parent_id	integer or null	8863
story_id	integer	49962012
story_title	string or null	"My YC app: Dropbox - Throw away your USB drive"
depth	integer or null	0
reply_count	integer or null	32
rank	integer or null	1
hn_url	string	"https://news.ycombinator.com/item?id=49962012"
error	null	null
company	string	"PrairieLearn (Remote US"
role	string or null	"Full-Stack Software Engineer"
location	string or null	"San Francisco, CA & New York City, NY"
remote	boolean	true
work_mode	string	"remote"
salary_text	string or null	"$100k-$180k"
visa	string or null	"(We’re not able to sponsor visas at this time.)"
visa_sponsorship	boolean or null	false
employment_type	array or null	["contract"]
apply_email	string or null	"eric+hn@sidekicklabs.com"
all_urls	array	["https://www.prairielearn.com"]
header	string	"PrairieLearn (Remote US) — Full-Stack Software Engineer — TypeScript / Postgres / React / AI"
header_structured	boolean	false
unclassified_segments	array or null	["TypeScript / Postgres / React / AI"]
raw_text	string	"PrairieLearn (Remote US) — Full-Stack Software Engineer — TypeScript / Postgres / React / AI\nPrair…
posted_at	string	"2026-10-01T15:02:37Z"
thread_id	integer	49922569
thread_title	string	"Ask HN: Who is hiring? (October 2026)"

Story, comment and search rows (kind: "item", event item):

Field	Meaning
id, type, hn_url	Item id, story / comment / poll / job, link to the HN page
title, url, domain	Story title, external link and its domain
text	Plain text of a comment, Ask HN or job body (text_html too if includeHtml)
author, points, num_comments	As shown on HN
created_at, created_at_i	UTC ISO time and Unix seconds
parent_id, story_id, story_title	Where a comment sits
depth, reply_count	Comment trees: 0 = top-level comment; null on the story row
rank	Position in a list mode
error	Set only on error rows, which are never charged

Who-is-hiring rows (kind: "job", event job):

Field	Meaning
company, role	From the post header (Company | Role | Location | ...)
location, remote, work_mode	work_mode is remote, hybrid, onsite or remote_or_onsite when the header says so
salary_text	The salary as written, for example $180k-$250k or 100k-160k GBP
visa, visa_sponsorship	The sentence(s) that mention visa / sponsorship / relocation; true / false / null when unclear
employment_type	full-time, part-time, intern, contract when named in the header
url, all_urls, apply_email	First non-HN link, other links, and the e-mail the post publishes for applying
header, header_structured, unclassified_segments	The first line, whether it had the usual pipe layout, header parts the parser could not place
raw_text	Full post text, always kept
author, posted_at, hn_url, thread_id, thread_title	Source of the post

The parsing is heuristic: HN does not force a format, so some fields can be empty or wrong. On the October 2026 thread (194 job posts returned) the fields were filled like this: company 100%, location 88%, URL 86%, role 72%, work mode 95%, salary 38%, visa mention 7%, apply e-mail 27%. Always check raw_text when a field matters. Off-topic top-level comments (replies, meta) are skipped and not charged.

Input example

{
  "mode": "search",
  "query": "sqlite",
  "type": "story",
  "searchSort": "date",
  "dateFrom": "2026-01-01",
  "minPoints": 100,
  "maxItems": 200
}

Who is hiring, remote Rust jobs from two months:

{ "mode": "whoishiring", "hiringMonths": ["latest", "2026-09"], "hiringRemoteOnly": true, "hiringKeywords": ["rust"] }

Full comment tree of a story:

{ "mode": "comments", "storyIds": ["8863", "https://news.ycombinator.com/item?id=49962012"], "maxCommentDepth": 0 }

Output example

{
  "kind": "job",
  "id": 49922693,
  "company": "Planlab.ai",
  "role": "Product Engineer (Design) & Platform Engineer",
  "location": "ONSITE in London, UK",
  "remote": false,
  "work_mode": "onsite",
  "salary_text": "100k-160k GBP",
  "visa_sponsorship": true,
  "url": "https://planlab.ai/jobs",
  "posted_at": "2026-10-01T15:08:44Z",
  "hn_url": "https://news.ycombinator.com/item?id=49922693"
}

A real 20-row sample (5 front-page stories, 4 comment rows, 3 search hits, 8 hiring posts) is in sample_output.json.

Pricing (pay per event)

Event	Price
item (story, comment or search row)	0.00025 USD
job (parsed Who-is-hiring post)	0.002 USD

No start fee. Error rows are free. Set maxItems and the run's maximum cost to cap spend. See PRICING.md for the comparison with other Hacker News actors.

Measured speed (local run, public APIs)

• Front page, 30 rows: 2.2 s; 100 rows: 2.8 s.
• Search, 1,200 newest-first stories across two Algolia windows: 4.2 s, no duplicates.
• Comment tree: 72 rows (the 2007 Dropbox thread, matches HN's own count) in 2.3 s; a 136-row tree in 1.9 s.
• Who is hiring, latest thread (October 2026): 194 job rows in 1.7 s.

Limits

• Algolia returns at most 1,000 hits per query. With searchSort: "date" the actor keeps going by moving the date window; relevance search stops at 1,000 and logs a warning.
• Hacker News data is what is public on the site; dead and deleted items are skipped.
• minPoints / minComments apply to search and list modes; they do not apply to comment trees or hiring posts.
• Comment trees of very large threads (thousands of comments) are one big response; keep memory at 512 MB or more for those.
• Requests are retried with exponential backoff. If an item still fails you get an error row instead of a crashed run.

FAQ

What limits the output? maxItems bounds saved rows. Comment depth and hiring month inputs further narrow the selected mode.

Do I need a proxy, login, or API key? No. It reads the public Hacker News Firebase and Algolia search APIs.

How is it priced? The proposed pay-per-event model charges item for story, comment, or search-hit rows and job for parsed hiring rows. Error rows have no custom event charge.

How are rate limits handled? The API client retries temporary failures with backoff. Source throttling or unavailable items can leave fewer rows than the requested cap.

How fresh are posts? The actor fetches public API data during each run. created_at or posted_at is the source posting time, not a guarantee that a listing remains open.

Why is a hiring field empty? Hiring headers vary; inspect raw_text before relying on parsed job fields.

Use with AI agents / MCP

Every mode is one JSON input and a flat JSON dataset, so an agent can call it as a tool: through the Apify MCP server (mcp.apify.com) add this actor, then ask for "the latest Who is hiring posts that mention Rust and are remote" or "all comments on this HN story". The input schema documents each field, and hiringKeywords, hiringRemoteOnly and maxItems keep the result small enough for a model context.