Hacker News Scraper

Search the entire Hacker News archive, pull the current front page, expand whole comment trees, and look up user profiles. No API key, no login.

---

Why this scraper

Search past the 1,000-result cap. Hacker News search returns at most 50 pages — 1,000 results — no matter how many matches exist. Ask for 10,000 posts about Rust and every scraper built the obvious way stops at 1,000. This one walks a moving time window instead and keeps going, verified duplicate-free.

Comments as readable text. Hacker News stores comments as HTML with escaped entities. Exported raw, a CSV is full of ' and <p> tags. Here it arrives as plain text.

Breadth-first comment trees. When you limit how many comments to fetch, you want the top of the discussion, not one deep thread. Comments come back level by level, each row tagged with its depth.

---

What you can extract

Mode	What you get
Search the whole archive	Every story or comment matching a keyword, filterable by points, comment count and date range
Current front page or list	Top, newest, best, Ask HN, Show HN or job listings as they stand right now
Comments on a story	The full comment tree, flattened, with depth and parent for each reply
User profiles	Karma, join date, submission count and bio

---

Quick start

1. Pick Search the whole archive.
2. Type a keyword, or leave it empty and set Minimum points to 100.
3. Set Maximum results, click Start, download as CSV or JSON.

---

Example inputs

Everything ever posted about a topic

{
  "mode": "search",
  "searchQuery": "postgres",
  "sortBy": "date",
  "maxItems": 10000
}

Only stories that actually landed

{
  "mode": "search",
  "searchQuery": "startup",
  "minPoints": 100,
  "maxItems": 2000
}

Track a competitor being discussed

{
  "mode": "search",
  "searchQuery": "\"your product name\"",
  "contentType": "comment",
  "maxItems": 1000
}

Show HN launches in a date range

{
  "mode": "search",
  "contentType": "show_hn",
  "since": "2026-01-01",
  "until": "2026-06-30",
  "maxItems": 3000
}

Today's front page

{
  "mode": "frontPage",
  "listName": "topstories",
  "maxItems": 30
}

A whole discussion

{
  "mode": "comments",
  "storyIds": ["https://news.ycombinator.com/item?id=49195231"],
  "maxDepth": 10,
  "maxItems": 2000
}

---

Output fields

Field	Description
id, type	Item ID and whether it is a story, comment, poll or job
title	Story headline; empty for comments
url, domain	The linked article and its domain
hnUrl	Link to the Hacker News discussion
text	Comment or self-post content, as plain text
author	Who posted it
points, numComments	Upvotes and comment count
createdAt, createdAtTimestamp	When, as ISO date and Unix timestamp
storyId, parentId	For comments: which story and which parent
depth	Comment mode: 1 is a top-level reply
childCount	Direct replies to this item
tags	Categories such as story or show_hn
username, karma, about, submissionCount	User mode fields

---

Input notes

• Stories accept an item ID or a full news.ycombinator.com link.
• Sort order matters: newest first pages through the entire archive, most relevant stops at 1,000 results — that is a limit of the search backend, not of this Actor.
• Several stories split the maxItems budget evenly between them.
• Quoted phrases work in the search query: "rust vs go".

---

Limits

• Points are snapshots. Scores change; a story scraped today may differ tomorrow.
• Relevance sort caps at 1,000. Use newest first for full coverage.
• Deleted and dead comments are skipped rather than returned as empty rows.
• Very deep trees on huge threads take time — one request per comment.

---

FAQ

Do I need an API key? No. Hacker News runs both of its interfaces openly.

Can I really get more than 1,000 search results? Yes, with newest first. That is the main thing this Actor does differently.

How far back does the archive go? To 2007, when Hacker News launched.

Can I search comments, not just stories? Yes — set Stories or comments to Comments.

What formats can I export? CSV, Excel, JSON, XML, or straight from the Apify API.

---

Legal and responsible use

This Actor reads publicly available data through Hacker News's own public interfaces. It does not bypass authentication or access private content.

Usernames and comment text are personal data under GDPR. You remain responsible for how you store, analyse and republish the output.