Hacker News Scraper: Stories, Search, Comments & Users

Collect Hacker News data through its two official free APIs — the Firebase feed that powers the site and the Algolia index behind its search box. No key, no login, no HTML parsing.

Mode	What you get
Story lists	Front page, newest, best, Ask HN, Show HN and Jobs, with a rank on every row
Search	Full-text search over every story and comment ever posted, with points, comment-count and date filters
Item + comments	Any story plus its whole reply tree, flattened with a depth number
Users	Karma, about text, join date and submission count

Why use this one

• Both official APIs in one tool. The Firebase API gives live, ranked lists; Algolia gives deep history and filtering. Most HN scrapers use only one and inherit its blind spot.
• Search the full archive. Points, comment count and date-range filters are passed straight through, so "every Show HN since January with 100+ points" is one run.
• Comment trees that terminate. The walk is breadth-first with a seen-set and hard caps, so a malformed reply chain can't spin forever or blow up your bill.
• Monitor a keyword. Turn on Sort search by newest, schedule the Actor, and every run returns the latest mentions.
• Flat output. Stories, comments and users come back as plain rows, not nested JSON.

How to use it

1. Choose What to scrape.
2. Fill in the matching field: Story lists, Search terms, Items or Users.
3. Set your caps — these drive both runtime and cost.
4. Click Start, then download results as JSON, CSV, Excel or HTML, or fetch them through the Apify API.

Example input: today's front page

{
  "mode": "stories",
  "storyLists": ["top"],
  "maxStoriesPerList": 30
}

Example input: every Show HN this year with 100+ points

{
  "mode": "search",
  "contentType": "showHn",
  "searchTerms": [""],
  "minPoints": 100,
  "postedAfter": "2026-01-01",
  "newestFirst": true,
  "maxResultsPerSearch": 500
}

Example input: a story and its whole discussion

{
  "mode": "item",
  "items": ["https://news.ycombinator.com/item?id=8863"],
  "includeComments": true,
  "commentDepth": 10,
  "maxCommentsPerItem": 1000
}

Output

Every row has a type field: story, comment, job, poll or user. The dataset ships with Stories & comments and Users views.

Story

{
  "type": "story",
  "id": 8863,
  "rank": 1,
  "title": "My YC app: Dropbox - Throw away your USB drive",
  "url": "http://www.getdropbox.com/u/2/screencast.html",
  "author": "dhouston",
  "points": 104,
  "commentCount": 71,
  "createdAt": "2007-04-04T19:16:40Z",
  "hnUrl": "https://news.ycombinator.com/item?id=8863",
  "storyList": "top"
}

Comment

{
  "type": "comment",
  "id": 9224,
  "parentId": 8863,
  "storyId": 8863,
  "commentDepth": 1,
  "text": "I have a few qualms with this app…",
  "author": "BrandonM",
  "createdAt": "2007-04-05T15:16:54Z",
  "hnUrl": "https://news.ycombinator.com/item?id=9224"
}

Pricing

Pay only for results:

Result	Price
Story, comment or job	$2.00 per 1,000
User profile	$4.00 per 1,000

Set a maximum spend per run in the run options and the Actor stops cleanly when it's reached.

Limits to know about

• One request per item. The Firebase API has no bulk endpoint, so 500 front-page stories costs 500 calls and a 1,000-comment thread costs about 1,000. Runtime and cost both scale with your caps, not with the number of stories you asked for.
• Search returns at most 1,000 hits per query. That is the index's own ceiling. Narrow with a date range to go deeper into the archive.
• List lengths are HN's. Top, new and best publish up to 500 ids; Ask, Show and Jobs publish about 200.
• Comment counts drift. commentCount is the story's descendants value, which can differ from the number of comments actually returned once dead and deleted ones are skipped.
• Deleted and flagged items come back with deleted or dead set to true and little else.
• storyId is filled on search hits and on walked comments, but the Firebase item endpoint only gives parentId, so a comment fetched on its own has no story id.

FAQ

Do I need an API key? No. Both APIs are public and free.

Which API does each mode use? Story lists, Item + comments and Users use Firebase. Search uses Algolia.

Can I track mentions of my product? Yes. Use Search with Sort search by newest, schedule it, and connect the dataset to Slack or email.

Why are some comments missing from a thread? Either your depth or comment cap was reached, or the comments were deleted or flagged dead.

Is this allowed? Both endpoints are published by Hacker News for exactly this purpose. The Firebase API is documented at github.com/HackerNews/API and the search index at hn.algolia.com/api. Output contains public usernames and post text, so handle it in line with any privacy law that applies to you.

Feedback

Found a bug or need a field that's missing? Open an issue on the Actor's Issues tab.