Hacker News Scraper: Search Stories, Comments and Front Page (Official API)

Search Hacker News stories and comments, read the front page and the top, best, new, Ask HN and Show HN lists, or pull the whole comment tree of any story. It reads the official Hacker News Search API and the official Hacker News Firebase API. No browser, no proxies, no login.

Every record carries verify_url, the item's own page on news.ycombinator.com, so you can check any row by hand in seconds.

What people use it for

• Track mentions of a product, library or company: search "your-product" over stories and comments, newest first, since a date.
• Find every submission from a site: match the submitted link (for example github.com) with a minimum number of points.
• Watch the front page: take the top 30 with rank, points and comment counts on a schedule.
• Read a whole discussion: give a story id and get every comment in reading order with its depth and parent.
• Feed an AI agent: one call returns clean plain text for each comment, plus the original HTML if you need it.

An AI agent calls this tool when it needs current or historic Hacker News discussion as structured data it can quote and cite, with a link back to the source item on every record.

Sample output

Input:

{ "mode": "story_comments", "story_ids": [8863], "max_items": 3 }

Output (three records, shortened):

{
  "type": "story",
  "id": 8863,
  "title": "My YC app: Dropbox - Throw away your USB drive",
  "url": "http://www.getdropbox.com/u/2/screencast.html",
  "domain": "getdropbox.com",
  "author": "dhouston",
  "points": 104,
  "num_comments": 71,
  "created_at": "2007-04-04T19:16:40Z",
  "verify_url": "https://news.ycombinator.com/item?id=8863"
}
{
  "type": "comment",
  "id": 8865,
  "story_id": 8863,
  "parent_id": 8863,
  "author": "dhouston",
  "text": "oh, and a mac port is coming :)",
  "created_at": "2007-04-04T19:22:55Z",
  "depth": 1,
  "verify_url": "https://news.ycombinator.com/item?id=8865"
}
{
  "type": "comment",
  "id": 9007,
  "story_id": 8863,
  "parent_id": 8865,
  "author": "vlad",
  "text": "Drew, this is awesome! All of the features you mentioned are exactly what people need.",
  "created_at": "2007-04-05T01:48:11Z",
  "depth": 2,
  "verify_url": "https://news.ycombinator.com/item?id=9007"
}

You can check this against the source: open https://news.ycombinator.com/item?id=8863. The title, link, submitter, the creation time (hover over the age) and the 71 comments match. Comment 9007 sits under comment 8865.

A handled error. Input { "max_items": 0 } fails the run immediately with the message max_items must be at least 1. Got 0. and nothing is charged.

Modes

mode	What you get
search	Keyword search over stories, comments or both. Filter by dates, minimum points and minimum comments. Sort by relevance or date.
front_page	The 30 stories on the first page right now, with rank.
top, best, new, ask, show	The official Hacker News lists in rank order, up to the size of the list (about 500 for top and new).
story_comments	The full comment tree of each story or comment id you give, in reading order, with depth and parent_id.

Input

Field	Type	Default	Meaning
mode	string	search	One of the modes above.
query	string	web scraping	Search mode. Empty matches everything. Use "double quotes" for an exact phrase and a leading - to exclude a word. Up to 500 characters.
content_type	string	stories	Search mode. stories, comments or both. With both, stories take up to half of max_items and comments take the rest.
sort_by	string	relevance	Search mode. relevance returns at most 1,000 results. date (newest first) goes up to 10,000.
search_in	string	all	Search mode. all, title, url (the submitted link) or text.
story_tag	string	any	show_hn, ask_hn or poll to narrow stories.
date_from, date_to	string	none	YYYY-MM-DD, UTC. date_to includes the whole day.
min_points, min_comments	integer	none	Stories only. Comments have no public points.
max_items	integer	25	Most records returned, and the most you can be charged for. 1 to 10,000.
story_ids	list	none	Required for story_comments. Item numbers or news.ycombinator.com/item?id= links. Up to 200.
include_story	boolean	true	story_comments only. Adds the story as the first record of each thread.

Output

Story records:

Field	Meaning
type	story
id	Hacker News item number
title, url, domain	What was submitted. url and domain are null for Ask HN and text posts. domain is the link's host without www.
author	The public handle that posted it
points, num_comments	As of the time of the run
created_at, created_at_unix	UTC
story_text, story_text_html	Body of text posts as plain text and as the original HTML
tags	story or poll, plus show_hn or ask_hn. In list modes these are rebuilt from the title, because that feed has no tags.
rank	Position in the list (list modes only)
verify_url	The item on news.ycombinator.com
source	algolia or firebase, the service the record came from

Comment records: type (comment), id, story_id, story_title, parent_id, author, text (plain), text_html (original), points (usually null), created_at, created_at_unix, tags, depth (thread mode only), verify_url, source.

Plain text keeps paragraph breaks and code blocks, turns entities into characters, and replaces shortened link text (https://example.com/a...) with the full address.

A summary of each run, including any notes such as a story that was not found, is saved in the key-value store as RUN_SUMMARY.

What it does not do

• No user data beyond the author handle. No profiles, karma or "about" text are fetched, and you cannot search by username.
• No job posts. Job items and "who is hiring" threads are not returned by the lists.
• No dead, flagged or deleted items. A deleted comment is skipped but its replies are kept.
• Relevance search stops at 1,000 results. That is a limit of the search service. Use sort_by: date with date ranges to go further.
• Very new items can take a few minutes to appear in search. The official lists (front_page, top, new and the others) read the live feed.
• It does not post, vote or log in.

Pricing

Pay per result: $0.0002 per record returned ($0.20 per 1,000), plus the platform's small start charge. You are not charged for a run that fails on bad input or for an item that was not found. Set max_items or a maximum charge per run to cap spend.

Reliability

Search and full threads go to the Hacker News Search API (hn.algolia.com), one request at a time with a short pause. The lists go to the official Firebase API (hacker-news.firebaseio.com). If either service answers with a rate limit or an error, the run backs off twice (5 s, then 15 s) and then stops with the records it already has, instead of pushing harder.

Data source

The Hacker News Search API and the Hacker News Firebase API, both public and documented. Hacker News is operated by Y Combinator. This tool is not affiliated with Y Combinator or Algolia. Content belongs to the people who posted it; check the terms of use before republishing it.