Hacker News Scraper: Search, Stories & Comments

Search every story and comment ever posted on Hacker News by keyword, type, date, points and author. Read the live top, new, best, Ask HN, Show HN and job lists, or fetch any story or comment by its ID, with its whole comment thread if you want it. One clean row per item, ready for a spreadsheet, a database or an AI pipeline.

It reads Hacker News's two public APIs: the Algolia HN Search API (the search behind hn.algolia.com) and the official HN API on Firebase. No browser, no proxies, no logins, and nothing to break when the page layout changes.

$0.50 per 1,000 items. You pay only for stories, comments, jobs and polls saved. IDs that don't exist, deleted or dead items and unusable inputs still get a row, free.

Who it's for

• Founders and marketers: find every discussion of your product, a competitor or a market, with points and comment counts.
• Researchers and data teams: build datasets of tech discussion by topic and date, going back to 2006.
• AI and RAG pipelines: comment threads as plain text, with parentId, storyId and depth to rebuild each conversation.
• Newsletters, alerts and dashboards: schedule a run on the front page or a search sorted by date, and send new items to Sheets, Slack, Make, Zapier or n8n.

How it compares

	This Actor	Hacker News scrapers that read the website
Source	Official public APIs (Algolia HN Search and HN Firebase)	news.ycombinator.com pages
Search all of HN history	Yes, stories and comments since 2006	Usually only the pages you point it at
Filters	type (story, comment, Ask HN, Show HN, job, poll, front page), date range, minimum points and comments, author	varies by tool
More than 1,000 results per search	Yes, split into date windows for you	varies by tool
Comment threads	Flattened rows with parentId, storyId, depth	varies by tool
Text	HTML turned into plain text, shortened links expanded	varies by tool
You pay for	saved items only	varies by tool

Input

Field	What it does
Search queries	Keywords, one search per line, e.g. rust, "vector database". Searches titles, links, story text and comments. Repeated queries are run once.
What to search	Stories (default), comments, Ask HN, Show HN, jobs, polls, stories that made the front page, or everything.
Sort by	Relevance (Algolia's ranking, which favours points and comments) or newest first.
From date / To date	2024-01-31 or 2024-01-31T18:00:00Z, in UTC. A bare end date includes that whole day.
Minimum points / Minimum comments	Only stories above these numbers. Comments have no public points, so these are ignored for comment searches.
Author	A Hacker News username, e.g. pg. Case-sensitive. With no query, lists everything that user posted of the chosen type.
Max results per search	1 to 50,000 (default 100).
Live lists	top (the front page), new, best, ask, show, job, in their current order.
Max items per list	From the top of each list (default 30). The top and new lists hold up to 500, the others up to 200.
Item IDs or links	8863 or https://news.ycombinator.com/item?id=8863. Stories, comments, jobs, polls. Only news.ycombinator.com links are read.
Include comments	Also save each story's comments right after it. Each comment is a charged row.
Max comments per story	Stop each thread after this many comments (default 100), read in thread order.

Leave Search queries empty and set a filter (author, dates, points) to list everything that matches the filters. Search filters apply to searches only; lists and item IDs are returned as they are.

Example input:

{
  "searchQueries": ["rust", "\"vector database\""],
  "searchType": "story",
  "dateFrom": "2024-01-01",
  "minPoints": 100,
  "sortBy": "date",
  "maxItemsPerSearch": 500,
  "lists": ["top"],
  "itemIds": ["8863"],
  "includeComments": true,
  "maxCommentsPerStory": 50
}

Output

One row per item. A story and one of its comments (values illustrative):

{
  "id": 8863,
  "type": "story",
  "title": "My YC app: Dropbox - Throw away your USB drive",
  "url": "http://www.getdropbox.com/u/2/screencast.html",
  "domain": "getdropbox.com",
  "text": null,
  "author": "dhouston",
  "points": 111,
  "numComments": 71,
  "createdAt": "2007-04-04T19:16:40Z",
  "hnUrl": "https://news.ycombinator.com/item?id=8863",
  "parentId": null,
  "storyId": 8863,
  "storyTitle": null,
  "depth": 0,
  "tags": ["story"],
  "rank": null,
  "source": "item",
  "input": "8863",
  "status": "ok",
  "error": null,
  "scrapedAt": "2026-11-04T18:00:00+00:00"
}

{
  "id": 9224,
  "type": "comment",
  "title": null,
  "url": null,
  "domain": null,
  "text": "Plain text of the comment.\n\nParagraphs are kept, links are written out in full.",
  "author": "someuser",
  "points": null,
  "numComments": null,
  "createdAt": "2007-04-05T02:11:40Z",
  "hnUrl": "https://news.ycombinator.com/item?id=9224",
  "parentId": 8863,
  "storyId": 8863,
  "storyTitle": "My YC app: Dropbox - Throw away your USB drive",
  "depth": 1,
  "tags": ["comment"],
  "rank": null,
  "source": "comments",
  "input": "8863",
  "status": "ok",
  "error": null,
  "scrapedAt": "2026-11-04T18:00:00+00:00"
}

• type: story, comment, job, poll or pollopt. Ask HN and Show HN posts are stories with an ask_hn or show_hn tag.
• text: the story text (Ask HN and similar) or the comment, as plain text. HTML tags are removed, entities decoded, paragraphs kept as blank lines, code blocks kept, and links Hacker News shortens with ... written out in full.
• parentId / storyId / depth: for comments, the item replied to, the story the thread belongs to, and how deep the reply is (1 = a direct reply to the story). Comments found by a search have no depth, because search results don't include it.
• rank: the position in the search results or list.
• source / input: search, list, item or comments, and the query, list name or ID that produced the row.
• status: ok (charged), or free rows: not_found (no item has this ID), deleted, dead (flagged or killed, hidden on the site) and invalid_input. error says why.
• points and numComments are as of the moment the API was read. Comments have no public points.

Each item is saved once per run, even when two searches, a list and an ID all find it. The run also saves a SUMMARY record in the key-value store: rows by status and type, total matches per search, items charged, requests made, and whether the spending cap stopped the run.

Pricing

Pay per event, no start fee: $0.0005 per item saved ($0.50 per 1,000). Every story, comment, job or poll row with status ok is one item, so a story with 40 comments is 41 items. Rows for missing, deleted or dead items and unusable inputs are free.

Set a maximum charge per run in the run options and the Actor stops cleanly when it's reached, before fetching anything it couldn't charge for.

Good to know

• The 1,000-result cap. Algolia returns at most 1,000 results for one query. When you ask for more and more exist, the Actor splits the search into date windows for you:
  • Newest first: after the newest 1,000, it asks again for items older than the oldest one seen, and so on, so results stay in exact date order with no repeats and no cap.
  • Relevance: the date range is split into windows of at most 1,000 matches each, newest window first, and results are in relevance order within each window, not one ranking across all of HN. For one global ranking, keep Max results per search at 1,000 or below. The log and SUMMARY say how many windows were used. Algolia's match counts are its own estimates.
• Speed and politeness. Algolia's documented limit is 10,000 requests per hour per IP. The Actor sends at most 2.5 Algolia requests per second (9,000 an hour, under that limit), and each returns up to 100 results, so 1,000 results take about 10 requests. The official API has no published limit; the Actor keeps at most 10 requests to it in flight. Comment threads come from Algolia in one request per story. Failed requests are retried with backoff (1, 2, 4, 8 seconds) and the API's own Retry-After is honoured.
• Very new stories. Algolia indexes new items within minutes. If a story's thread isn't there yet, or a thread is too large to read in one answer, comments are read from the official API instead, level by level, up to your comment limit.
• Lists are live. The top, new and best lists change by the minute; two runs a few minutes apart will differ.
• Only the public APIs are contacted. Story links are returned as data and never opened, and item links must be news.ycombinator.com links.
• Usernames are the public pseudonyms Hacker News shows. The Actor returns only what the APIs return publicly and doesn't look up user profiles.

Source and terms

Data comes from Hacker News (news.ycombinator.com, run by Y Combinator) through its official public API, and from the Algolia HN Search API, a public search service Algolia runs for Hacker News. Posts and comments belong to their authors. Check Y Combinator's terms before republishing them, and credit Hacker News when you do.

This Actor isn't affiliated with or endorsed by Y Combinator, Hacker News or Algolia.

More tools from the same developer

• News Scraper: GDELT Articles, Full Text & 65 Languages
• RSS Feed Reader: Atom & RSS to JSON with Feed Finder
• Google Trends Scraper: Interest Over Time, Regions & Queries
• OpenAlex Papers Search: Scholarly Works, Citations & Abstracts
• ClinicalTrials.gov Search: Studies, Phases, Sponsors & Status
• Federal Contracts Search: SAM.gov Opportunities & Award History