Hacker News Scraper

Searches Hacker News and returns stories and comments as structured rows. It covers keyword search, browsing by item type (front page, Show HN, Ask HN, polls, jobs), a minimum score, and an arbitrary date range. No login and no proxy are required.

The data comes from the official Algolia-powered Hacker News Search API, the same index behind the search box on news.ycombinator.com.

Input

Field	Type	Notes
queries	array	Search terms. Each is searched separately and results are merged and de-duplicated by item ID. May be left empty when a tag alone is the search, for example front_page.
tags	array	story, comment, front_page, poll, pollopt, job, show_hn, ask_hn. Several tags are combined with AND. Every one of these was verified against the live API; an unrecognised tag is rejected up front, because the API answers it with an empty result set that looks exactly like a broken scraper.
sortBy	string	relevance (default) or date. These are two different indices, not a sort flag: relevance uses /search, date uses /search_by_date.
minPoints	integer	Minimum score. Applied as points>=N in the API's numericFilters.
startDate / endDate	string	YYYY-MM-DD, a full ISO 8601 timestamp, or a Unix timestamp. Converted into created_at_i bounds. A bare end date includes the whole of that day.
maxItems	integer	Total item budget across all queries. Default 100.
proxyConfiguration	object	Optional, and off by default. The API is open.

Output

Two shapes share the dataset, distinguished by type.

A story (type: "story" — also covers Ask HN, Show HN, polls and job posts):

{
  "type": "story",
  "object_id": "45751400",
  "title": "Uv is the best thing to happen to the Python ecosystem in a decade",
  "url": "https://emily.space/posts/251023-uv",
  "author": "todsacerdoti",
  "points": 1049,
  "num_comments": 605,
  "story_text": null,
  "created_at": "2025-10-29T18:57:29Z",
  "created_at_i": 1761764249,
  "hn_url": "https://news.ycombinator.com/item?id=45751400",
  "tags": ["story", "author_todsacerdoti", "story_45751400"]
}

A comment (type: "comment"):

{
  "type": "comment",
  "object_id": "9999987",
  "author": "pbaehr",
  "comment_text": "ABI Research | Oyster Bay, New York<p>We are looking for ...",
  "parent_id": 9996333,
  "story_id": 9996333,
  "story_title": "Ask HN: Who is hiring? (August 2015)",
  "points": null,
  "created_at": "2015-08-03T21:30:28Z",
  "created_at_i": 1438637428,
  "hn_url": "https://news.ycombinator.com/item?id=9999987",
  "tags": ["comment", "author_pbaehr", "story_9996333"]
}

Notes on the data

• Comment text contains HTML markup. Hacker News stores comment and story bodies as HTML fragments: <p> between paragraphs, <a href="..."> around links, <i> for italics, <pre><code> for code blocks. HTML entities are unescaped for you (/ becomes /, " becomes "), but the tags themselves are left in place, because stripping them would destroy the paragraph breaks. Strip them yourself if you want plain text.
• A missing field is null, never 0 or "". Hacker News does not score individual comments, so points on a comment is always null — a zero there would read as a measurement that was never made. Likewise a text post has no url, and a link post has no story_text.
• A single query reaches at most 1000 items. The search API pages through the first 1000 hits of any one query and then returns nothing, however the pages are sliced; hitsPerPage above 1000 is silently clamped. maxItems above 1000 therefore needs several narrower queries, or a date range walked in slices, to go further back. Both limits were measured on 2026-08-24.
• Items are de-duplicated by objectID before they are delivered, so the same story matching two of your queries is emitted and billed once.
• No rate limiting was observed in testing (40 requests back to back all returned HTTP 200), but the actor still treats HTTP 429, 403 and 5xx as transient and retries them with backoff. A malformed filter, which the API answers with HTTP 400, is treated as permanent and is not retried.

Billing

One event per delivered item, charged after the item is written to the dataset: post-scraped for stories and comment-scraped for comments. The event is chosen from what each item actually is, not from what was requested, so a mixed search bills each row correctly. A small actor-start event covers the fixed cost of a run that legitimately finds nothing.