Hacker News Scraper

Stories, comments, users and the live ranked lists from Hacker News, straight from the two public APIs — hn.algolia.com (full-text search over every item ever posted) and hacker-news.firebaseio.com (the canonical item store and the live front page). No login, no API key, no browser.

What it collects

Mode	What you get
search	Full-text search over ~46M items. Filter by kind (story, comment, poll, job, Ask HN, Show HN, Launch HN, front page), by author, by date range, by minimum points or comments.
items	The entire comment thread for an item ID — pre-joined by upstream, so a 72-node thread costs one request, not 72.
users	Account profiles: karma, account age, submission count, and optionally the newest N submissions hydrated into full rows.
lists	The live ranked lists — front page, newest, best, Ask HN, Show HN and jobs — with list position preserved.

The thing that makes this different: it gets past 1,000 results

Hacker News caps every search at 1,000 results. nbPages is always ceil(1000 / hitsPerPage), never the real page count, and the moment you page past the ceiling the API returns HTTP 200 with hits: [] and nbHits: 0 — so a scraper that reads its total off the last page it fetched reports "0 results" for a query with 543,213 matches.

Sorted by date, this actor time-slices the query automatically on created_at_i, because the ceiling is per time window, not per query. Measured: 8,000 stories in 8 requests, zero duplicates, where a naive client stops at 1,000.

Sorted by relevance there is no workaround — slicing relevance would rank within windows instead of overall — so the run stops at 1,000 and the summary says depthCapped: true rather than pretending.

Upstream's match count is a guess, and this actor says so

nbHits is an Algolia estimate unless exhaustiveNbHits is true, and the two search endpoints disagree about it by up to 111× on the same filter:

Filter	/search	/search_by_date	Ratio
tags=comment	316,444	35,112,651	0.01×
tags=story	44,067,014	3,978,936	11.08×
tags=ask_hn	2,247,258	180,703	12.44×
(no tags)	45,837,157	45,837,157	1.00× ✅ exact

So the total travels with upstreamTotalIsExact, and when it is not exact the actor spends one extra request on the other endpoint and publishes that number too, plus the ratio. When the total is exact it is trustworthy: verified against a real walk over a fixed 24-hour window — upstream said 742, the other endpoint said 742, and 742 unique stories came back.

reachableFraction is only computed against an exact total. A fraction of a number that can be wrong by two orders of magnitude is not a measurement.

Other things it gets right

• Four ways to say "does not exist", only one of them a 404. Algolia answers an unknown item with 404 but an unknown username with HTTP 500; Firebase answers both with HTTP 200 and a body of null. A generic retry ladder burns four attempts and 14s of backoff on that 500. All four become a clean ERROR row naming the cause.
• Deleted and dead items. 8.3% of recent Firebase items are one or the other (deleted items are a bare tombstone with no author or text; dead ones read [flagged]). Algolia indexes neither, so the two hosts genuinely disagree about what exists — rows carry isDeleted / isDead and the summary counts them, so a gap is explained rather than mysterious.
• Job posts have no real score. Firebase returns score: 1 on every job post (20/20 sampled) and Algolia usually omits points entirely. The number is passed through with pointsAreMeaningful: false beside it, so job ads don't sort as though they'd been downvoted to 1.
• There is no field set common to all five item types. A job post has no points and no story_id; a poll has no url; an Ask HN story has url: null with the body in story_text. Item text arrives under four different keys — normalised into one text field, with textField naming where it came from so a null reads as "this type has no body" rather than "the body went missing".
• The time cursor is inclusive (<=), never <. Items share whole-second timestamps, and a strict < drops every tie on the boundary second without saying so. The handful of known repeats this causes are deduped and counted in duplicateItemsDropped.

Output

One dataset with recordType telling the rows apart:

• SEARCH_SUMMARY — one per query: upstream's total and whether it is exact, time windows fetched, ceiling and cap flags, the window actually covered, duplicates dropped, deleted/dead counts, and anything not found.
• ITEM — one per story, comment, poll, poll option or job.
• USER — one per account.
• ERROR — one per input that failed, so every input maps to at least one row.

Every row carries _input, _source, _scrapedAt and recordType, and upstream's own fields are passed through verbatim alongside the normalised ones.

Anti-bot and limits

No WAF of any kind. Seven TLS profiles (chrome124, chrome131, chrome136, safari17_0, edge101, firefox133, chrome99_android) all returned 200 cold with byte-identical bodies on both hosts, and so did a naked urllib.request with a python-urllib/3 User-Agent. 60 back-to-back requests returned 200 sixty times, in 28.6s, with no rate-limit headers sent at all. A proxy is optional and off by default.

Known limits:

• 1,000 results per time window (worked around for date sort, unavoidable for relevance sort).
• hitsPerPage above 1,000 is silently clamped, with no error.
• front_page is a live snapshot of roughly 30 items and the exact number moves (30 one hour, 29 the next); it reports its own count exactly.
• Firebase ranked lists are as long as HN has live items — measured 500, 500, 200, 165, 31 and 28 — not a fixed size.

Policy

hn.algolia.com serves no robots.txt (HTTP 404 — RFC 9309 §2.3.1.3 treats an unavailable robots.txt as no restriction). hacker-news.firebaseio.com serves Disallow: / with an explicit Allow: /*.json$; every path this actor touches ends in .json. Neither names an AI crawler. news.ycombinator.com itself is not scraped — it asks for Crawl-delay: 30 and everything it renders is available from the APIs without the wait.

Hacker News content is user-submitted; check HN's own terms before redistributing it.