Hacker News Scraper - top, new and search stories, comments
Pricing
from $5.00 / 1,000 records
Hacker News Scraper - top, new and search stories, comments
Hacker News stories from the official public APIs: the top, new and best lists, or a full-text search by keyword, tag and date. One row per story with id, title, URL, domain, points, author, comment count, created time and HN link; optionally top comments. For trend trackers and researchers.
Pricing
from $5.00 / 1,000 records
Rating
0.0
(0)
Developer
RetrainMap Team
Maintained by CommunityActor stats
0
Bookmarked
2
Total users
1
Monthly active users
6 hours ago
Last modified
Categories
Share
Hacker News stories + comments — Algolia + Firebase APIs
One row per Hacker News story from the official public APIs: the top / new / best lists (Firebase API) or a full-text search (Algolia API), with id, title, URL, domain, points, author, comment count, created time, HN link and the text of Ask/Show posts. Optionally the top N comments per story (HN's own ranked order) as extra rows. Nothing is scraped from news.ycombinator.com — only the two APIs are read.
The default input (top stories, 100 rows) succeeds within 5 minutes and returns rows — Apify auto-tests it daily.
Data source and status
| Publisher | Y Combinator (Hacker News). Search index operated by Algolia |
| APIs | https://hacker-news.firebaseio.com/v0 — public, no key (documentation: https://github.com/HackerNews/API); https://hn.algolia.com/api/v1 — public, no key (documentation: https://hn.algolia.com/api) |
| What this Actor reads | Firebase: /v0/topstories.json, /v0/newstories.json, /v0/beststories.json and /v0/item/{id}.json. Algolia: /search, /search_by_date (query, tags, numericFilters on created_at_i, 100 hits per page) and /items/{id} for comment trees. At most one request per request_interval_ms (default 100 ms) |
| Refresh cadence | Both APIs are live — the Firebase lists change within seconds of the site; the Algolia index is updated continuously. The Actor reads live on every run; fetched_at on each row is the fetch time |
| Terms / attribution | The Firebase API is published by Y Combinator under the MIT licence (https://github.com/HackerNews/API/blob/master/LICENSE); the Algolia HN Search API is free for use with attribution ("Search by Algolia", per https://hn.algolia.com/api). User-submitted content belongs to its authors; please credit Hacker News and Algolia as the source and respect Algolia's limit of 10,000 requests per hour per IP |
| Known caveats | Firebase lists hold 500 (top, new) or 200 (best) ids; Algolia serves at most 1,000 hits per query — narrow the date range for more; url/domain are empty for Ask HN, jobs and polls; points/num_comments are null for comments; deleted or dead items are skipped and counted in the run summary; text is the item HTML converted to plain text and cut at 3,000 characters (text_truncated says so) |
Honesty note: the rows are the APIs' own fields renamed and flattened — nothing is inferred, summarised or classified by this Actor, and a run stops with an error rather than guess when an API's shape changes.
Identification: the requests carry a product-token User-Agent and the operator's contact
address in the standard From: header (RFC 9110 §10.1.2).
Input
| Field | Type | Meaning |
|---|---|---|
mode | top / new / best / search | Lists from Firebase (HN's order) or an Algolia search. Default top |
query | string | Search mode: full-text query (title, URL, author, text). Empty = no term |
tags | array | Search mode: story, comment, ask_hn, show_hn, front_page; several are ORed. Empty = story |
search_sort | relevance / date | Search mode: /search or /search_by_date (newest first) |
date_from / date_to | YYYY-MM-DD | Search mode, UTC, inclusive |
include_comments | boolean | Add up to comments_per_story top-level comment rows after each story (ranked order). Default false |
comments_per_story | integer | Default 10, at most 100 |
max_records | integer | Total rows (stories + comments). Default 200 (prefilled 100), at most 5,000 |
request_interval_ms | integer | Default 100 (floor 50) |
contact_email | string | Sent in the From: header |
Example — stories about Python since August 2026 with their top 5 comments:
{ "mode": "search", "query": "python", "tags": ["story"], "date_from": "2026-08-01", "include_comments": true, "comments_per_story": 5, "max_records": 30 }
Output (dataset row)
rank (position in the list / search results, or among a story's comments), type (story /
comment / job / poll / pollopt), id, title, url, domain, points, author,
num_comments, created_at (UTC), hn_url, text (Ask/Show/comment text, plain, ≤ 3,000
chars), text_truncated, story_id, parent_id, story_title (comments only), hn_tags,
list (top / new / best / search / comments), fetched_at, source.
A run summary (filters, list size or Algolia hit count, counts of stories, comments and
skipped items, requests, whether the pay-per-event budget stopped the run) is stored as
RUN_SUMMARY in the run's key-value store.
Pricing (pay per event)
| Event | Price |
|---|---|
run-start — once per run | $0.10 |
record — per row written (story or comment) | $0.005 |
The default top-100 pull costs $0.60; 1,000 rows cost $5.10. Rows stop when your run's maximum charge is reached; the run summary says so.
Operator
Steelyard Ventures LLC (RetrainMap) — info@steelyardclinical.com. Not affiliated with Y Combinator, Hacker News or Algolia. The Actor writes only to its own dataset and key-value store; it stores no credentials and sends nothing else.