Hacker News Scraper - top, new and search stories, comments avatar

Hacker News Scraper - top, new and search stories, comments

Pricing

from $5.00 / 1,000 records

Go to Apify Store
Hacker News Scraper - top, new and search stories, comments

Hacker News Scraper - top, new and search stories, comments

Hacker News stories from the official public APIs: the top, new and best lists, or a full-text search by keyword, tag and date. One row per story with id, title, URL, domain, points, author, comment count, created time and HN link; optionally top comments. For trend trackers and researchers.

Pricing

from $5.00 / 1,000 records

Rating

0.0

(0)

Developer

RetrainMap Team

RetrainMap Team

Maintained by Community

Actor stats

0

Bookmarked

2

Total users

1

Monthly active users

6 hours ago

Last modified

Categories

Share

Hacker News stories + comments — Algolia + Firebase APIs

One row per Hacker News story from the official public APIs: the top / new / best lists (Firebase API) or a full-text search (Algolia API), with id, title, URL, domain, points, author, comment count, created time, HN link and the text of Ask/Show posts. Optionally the top N comments per story (HN's own ranked order) as extra rows. Nothing is scraped from news.ycombinator.com — only the two APIs are read.

The default input (top stories, 100 rows) succeeds within 5 minutes and returns rows — Apify auto-tests it daily.

Data source and status

PublisherY Combinator (Hacker News). Search index operated by Algolia
APIshttps://hacker-news.firebaseio.com/v0 — public, no key (documentation: https://github.com/HackerNews/API); https://hn.algolia.com/api/v1 — public, no key (documentation: https://hn.algolia.com/api)
What this Actor readsFirebase: /v0/topstories.json, /v0/newstories.json, /v0/beststories.json and /v0/item/{id}.json. Algolia: /search, /search_by_date (query, tags, numericFilters on created_at_i, 100 hits per page) and /items/{id} for comment trees. At most one request per request_interval_ms (default 100 ms)
Refresh cadenceBoth APIs are live — the Firebase lists change within seconds of the site; the Algolia index is updated continuously. The Actor reads live on every run; fetched_at on each row is the fetch time
Terms / attributionThe Firebase API is published by Y Combinator under the MIT licence (https://github.com/HackerNews/API/blob/master/LICENSE); the Algolia HN Search API is free for use with attribution ("Search by Algolia", per https://hn.algolia.com/api). User-submitted content belongs to its authors; please credit Hacker News and Algolia as the source and respect Algolia's limit of 10,000 requests per hour per IP
Known caveatsFirebase lists hold 500 (top, new) or 200 (best) ids; Algolia serves at most 1,000 hits per query — narrow the date range for more; url/domain are empty for Ask HN, jobs and polls; points/num_comments are null for comments; deleted or dead items are skipped and counted in the run summary; text is the item HTML converted to plain text and cut at 3,000 characters (text_truncated says so)

Honesty note: the rows are the APIs' own fields renamed and flattened — nothing is inferred, summarised or classified by this Actor, and a run stops with an error rather than guess when an API's shape changes.

Identification: the requests carry a product-token User-Agent and the operator's contact address in the standard From: header (RFC 9110 §10.1.2).

Input

FieldTypeMeaning
modetop / new / best / searchLists from Firebase (HN's order) or an Algolia search. Default top
querystringSearch mode: full-text query (title, URL, author, text). Empty = no term
tagsarraySearch mode: story, comment, ask_hn, show_hn, front_page; several are ORed. Empty = story
search_sortrelevance / dateSearch mode: /search or /search_by_date (newest first)
date_from / date_toYYYY-MM-DDSearch mode, UTC, inclusive
include_commentsbooleanAdd up to comments_per_story top-level comment rows after each story (ranked order). Default false
comments_per_storyintegerDefault 10, at most 100
max_recordsintegerTotal rows (stories + comments). Default 200 (prefilled 100), at most 5,000
request_interval_msintegerDefault 100 (floor 50)
contact_emailstringSent in the From: header

Example — stories about Python since August 2026 with their top 5 comments:

{ "mode": "search", "query": "python", "tags": ["story"], "date_from": "2026-08-01", "include_comments": true, "comments_per_story": 5, "max_records": 30 }

Output (dataset row)

rank (position in the list / search results, or among a story's comments), type (story / comment / job / poll / pollopt), id, title, url, domain, points, author, num_comments, created_at (UTC), hn_url, text (Ask/Show/comment text, plain, ≤ 3,000 chars), text_truncated, story_id, parent_id, story_title (comments only), hn_tags, list (top / new / best / search / comments), fetched_at, source.

A run summary (filters, list size or Algolia hit count, counts of stories, comments and skipped items, requests, whether the pay-per-event budget stopped the run) is stored as RUN_SUMMARY in the run's key-value store.

Pricing (pay per event)

EventPrice
run-start — once per run$0.10
record — per row written (story or comment)$0.005

The default top-100 pull costs $0.60; 1,000 rows cost $5.10. Rows stop when your run's maximum charge is reached; the run summary says so.

Operator

Steelyard Ventures LLC (RetrainMap) — info@steelyardclinical.com. Not affiliated with Y Combinator, Hacker News or Algolia. The Actor writes only to its own dataset and key-value store; it stores no credentials and sends nothing else.