Hacker News Scraper: Stories, Comments, Search, Users (API)

Hacker News Scraper gets stories, comments, search results and user profiles from Hacker News (news.ycombinator.com) through its official public APIs. No HTML scraping, no browser and no proxy. Pick a mode:

• Lists: the top (front page), new, best, Ask HN, Show HN and job lists, with each story's rank.
• Search: Algolia's HN search by keyword, tags (story, comment, show_hn, ask_hn, author_…), date range, minimum points and minimum comments, newest first or by relevance. Date-sorted searches can go past Algolia's 1,000-result cap.
• Items: any story, comment, job or poll by ID or news.ycombinator.com/item?id=… URL.
• Users: profile (created date, karma, about) and, optionally, their latest stories or comments.

Turn on Include comments to get each story's whole comment thread, either nested inside the story row or flat (one row per comment with storyId, parentId and depth). Turn on Only new items and each scheduled run returns only stories you haven't had before.

Why this Actor

• Official APIs only. Lists, items and users come from HN's Firebase API; search and comment trees come from the Algolia HN Search API. Both are public and need no key. Nothing breaks when the HN website's layout changes.
• Whole threads in one request. A story's full comment tree comes from one Algolia request instead of one request per comment. Top-level comments are put in HN's own ranking order (from the story's Firebase kids list) when the story came from a list or an item ID.
• Clean text. HN returns story text, comments and profiles as HTML. You get plain text (links written out in full, code blocks kept), Markdown, or the original HTML.
• Monitoring built in. Only new items remembers the last 50,000 story IDs per task and skips them on later runs. Pair it with an email or Slack step to get new Show HN posts, or every new mention of your product.
• Polite. At most 10 parallel requests to Firebase (adjustable) and 2 to Algolia, with Algolia requests spaced 0.4 s apart to stay within its public limit of 10,000 requests per hour. Busy (429) and server errors (5xx) are retried with backoff, honouring Retry-After.
• Honest output. Deleted and dead (flagged) items are left out by default, or included with deleted / dead flags. An item that can't be fetched gets a free row with the reason in error, and the run carries on.

Use cases

• Front-page tracking: snapshot the top 30 every hour with ranks and points, to chart how stories rise and fall.
• Brand and competitor monitoring: search for your product or competitors' names, sort by date, with Only new items, and send each new mention to Slack.
• Show HN / Launch HN research: every Show HN post with 100+ points in a date range, with its comments.
• Discussion analysis and datasets: full comment threads for sentiment analysis, summarising, or training data, in a flat table.
• Hiring: the latest job posts from the job list, or "Who is hiring?" comments with tags: story_ID.
• User research: karma, account age and latest submissions for a list of usernames.

How to use it

1. Choose a Mode and fill in its section: lists, a search, item IDs/URLs, or usernames.
2. Optional: Include comments (nested or flat), choose a Text format, and turn on Only new items for scheduled runs.
3. Run it, or schedule it. Export as CSV, Excel or JSON, or chain it with an integration (see Related Actors).

Input examples

The front page, top 30 stories:

{ "mode": "lists", "lists": ["top"], "maxItems": 30 }

Show HN posts from the last 7 days with at least 50 points, with up to 300 comments each, one row per comment:

{
  "mode": "search",
  "tags": ["show_hn"],
  "dateFrom": "7 days",
  "minPoints": 50,
  "sort": "date",
  "maxItems": 500,
  "includeComments": true,
  "commentsFormat": "flat",
  "maxCommentsPerStory": 300
}

Every new comment that mentions your product, for an hourly schedule:

{ "mode": "search", "query": "acme db", "tags": ["comment"], "sort": "date", "maxItems": 200, "onlyNew": true }

Output example (a story row; illustrative values)

{
  "id": 41234567,
  "type": "story",
  "title": "Show HN: A tiny SQLite-backed job queue",
  "url": "https://github.com/example/tinyqueue",
  "domain": "github.com",
  "text": null,
  "author": "example_user",
  "points": 182,
  "commentCount": 64,
  "createdAt": "2026-10-20T14:03:11Z",
  "hnUrl": "https://news.ycombinator.com/item?id=41234567",
  "rank": 4,
  "list": "top",
  "storyId": null,
  "parentId": null,
  "depth": null,
  "storyTitle": null,
  "tags": ["story", "show_hn"],
  "karma": null,
  "submissionCount": null,
  "deleted": false,
  "dead": false,
  "error": null
}

With Include comments, story rows also get commentsIncluded (how many comments were output) and commentsTruncated (true when Max comments per story cut the thread). In nested mode they get a comments array:

"comments": [
  {
    "id": 41234601, "author": "another_user", "text": "How does it handle crashes mid-job?",
    "createdAt": "2026-10-20T14:20:45Z", "hnUrl": "https://news.ycombinator.com/item?id=41234601",
    "parentId": 41234567, "depth": 0, "deleted": false,
    "replies": [ { "id": 41234688, "author": "example_user", "text": "Jobs are leased with a timeout…", "depth": 1, "parentId": 41234601, "replies": [] } ]
  }
]

In flat mode each comment is its own row after its story, with type: "comment", storyId, storyTitle, parentId and depth (0 = a direct reply to the story).

Other columns: listRanks (when you pick several lists: the story's rank in each, e.g. {"top": 3, "best": 12}), pollOptions (polls: [{id, text, points}]), and for users karma, submissionCount and text (their About section). A run summary is saved as OUTPUT in the key-value store.

How much does it cost?

Pay per event: $0.0002 per output item, plus $0.00005 per run start. Apify platform usage is included. An output item is a story, job or poll row, a user row, or a comment (a flat comment row, or each comment inside a nested story row: a story row with 40 nested comments counts as 41 items, the same as in flat mode).

These are free: error rows (items or users that couldn't be fetched or don't exist, invalid input), deleted-comment placeholders, and everything left out: deleted or dead items, duplicates, and items already seen by Only new items.

Apify Store discounts: the price above is for Apify's Free plan. Paid plans pay less per 1,000 items: $0.18 on Starter, $0.16 on Scale, $0.14 on Business. The "from" price at the top of the Store page is the Business price.

Worked examples:

Examples use the Free-plan rate. The run-start charge applies to every run, counted per GB of Actor memory (minimum one; one at the default 256 MB).

• The top 30 stories: 30 × $0.0002 + $0.00005 = $0.00605.
• 30 stories with their comments, 6,000 comments in total (with Max comments per story at 200 or more): 6,030 × $0.0002 + $0.00005 = $1.20605.
• A daily Only new items search (for example, all Show HN posts) that finds 150 new stories a day: 150 × $0.0002 + $0.00005 = $0.03005 per day, about $0.90 per month.
• An hourly front-page snapshot of 30 stories: 24 × $0.00605 = $0.1452 per day, about $4.36 per month.

If you set a maximum cost per run, the Actor stops cleanly at that limit. A nested story row that doesn't fully fit is output with as many of its comments as the budget allows (commentsTruncated: true), and the run stops there.

FAQ and limits

Is this allowed? Do I need an API key?

No key is needed. The Firebase API is HN's official public API, and the Algolia HN Search API is the public search service HN links to from its own search box. This Actor reads only those two hosts. It isn't affiliated with Y Combinator or Algolia.

Why do comments lag behind for very new stories?

Comment trees come from Algolia's index, which usually trails the live site by a few minutes. A story with comments that aren't indexed yet is still output, with a note in error. Run again later for its comments. commentCount is HN's own count; commentsIncluded can be lower (deleted comments, index lag, or your Max comments per story).

In what order are comments?

Depth-first: each comment is followed by its replies. Top-level comments follow HN's ranking when the story came from a list or an item ID. For search results, and for replies below the top level, the order is the one Algolia returns.

Comments don't have points?

HN's APIs don't publish comment scores, so points is null for comments.

How do I get more than 1,000 search results?

Algolia returns at most 1,000 results per query. With Sort: Date the Actor continues automatically with older time windows until it has Max stories results or none are left, without duplicates. Sort: Relevance can't be split this way and stops at 1,000; narrow the query or the dates instead.

Can it monitor every new story?

The new list holds only the latest 500 stories, so a monitor that runs rarely can miss some. For a complete feed, use Search mode with tags: ["story"], Sort: Date, Posted after (for example 2 days) and Only new items.

Deleted and dead items

By default, deleted and dead stories are left out (free). A deleted comment is dropped unless it has replies; then it stays as a free placeholder (deleted: true, no author or text) so the thread stays connected. Turn on Include deleted and dead items to output them all. Search results come from Algolia's index, which has no dead or deleted flags.

Rate limits and speed

Firebase requests run in parallel (10 by default). Algolia requests run at most 2 at a time and at most 2.5 per second, so 500 stories with comments take about 3–4 minutes. Lists, items and users without comments are fast.

Usernames

HN usernames are case-sensitive. A user who doesn't exist gets a free row with an error. A user's latest submissions come from Algolia's search index.

Related Actors

Actor	Use it to
Send Email Report & Slack, Discord, Teams, Telegram Alerts	Get new Show HN posts, or new mentions of your product, in Slack or by email after each run.
Article Extractor: Text, Markdown, Author, Date	Extract the linked articles: full text, author and date for each story's url.
RSS & Atom Feed Reader	Watch blogs and news feeds alongside Hacker News, with only new items.
Dataset Filter & Transform	Keep only stories above a points threshold, or rename and drop columns before export.
Google Sheets Sync	Append stories or comments to a Google Sheet.
HTTP Request & API Calls	Send the results on to your own API or webhook, or call other JSON APIs in bulk.

Data: Hacker News API (Firebase) and HN Search API by Algolia.