Hacker News Scraper: stories by keyword, points, comments

Hacker News scraper for stories by keyword, up to 100 queries per run: one row per story with rank, title, link, domain, points, comment count, date and type (story, Ask HN, Show HN, job), sorted by relevance or date, with points and recency filters. Pay per story.

Why this scraper

• Only delivered results are charged. Inputs that fail come back as clear error records at no cost.
• The official Hacker News search API (Algolia), not a scraped page: one request per 100 stories, no browser, no blocking. Measured: 40 stories on "web scraping" plus the 30 front-page stories in one run for a few hundredths of a cent.
• Ready to filter and rank: minPoints and sinceDays are applied by the API, sortBy switches between relevance and newest, type separates stories from Ask HN, Show HN and jobs, domain is extracted from the link, and Ask HN text comes as plain text instead of HTML. The submitter's username is not delivered.

Who this is for

Type queries in queries (up to 100 per run), or front_page for the current front page; choose sortBy, minPoints, sinceDays and maxResultsPerQuery (default 50, ceiling 500). Every row carries the query, rank, story id, title, link, domain, points, comment count, posting time, type, the Ask HN text when there is one, and the Hacker News link.

Who this is not for

Comments are not delivered, only their count. The submitter's username is left out on purpose. url and domain are empty for Ask HN and other text-only posts, text is filled only for those, and points is empty for job postings, which Hacker News does not score. Relevance is Algolia's ranking (text match weighted by points and recency); it is not the front-page order, which front_page gives.

Input fields

Field	Type	Required or default	What it does
queries	list of text	required	One per row, up to 100. Use 'front_page' for the current front page.
sortBy	text (relevance, date)	relevance	Relevance (points and recency weighted) or newest first.
minPoints	number	0	Skip stories below this score.
sinceDays	number	0	0 means all time.
maxResultsPerQuery	number	50	Cost ceiling per query.

Input example

{
    "queries": [
        "web scraping",
        "front_page"
    ],
    "sortBy": "relevance",
    "minPoints": 0,
    "sinceDays": 0,
    "maxResultsPerQuery": 50
}

Output example

Field	Type	What it holds	Example
query	text	The search term this row was found with, so rows from several inputs can be told apart.	web scraping
rank	number	The place in the source's own order, counting from 1.	1
storyId	text	The id of the story, the key to its comments.	22180559
title	text	The title as the source shows it.	Congrats! Web scraping is legal! (US precedent)
url	text	The article the story points to; empty on a text post.	https://parsers.me/us-court-fully-legalized-website-scrap...
domain	text	The domain that article stands on, the quickest way to see who is being read.	parsers.me
points	number	How many points the story has, the measure of its reach.	1057
commentCount	number	How many comments the story drew.	388
createdAt	text	When the item was posted, in UTC.	2020-01-29T14:06:37Z
type	text	What the item is: a story, an Ask HN, a Show HN or a job post.	story
text	text	The text of the post itself, filled on an Ask HN and empty on a link post.	I'm plowing through the Dungeon Crawler Carl series, whic...
hnUrl	text	The item's own page on Hacker News.	https://news.ycombinator.com/item?id=22180559

Error codes: INVALID_QUERY, NO_RESULTS, BLOCKED.

One delivered row looks like this:

{
  "query": "web scraping",
  "rank": 1,
  "storyId": "22180559",
  "title": "Congrats! Web scraping is legal! (US precedent)",
  "url": "https://parsers.me/us-court-fully-legalized-website-scraping-and-technically-prohibited-it/",
  "domain": "parsers.me",
  "points": 1057,
  "commentCount": 388,
  "createdAt": "2020-01-29T14:06:37Z",
  "type": "story",
  "text": null,
  "hnUrl": "https://news.ycombinator.com/item?id=22180559",
  "status": "ok"
}

Related actors from steadydata

• hacker-news-comments: the full discussion under the stories you found
• rss-feed-reader: the feeds of the sites the stories link to
• google-news: the same topic in mainstream news
• website-tech-stack: the technology behind a linked site

Pricing

Pay per event: one story-listed event per delivered result. No charge for inputs that fail, no separate platform-usage surcharge.

Free Apify plan: this actor delivers up to 25 rows per run for accounts on the Apify free plan, and then stops with a message. That limit is set by us, not by Apify. It exists so the actor keeps paying for itself for the people who do pay. Any paid Apify plan runs it at full size, billed per delivered row, with failed rows never charged.

Reviews: if this actor saves you time, a short review on this page is the one thing that helps most. Ratings are what other buyers look at first, and we have no other way to ask.

FAQ

Can I get the newest stories about a topic? Yes: sortBy = date, and sinceDays = 7 for the last week.

How do I get only popular stories? Set minPoints, for example 100; the filter is applied by the API, so the rows you get are all above it.

Do I pay for queries without results? No. A query with no stories comes back as one free error row.

Can I read the linked articles? Not with this actor; the url column is the input for a page or feed reader. The RSS and Atom Feed Reader from the related actors below reads the site's feed.

Is the front page live? Yes, front_page returns the 30 stories on the front page at run time, in front-page order.

Is personal data collected? createdAt is the posting time in UTC as an ISO timestamp; hnUrl is the story's discussion page.

What happens when the source changes? Sources change from time to time; that is the nature of this work. The actor is monitored daily and fixed fast, and while it is broken you are not charged, because only delivered results cost anything.