Hacker News Scraper

Search posts and comments by keyword, or pull the live front page with each post's rank. One clean row per post, comment or user. No login and no API key.

What it does

• Keyword search across posts and comments, with filters for minimum points, minimum comments and a date range.
• Live rankings - front page, best, newest, Ask, Show and jobs. Every row carries its position, so a scheduled run builds a ranking history.
• Full comment threads - each comment is its own row with its reply depth, parent and the post it belongs to.
• User profiles - karma, account age, bio and submission count.
• Domain extracted for every linked post, so you can group by the site being discussed.
• No result cap on large runs - sort by date and the run walks backwards through time for as long as you want.

Common uses: tracking what gets discussed about a product or topic, developer market research, monitoring your own launches, building a training or analysis dataset, and finding the sites that get traction.

Input

All four modes are optional and can be combined in one run.

Field	Type	Description
searchTerms	array	Keywords to search. One search per entry.
postType	string	any, story, comment, ask, show, poll, front_page.
sort	string	relevance or date. See the note below.
maxResultsPerTerm	integer	Cap per term. Default 100, up to 5000.
minPoints	integer	Only posts with at least this many points.
minComments	integer	Only posts with at least this many comments.
dateFrom / dateTo	string	Date range, YYYY-MM-DD.
lists	array	Live rankings: top, best, new, ask, show, job.
maxItemsPerList	integer	Cap per list. Default 100, up to 500.
usernames	array	User profiles to fetch.
itemIds	array	Specific posts, by link or numeric id.
includeComments	boolean	Also pull comment threads for every post collected.
maxCommentsPerPost	integer	Cap on comments per post. Default 100.
proxyConfiguration	object	Optional. Enable to spread requests across IPs.

A note on sorting. Relevance ranks by popularity but stops at 1000 results per term. Sorting by date walks backwards through time and has no limit. Ask for more than 1000 and the run switches to date automatically, so a large request always returns the full amount.

Example input

{
  "searchTerms": ["kubernetes"],
  "sort": "date",
  "maxResultsPerTerm": 2500,
  "minPoints": 10,
  "dateFrom": "2026-01-01",
  "lists": ["top"],
  "includeComments": true,
  "maxCommentsPerPost": 50
}

Output

One row per post, comment or user. Every row starts with id and type so a mixed run stays easy to split.

A post:

{
  "id": 100000001,
  "type": "story",
  "title": "An example post title",
  "text": null,
  "url": "https://example.com/an-article",
  "domain": "example.com",
  "author": "example_user",
  "points": 89,
  "numComments": 10,
  "createdAt": "2026-07-14T19:25:10Z",
  "rank": 1,
  "list": "top",
  "commentsUrl": "https://news.ycombinator.com/item?id=100000001",
  "authorUrl": "https://news.ycombinator.com/user?id=example_user"
}

A comment:

{
  "id": 100000002,
  "type": "comment",
  "text": "The comment body as plain text.",
  "author": "example_user",
  "points": null,
  "createdAt": "2026-07-14T20:02:44Z",
  "depth": 2,
  "parentId": 100000001,
  "storyId": 100000001,
  "storyTitle": "An example post title",
  "commentsUrl": "https://news.ycombinator.com/item?id=100000002",
  "authorUrl": "https://news.ycombinator.com/user?id=example_user"
}

A user:

{
  "id": "example_user",
  "type": "user",
  "username": "example_user",
  "karma": 12345,
  "about": "The profile bio as plain text.",
  "submissionCount": 420,
  "createdAt": "2010-02-11T18:21:32Z",
  "profileUrl": "https://news.ycombinator.com/user?id=example_user"
}

rank and list are filled only for rows that came from a live list. Ask posts have no link, so url and domain are null on those.

Notes

• No login and no API key. Set your input and run.
• Comment bodies are converted from markup to plain text, with paragraph breaks kept.
• Everything is deduplicated across every mode in a run, so a post that appears in two lists is delivered once and charged once.
• Schedule a run on lists to record how posts rise and fall over the day.

To improve our actors we collect anonymized usage telemetry (run stats and input patterns). No personal account data is collected.