Hacker News Scraper: stories, comments and keyword search

Read the Hacker News front page lists, single items and whole comment trees, and search every story and comment by keyword. Stories and comments come from the official Hacker News API. Keyword search runs on the Algolia Hacker News Search API.

What you can get in one run:

• Lists: top, new, best, Ask HN, Show HN and jobs, in ranked order, with the rank of each item
• Single items: stories, jobs, polls or comments by id or by item URL
• Comment trees: every reply under a story down to the depth you set, one record per comment with its parent, depth and reply count
• Keyword search: stories, comments, Ask HN, Show HN, polls, jobs or front page stories, sorted by relevance or by date, filtered by date range, minimum points and minimum comments
• More than 1,000 search results: sorted by date, a search keeps going past the 1,000 result limit of the search API until it reaches your maximum

Input

• Lists, and how many items to take from each (default 30, up to 500)
• Item ids or URLs, such as 8863 or https://news.ycombinator.com/item?id=8863
• Search keywords, one search per keyword. Put a phrase in double quotes to match it exactly.
• Search in: stories (default), comments, Ask HN, Show HN, polls, jobs, or stories that reached the front page
• Sort search results by relevance (default) or by date
• From date and to date, written as 2025-01-31
• Minimum points and minimum comments
• Maximum results per keyword (default 100, up to 10,000)
• Include comments (default off), comment depth (default 2) and maximum comments per item (default 100)

Output

Dataset: one record per story, job, poll or comment.

• Stories, jobs and polls: recordType, id, type, title, url, domain, text, score, commentCount, createdAt, hnUrl, source (the list, item id or keyword it came from) and rank for list items
• Comments from a comment tree: recordType, id, storyId, parentId, depth, text, replyCount, createdAt and hnUrl. Comments found by search carry storyId, parentId and storyTitle in place of depth and replyCount.
• Problems: a record with recordType set to error, the input it concerns in source, and the reason in error, for example an item id that does not exist. The rest of the run carries on.

Text arrives as plain text with paragraphs kept. Deleted and dead items are skipped in lists and comment trees; an item id that points to one returns an error record. An item that appears in two lists or two searches is returned once. Export as JSON, CSV or Excel, or read it over the API.

Pricing

Pay per event: one event per story, job, poll or comment returned. Error records are not charged. The run stops when it reaches the maximum charge you set.

Scope and conduct

Only the official Hacker News API (hacker-news.firebaseio.com) and the Algolia Hacker News Search API (hn.algolia.com) are called. The Hacker News website is never loaded. Author names, user profiles and karma are not collected. Search requests are paced at two a second, inside the Algolia limit of 10,000 requests an hour.

Support

Questions and problems: the Issues tab on this page. Replies are written by an AI assistant on behalf of the developer and reviewed by the developer.