Hacker News Scraper

Hacker News Scraper extracts stories, comments, and user profiles from Hacker News in four modes: search, user, listing, and threads. Built with Scrapy and the Apify SDK for Python, it runs on the Apify platform with proxy rotation, scheduling, and API access to results.

Why use this Hacker News Scraper?

• Track how a topic or company is discussed on Hacker News over time (search mode).
• Monitor a specific user's karma, bio, and submission history (user mode).
• Pull the current front page, newest posts, Ask/Show HN, or job postings (listing mode).
• Archive full comment trees for specific threads, with parent/child relationships preserved (threads mode).

How to use Hacker News Scraper

1. Click Try for free (or Start) on the Actor page.
2. Choose a Mode in the Input tab: search, user, listing, or threads.
3. Fill in the field(s) relevant to that mode (search query, username, list type, or thread item IDs).
4. Optionally set Max items and Max pages to bound the run.
5. Click Start and download results from the Dataset tab once the run finishes.

Input

Field	Applies to	Description
mode	all	search, user, listing, or threads
searchQuery / searchSort	search	Query text and relevance/date sort order (uses the HN Algolia Search API)
searchType	search	Result type filter: story (default, matches hn.algolia.com), comment, poll, job, ask_hn, show_hn, front_page, or all
username	user	HN username to scrape the profile and submissions of
listType	listing	top, new, best, ask, show, or jobs
threadUrls	threads	List of HN thread URLs, e.g. https://news.ycombinator.com/item?id=1
includeComments	threads	Whether to scrape the full comment tree
maxItems	all	Maximum items to output (0 = unlimited)
maxPages	search, listing	Maximum result pages to fetch (0 = unlimited)
proxyConfiguration	all	Apify proxy settings

See the Input tab for the full schema.

Output

Each dataset item is a story, comment, or user record. Example (listing mode):

{
    "id": "49033127",
    "type": "story",
    "title": "Flux 3 X Mimic: The Next Generation of Video-Action Models",
    "url": "https://bfl.ai/blog/flux-3-mimic",
    "domain": "bfl.ai",
    "points": 170,
    "author": "kensai",
    "created_at": "2026-07-24T09:31:48 1784885508",
    "num_comments": 21,
    "rank": 1,
    "source": "top"
}

Example (threads mode, a comment):

{
    "id": "49034857",
    "type": "comment",
    "author": "forgotusername6",
    "text": "Is anyone feeding models touch data? ...",
    "created_at": "2026-07-24T12:53:03 1784897583",
    "parent_id": "49031796",
    "story_id": "49031796",
    "depth": 0,
    "source": "thread"
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

Data fields

Field	Description
id	HN item ID, or username in user mode
type	story, job, poll, comment, or user
title / url / domain	Story title, external URL, and URL domain
text	Comment body or self-post text
points	Story score
author	Submitter/commenter username
created_at	ISO timestamp (or HN's raw date string)
num_comments	Total comment count on a story
rank	Position on a listing page
parent_id / story_id / depth	Comment tree position
karma / about / created	User profile fields

Cost of scraping Hacker News

Hacker News pages are lightweight static HTML, so runs are fast and cheap. A single listing page (30 items) or a small thread typically completes in a few seconds on the free Apify plan. Scraping large threads with thousands of comments, or many listing/search pages, will use proportionally more compute units.

Tips

• Use maxItems to cap runs when a thread or user has far more items than you need.
• listing and search modes support maxPages for pagination; set it to 0 to fetch all available pages.
• HN rate-limits aggressive crawling — enable the Apify proxy and avoid very high concurrency for large jobs.

FAQ, disclaimers, and support

This Actor is for scraping publicly available data from Hacker News. Use it in accordance with Hacker News' terms of service and applicable law; do not use it to collect personal data beyond what users have made public. HN's page structure can change, which may require selector updates. Found a bug or need a custom variant? Open an issue on the Actor's Issues tab.