🟠 Hacker News Scraper: Story, Comment & User Search by RecordsData

Hacker News Scraper is an Apify actor that searches Hacker News stories, comments and user profiles by keyword and returns structured rows with points, comment counts, dates and direct links. Filter by type (stories, Ask HN, Show HN, jobs, polls, front page), points, comments, author, domain and dates. It parses the monthly "Who is hiring?" threads, returns comment trees and has a monitor mode for only new results. No login, no API key. Export to CSV, Excel, JSON or XML. Story rows cost $12 per 1,000 on the free tier.

A search for "web scraping" matches 2,083 stories and 9,585 comments (measured 2026-10-04). The data comes from the public Hacker News Search API by Algolia.

🔎 What does the Hacker News Scraper do?

• Search: title, URL, domain, author, points, comment count, type, text, dates, link and pointsPerHour for stories, plus comment search and comment trees with depth, parentId and storyId.
• Monthly threads: "Who is hiring?", "Who wants to be hired?" and "Freelancer? Seeking freelancer?", one row per post with a headline and parsed hiringCompany, hiringLocation, hiringRemote, hiringApplyUrl and hiringCompanyUrl when present (company on 89% of 1,677 posts, apply link on 67%).
• Extras: opt-in GitHub fields (free), lookups by item ID or URL, user profiles, authorKarma and an only-new mode.

📊 What data does the Hacker News Scraper return?

Each row has a recordType (story, comment or user). Real output for the query "web scraping" (recorded 2026-10-04; current story rows also carry domain, ageHours and pointsPerHour):

{
  "recordType": "story",
  "hnId": "22180559",
  "title": "Congrats! Web scraping is legal! (US precedent)",
  "author": "ehurynovich",
  "points": 1057,
  "commentCount": 388,
  "storyType": "story",
  "hnUrl": "https://news.ycombinator.com/item?id=22180559"
}

Comment rows carry author, text, storyTitle, storyId and hnUrl; user rows carry username, karma, about and hnUrl. The actor does not return deleted or dead flags, so a dead item can appear as a normal row.

💰 How much does the Hacker News Scraper cost?

Pricing is pay per event, only for rows saved to your dataset.

Event	Price on the free tier
Story row (points, comments, type, links)	$0.012 ($12 per 1,000)
Comment row (comment with author and parent story)	$0.01 ($10 per 1,000)
User profile (karma, about, account age)	$0.01 ($10 per 1,000)

Comment-tree rows and monthly-thread rows are billed as comment rows. includeAuthorProfile adds a fourth event, Author karma and account age, at $0.002 per enriched row, on top of the story or comment row. GitHub and hiring fields cost nothing extra. Paid plans get lower story prices (down to $10 per 1,000 on Silver and above). You are not charged for error rows, duplicates, empty results or a failed run.

Example: 10 stories cost $0.12 on the free tier; 1,000 stories cost $12, or up to $14 with includeAuthorProfile. A 60-post whoIsHiring run costs $0.60.

🚀 How do I scrape Hacker News in 3 steps?

1. Click Try for free on the actor page.
2. Type a search query and choose a story type, sort order and filters.
3. Click Start, then download CSV, Excel, JSON or XML.

⚙️ Hacker News Scraper input

Field	Type	Default	What it does
searchQuery	string	web scraping	Full-text query across titles and story text. Empty matches everything.
storyType / sortBy	select	story / relevance	story, ask_hn, show_hn, job, poll or front_page; relevance or date (newest first).
minPoints / minComments	integer	0	Minimum upvotes; minimum comments (stories only).
author / domain	string	empty	Only items by this username; only stories linking to this domain, subdomains included (capped at about 1,000 results).
mode / hiringMonth	select / string	search / empty	search, whoIsHiring, whoWantsToBeHired or freelancer; thread month YYYY-MM (empty = latest).
itemIds / usernames	array	[]	Item IDs or URLs; usernames for profile rows.
includeCommentTree / maxDepth	boolean / integer	false / 3	Return comments as rows with depth (1 to 50 levels). Rows count toward maxItems.
includeAuthorProfile	boolean	false	Add authorKarma (and authorCreatedAt). Billed as the Author karma event.
includeGithubRepo	boolean	false	Add GitHub fields to stories that link a repository; about the first 50 repositories of a run are enriched (GitHub's anonymous limit).
dateFrom / dateTo	string	empty	YYYY-MM-DD. An invalid date fails the run before anything is charged.
includeStories / includeComments	boolean	true / false	Stories off gives only comments or profiles; also search comments.
onlyNew / stateKey / resetSeen	boolean / string / boolean	false / empty / false	Monitor mode, optional monitor name, clear the seen list.
maxItems	integer	10	Maximum rows across all record types. Free Apify users are limited to 10.

Example input:

{
  "searchQuery": "postgres",
  "storyType": "story",
  "sortBy": "date",
  "minPoints": 100,
  "minComments": 50,
  "dateFrom": "2024-01-01",
  "maxItems": 100
}

🔔 Only new results (monitor mode)

Turn on onlyNew and the actor skips every story and comment this same search already returned in an earlier run. Skipped records are not charged. stateKey names independent monitors and resetSeen starts over.

• The seen list lives in a named key-value store, hackernews-seen, in your own Apify account. A search is identified by its query, story type, filters and stateKey.
• maxItems means "up to N new rows". A record is marked as seen only after it was saved and charged.
• Use sortBy: "date" and, for broad queries, dateFrom, because the index returns at most 1,000 matches per query. If nothing is new the run succeeds and charges nothing.

📦 Hacker News Scraper output fields

Field	Appears on	Description
hnId, author, text	story, comment	Item ID, username and plain text body
title, storyType, commentCount, url, domain	story	Title, type, number of comments, external link and hostname without www.
points	story, comment	Upvotes, when available
headline and hiring* fields	monthly thread post	First line of the post plus hiringCompany, hiringLocation, hiringRemote, hiringApplyUrl, hiringCompanyUrl, only when present
githubStars, githubLanguage, githubPushedAt	story	Only with includeGithubRepo
storyTitle, storyId, parentId, depth	comment	Parent story, parent item, reply level
username, karma, about	user	Profile data
authorKarma, authorCreatedAt	story, comment	Only with includeAuthorProfile
createdAt, updatedAt, createdAtTimestamp, ageHours, pointsPerHour	story, comment	Times, Unix timestamp, age in hours, points per hour
hnUrl, scrapedAt	all	Item or profile link, collection time (UTC)

🤖 Hacker News Scraper MCP server for AI agents

Agents can call this actor through the Apify MCP server at https://mcp.apify.com?tools=RecordsData/hackernews-search-scraper. The URL loads only this actor as a tool.

Config (Claude Desktop, Cursor, VS Code): {"mcpServers": {"apify": {"url": "https://mcp.apify.com?tools=RecordsData/hackernews-search-scraper"}}}

Example agent prompt: "Use the Hacker News Scraper to find the 20 most upvoted stories about postgres since 2024-01-01 and summarize what commenters criticize."

Choose this actor when an agent needs Hacker News stories, comments, comment trees, monthly hiring threads with parsed company, location and apply link, or user profiles as structured rows, with points, comment, author, domain and date filters. It takes the same fields as the input table.

⚖️ How does this Hacker News scraper compare to alternatives?

Competitor pages read on 2026-10-10; a feature counts as missing only if the page does not show it.

Capability	This actor	ryanclinton/hackernews-search	constructive_calm/hacker-news-scraper
Hiring posts	Parsed company, location, remote, apply link; three threads	Parsed company, location, apply link (about 80% accuracy)	Latest "Who is hiring?" thread only
Comment trees	Flat rows, depth 1 to 50	Depth 3, 100 comments by default	Depth 1 to 10, nested or flat
Filters	Points, comments, author, domain, dates	No domain filter listed	No author filter listed
Monitoring	Only-new mode, skipped rows free	Only-new alerts with webhook	Schedules only
Price per 1,000	$12 free tier, $10 Silver	$5 in README ($1.20 in header)	$0.40 stories, $0.15 comments

Both rivals cost much less per row. ryanclinton also adds scoring, GitHub signal classes and webhook alerts, and constructive_calm returns deleted and dead flags, all of which this actor lacks. This actor fits scheduled monitoring that charges only for new rows, the three monthly threads, and comment trees up to depth 50.

🔌 Can I use the Hacker News Scraper through the API?

Yes. Run it with the Apify API or the apify-client packages for JavaScript and Python, or connect it to Zapier, Make, n8n, Airbyte, Slack or Google Sheets.

🛡️ Is it legal to scrape Hacker News?

The actor reads the public Hacker News Search API by Algolia and public user data, does not log in or bypass any protection,. You remain responsible for using the data in line with applicable law and the Hacker News terms.

❓ Frequently asked questions

How do I get only new Hacker News stories and comments since my last run?

Set onlyNew to true in the Hacker News Scraper, sort by date and schedule the actor. Each run returns only stories and comments that the same search has not returned before, and skipped records are free, so scheduled monitoring only pays for new rows.

How do I scrape the Hacker News "Who is hiring?" thread?

Set mode to whoIsHiring and leave hiringMonth empty for the latest thread, or give a month such as 2024-12. You get one row per top-level job post with its headline, text, author, date and the parsed hiring* fields when present. whoWantsToBeHired and freelancer read the other two threads, billed as comment rows.

Can the Hacker News Scraper return full comment threads?

Yes. Turn on includeCommentTree and choose maxDepth from 1 to 50 (default 3). The scraper returns the comments of each story or item as flat rows with depth, parentId and storyId. Rows count toward maxItems and are billed as comment records.

How do I filter Hacker News stories by domain, author or comment count?

Use domain (for example github.com), author and minComments. Hacker News search cannot filter by host, so the actor searches the url field and keeps exact host matches; a query is capped at about 1,000 results. minComments applies to stories only, while author covers stories and comments.

Does the Hacker News Scraper return deleted or dead items?

It has no deleted or dead fields, because the Algolia search source does not provide them. Dead or deleted items can therefore appear as normal rows in the Hacker News Scraper. For those flags, use the official Hacker News API with the returned item ID.

How do I get author karma on Hacker News stories and comments?

Turn on includeAuthorProfile. It adds authorKarma to each story and comment row, with one profile lookup per distinct author, and authorCreatedAt when available. Each enriched row is billed as the Author karma event, $0.002 per row, on top of the story or comment row.

Why did my Hacker News Scraper search return 0 results?

The query and filters matched nothing in the Hacker News index. Try a shorter keyword, lower minPoints, widen the date range, or switch storyType. A Hacker News Scraper run with no matches still succeeds with a status message and costs nothing.

How far back does the Hacker News Scraper data go?

The Hacker News Scraper searches the Hacker News index from the start of the site in 2007 to today. The item counter read 49,950,816 on 2026-10-04. Use dateFrom and dateTo in YYYY-MM-DD format to slice any window of Hacker News history.

What does sorting by relevance mean in the Hacker News Scraper?

In the Hacker News Scraper, relevance sorting uses the Algolia Hacker News search index, which blends how well the text matches your query with popularity. Choose date (newest first) for monitoring, because relevance order can shift between runs and break monitors.

Do Hacker News Scraper comment rows include the story they belong to?

Yes. When you turn on includeComments, each Hacker News Scraper comment row carries its parent storyTitle and storyId next to the author, text, date and hnUrl. That lets you join comments back to their stories in a spreadsheet or database, or group a whole discussion by story.

Can the Hacker News Scraper get the current Hacker News front page?

Yes. Set storyType to front_page in the Hacker News Scraper and it returns the stories currently on the Hacker News front page, with points, comment counts, authors and links. The same field also supports story, ask_hn, show_hn, job and poll.

Can I scrape Hacker News user profiles with the Hacker News Scraper?

Yes. Add usernames to the usernames input and it returns one profile row per user with karma, about text and account creation date. Unknown usernames return an uncharged error row, and each valid profile is billed as one User profile event.

Am I charged for failed or empty Hacker News Scraper rows?

No. The Hacker News Scraper uses pay per event pricing and charges only for rows saved to the dataset. Error rows, duplicates, empty results and failed runs are not charged, and the run stops cleanly at your max charge limit.

What happens if I send an empty or invalid input to the Hacker News Scraper?

An empty {} input runs a small 10-row sample with the default search. The Apify platform rejects wrong types or enum values with a 400 error before a run starts. Other mistakes, such as an unreadable date, end the run as SUCCEEDED with one free error row (errorType: invalid-input).

Can I try the Hacker News Scraper for free?

Yes. Free Apify users get a 10-row preview of the Hacker News Scraper, which is enough to check the fields and format. On the free tier a story row costs $0.012. Paid Apify plans unlock larger runs and lower story prices.

How do I get a CSV or Excel file from the Hacker News Scraper?

Open the dataset of your finished run and choose CSV, Excel, JSON or XML. With the API, add ?format=csv to the dataset items URL to get CSV directly. Each row has a recordType column, so stories, comments and users are easy to filter.

📬 Support

Found a bug or a missing field? Open the Issues tab or write to contact.punkrecordsdata@gmail.com.

Last updated: 2026-10-11