Hacker News Scraper — Stories, Comments & News

Hacker News Scraper — Stories, Comments & News

Hacker News Scraper extracts structured Hacker News records, metadata, metrics, and direct links in real time. Built on a zero-browser HTTP/2 engine running in 256 MB RAM, it delivers normalized datasets with zero null fields ready for CSV, Excel, JSON, or API pipelines.

Looking for related scrapers? Check out Reddit Scraper, LinkedIn Jobs Scraper, Google Autocomplete & SEO Scraper, Web to Markdown Converter for AI & RAG, or browse our full catalog of 118+ dedicated scrapers.

---

What does Hacker News Scraper do?

Extract top tech stories, startup launches (Show HN), community questions (Ask HN), karma scores, and full discussion comment trees with .... Compatible Make.com, Zapier, n8n, Sheets & Webhooks. 256MB RAM & sub-second speed.

Whether you need to monitor Hacker News at scale, enrich B2B/B2C databases, or feed structured records into analytics and AI workflows, Hacker News Scraper handles pagination, field normalization, and schema validation automatically:

• Bulk Data Extraction: Collect Hacker News listings, items, and detailed attributes in batches using keywords, filters, or direct URLs.
• Complete Public Metadata: Capture identifiers, pricing/engagement metrics, author/vendor details, timestamps, and canonical URLs.
• Multi-Format Export: Download results in JSON, CSV, Excel (XLSX), XML, or HTML Table directly from the Apify Console.
• Ready for Automation & AI: Connect seamlessly to Apify Integrations (Make, Zapier, n8n, Google Sheets), REST API endpoints, or the Apify MCP Server.

Beyond the Official Hacker News API

• No API Keys or Approval Delays: Start extracting public Hacker News data immediately without registering developer apps or waiting for quota approvals.
• No Restrictive Rate Quotas: Bypass strict per-minute API limits and expensive enterprise tiers while capturing the complete public data points displayed on Hacker News.
• 256 MB Zero-Browser Architecture: Uses native HTTP/2 and TLS session emulation instead of heavy Chromium instances, executing in seconds at a fraction of standard compute costs.

---

What data can you scrape from Hacker News?

Hacker News Scraper extracts the following data points from Hacker News:

📝 Section (section)	📝 Min Points (minPoints)	💬 Min Comments (minComments)
💬 Scrape Comments (scrapeComments)	🆔 ID (id)	📝 Title (title)
👤 Author (author)	📝 Points (points)	💬 Num Comments (numComments)
🔗 URL (url)	🔗 Hn URL (hnUrl)	📝 Domain Icon (domainIcon)

Data Category	Extracted Fields & Data Points
Identity & Core Identifiers	id (ID), title (Title)
Metrics, Pricing & Engagement	minPoints (Min Points), minComments (Min Comments)
Details, Attributes & Context	section (Section), scrapeComments (Scrape Comments), author (Author), points (Points), numComments (Num Comments), domainIcon (Domain Icon)
Direct Links, Media & Timestamps	url (URL), hnUrl (Hn URL), createdAt (Created At)

---

Why scrape Hacker News? (Typical Use Cases & Guides)

You can use Hacker News Scraper for a wide range of data and business workflows:

• Tech & AI Trend Spotting: Catch breakout developer tools and GitHub repos before they go viral.
• Customer Research: Track what developers praise and criticize in Hacker News comment sections.
• Automated Tech Newsletters: Curate top weekly stories for developer newsletters and Discord channels.
• AI Agents, RAG & LLM Knowledge Pipelines: Stream clean, structured Hacker News JSON records into vector databases, Claude, GPT-4, or LangChain workflows.
• Automated BI Dashboards & Scheduled Alerts: Schedule daily or hourly runs and push live Hacker News updates directly to Google Sheets, Looker Studio, Slack, or webhooks.
• Market Research & Competitor Analysis: Benchmark Hacker News offerings, pricing strategies, and market share across regions.
• Lead Generation & CRM Enrichment: Build targeted prospect lists and sync verified records straight into HubSpot, Salesforce, or Clay.
• AI Agents & RAG Pipelines via MCP: Connect live Hacker News data directly to Claude Desktop, Cursor, or LangChain agents.

---

How to scrape Hacker News data

Hacker News Scraper is designed for a zero-configuration start, even if you have never scraped data before:

1. Create a free Apify account using your email (includes $5.00 in free monthly credits).
2. Open Hacker News Scraper and click on the Input tab — all required fields are already pre-filled with a working example.
3. Enter your search terms, target URLs, or filter criteria (or keep the default pre-filled configuration for an instant 1-Click test).
4. Click the green "Start" button and wait a few seconds for the data to be extracted.
5. Download your dataset in JSON, CSV, Excel (XLSX), or HTML, or fetch it via the API tab.

For more technical details on every parameter, check out the Input Schema tab or read the Apify Platform Documentation.

🔗 How to scrape Hacker News by Search Term or Direct URL

Scraping by Search Term / Filter 🔑	Enter keywords, categories, or filter parameters just as you would search on Hacker News. Set the maximum number of items you want to extract and click Start.
Scraping by Direct URL 🔗	Paste one or more direct Hacker News links (or import a list of URLs from a Google Sheet / CSV) to extract targeted item or profile details in bulk.

---

⬇️ Input Configuration & Examples

To start scraping Hacker News, fill in the visual input form in the Apify Console or pass a JSON payload via the API. Full technical specifications are available on the Input Schema tab.

Input Parameters Reference

Parameter	Type	Default / Prefill	Description
section	string	front_page	Feed section to extract stories from. Options: front_page, latest, ask_hn, show_hn, jobs.
searchQueries	array	[]	Search Hacker News archive for specific topics (e.g. 'LLM', 'AI agent', 'Kubernetes').
minPoints	integer	0	Only keep stories with at least this number of points/upvotes (e.g. 50, 100, 300).
minComments	integer	0	Only keep stories that generated active discussion with at least N comments.
scrapeComments	boolean	true	Extract top-level comments and discussion insights for each story.
maxCommentsPerStory	integer	10	Maximum number of comments to extract per story.
maxItems	integer	50	Maximum number of stories to retrieve.
proxyConfiguration	object	—	Apify Proxy configuration.

Example 1: Standard Extraction (1-Click Default)

{
  "section": "front_page",
  "minPoints": 0,
  "minComments": 0,
  "scrapeComments": true
}

Example 2: Filtered & Batch Extraction

{
  "section": "latest",
  "searchQueries": [],
  "minPoints": 0,
  "minComments": 0,
  "scrapeComments": true,
  "maxCommentsPerStory": 20,
  "maxItems": 100
}

---

⬆️ Output Example (Table View, JSON & Fields Reference)

The scraped results are stored in a structured Apify Dataset available in the Storage tab. Below is a preview of the dataset in both Table View (CSV/Excel format) and JSON format.

Table View (CSV / Excel Dataset Preview)

Section	Min Points	Min Comments	Scrape Comments
front_page	0	0	true
front_page	1	1	true
front_page	1	1	true

JSON Output Example

{
  "section": "front_page",
  "minPoints": 0,
  "minComments": 0,
  "scrapeComments": true
}

Output Fields Reference

Field Name	Type	Description
section	string	Extracted section value.
minPoints	number	Extracted min points value.
minComments	number	Extracted min comments value.
scrapeComments	boolean	Extracted scrape comments value.
id	string	Unique Hacker News story ID
title	string	Story headline
author	string	Submitter username
points	integer / number	Story upvotes or karma points
numComments	integer / number	Total comment count
url	string	Destination article URL
hnUrl	string	Hacker News discussion link
domainIcon	string	Extracted domain icon value.
storyText	string	Extracted story text value.
comments	array	Top comments with author and text
createdAt	string	Submission ISO timestamp

---

🔗 Scrape any data you need with dedicated scrapers

Combine Hacker News Scraper with our other dedicated scrapers from @RBBscrap if you want to extract specialized data across platforms:

🔹 Reddit Scraper	🔹 LinkedIn Jobs Scraper	🔹 Google Autocomplete & SEO Scraper
🔹 Web to Markdown Converter for AI & RAG	🔹 Glassdoor Scraper	🔹 YouTube Scraper

Scraper	Extracted Data & Primary Use Case
Reddit Scraper	Posts, Comments & Subreddits
LinkedIn Jobs Scraper	B2B Leads & Salaries
Google Autocomplete & SEO Scraper	Search Ideas
Web to Markdown Converter for AI & RAG	Extract structured Web to Markdown datasets and metrics
Glassdoor Scraper	Salaries, Reviews & Jobs
YouTube Scraper	Videos, Shorts & Channels

---

⚠️ Error Items & Resilience Reference

Hacker News Scraper is engineered to process multi-query and batch runs resiliently without crashing if a single query or URL fails:

Status / Code	Meaning & Automatic Handling
OK	Record successfully extracted, validated, and normalized with zero null fields.
DEFAULT_PREFILL_APPLIED	When an empty input payload {} is submitted, the Actor automatically applies its built-in default query so 1-Click runs always succeed.
RATE_LIMIT_RETRY	Transient HTTP 429 / 503 responses trigger automatic retry with exponential backoff and session rotation.
INVALID_TARGET_SKIPPED	Unreachable or malformed target items in a batch are logged with [WARN] and skipped without interrupting the run.

---

❓ Frequently Asked Questions (FAQ)

How much does scraping Hacker News cost?

Hacker News Scraper uses transparent Pay-Per-Event (PAY_PER_EVENT) pricing set 10x cheaper than the market leader ($\div 10$):

• Event Rate: $0.00025 per Scraped Hacker News Scraper Record ($0.25 per 1,000 items).
• 🎁 1st Run 100% FREE: Your very first execution is 100% free ($0.00 event charges are applied on your initial run).
• Apify Free Plan ($5.00/month included): Every Apify account receives $5.00 in free monthly credits, which is enough to extract up to ~20,000 items per month for free.
• View full details on the Pricing tab and the Apify Pricing page.

Can I integrate Hacker News Scraper with Make, Zapier, n8n, or Google Sheets?

Yes. Hacker News Scraper can be connected with almost any cloud service or web app thanks to integrations on the Apify platform, including Make, Zapier, n8n, Slack, Airbyte, Google Sheets, Google Drive, and webhooks.

Can I use Hacker News Scraper with the Apify API (Python & Node.js)?

Yes. The Apify API gives you programmatic access to run Hacker News Scraper, schedule jobs, and fetch datasets. Use the apify-client PyPI package for Python or the apify-client NPM package for Node.js, or check the Apify API Reference:

Python (apify-client)

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

run = client.actor("rbbscrap/hacker-news-scraper").call(run_input={
    "section": "front_page",
    "minPoints": 0,
    "minComments": 0,
    "scrapeComments": True
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

Node.js (apify-client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_API_TOKEN' });

const run = await client.actor('rbbscrap/hacker-news-scraper').call({
    section: "front_page",
    minPoints: 0,
    minComments: 0,
    scrapeComments: true
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Can I use Hacker News Scraper through an MCP Server (Claude Desktop, Cursor, AI Agents)?

Yes. With the Apify API, you can use Hacker News Scraper in conjunction with an MCP server to connect directly to Claude Desktop, Cursor, Windsurf, or LibreChat. Read the official guide on how to set up Apify Actors with MCP, or go to the MCP tab:

1. Start a Server-Sent Events (SSE) session to receive a sessionId.
2. Send API messages using that sessionId to trigger Hacker News Scraper with your input parameters.
3. Receive structured Hacker News dataset records directly inside your AI agent conversation.

Do I need residential proxies or login cookies to scrape Hacker News?

No. Hacker News Scraper operates on a zero-browser HTTP/2 protocol engine (256 MB RAM) and works out-of-the-box without requiring login cookies or external proxies for standard workloads. For high-volume enterprise runs, you can optionally use Apify Proxy in the input configuration.

Is it legal to scrape data from Hacker News?

Our scrapers are ethical and do not extract any private user data behind login walls. They only extract what is shared publicly on the web. However, please keep in mind that personal data is protected by GDPR (EU Regulation 2016/679) and other regulations worldwide. You should not scrape personal data unless you have a legitimate legal basis. Read more in Apify's guide on the legality of web scraping.

Your feedback & support

We are continuously improving the performance of our Actors. If you have any technical feedback for Hacker News Scraper, need a custom feature, or found a bug, please create an issue on the Actor's Issues tab in the Apify Console.