Hacker News Scraper — Stories, Comments, Jobs & Users 🚀

🎉 Pull the entire Hacker News front page, full comment threads, search results and user profiles into one clean dataset — no API key, no code.

Built for founders, researchers, growth teams and devs who track what tech is talking about.

---

📋 Overview

Want to know what tech is reading right now? Hacker News Scraper turns any front-page list, search query, comment thread or user profile into clean, structured JSON/CSV — ready for your dashboard, model or spreadsheet.

Why thousands of professionals choose us:

• ✅ Everything HN, one tool: stories, comments, Ask HN, Show HN, jobs, polls and user profiles
• ⚡ Fast overview mode: top/new/best lists and full-text search in seconds
• 🎯 Deep detail mode: full body text, the parent chain and the whole comment thread (kids)
• 💰 Pay-per-result: only pay for items you actually extract — generous free tier, unlimited paid
• 💎 Author enrichment: optionally attach each author's karma, account age and submission count
• ✨ Zero setup: no account, no API key, no rate-limit juggling — just press Start

---

✨ Features

• 📰 Front-Page Lists: scrape Top, Newest, Best, Ask HN, Show HN and Jobs
• 🔎 Full-Text Search: find every story or comment mentioning a keyword, product or company
• 🏷️ Type Filters: restrict to stories, comments, polls, jobs, Ask HN or Show HN
• 🧵 Full Comment Threads: detail mode returns the body text and child comment IDs
• 👤 User Profiles: karma, account creation date, bio and submission count by username
• 🔗 Paste-a-URL: drop in a news.ycombinator.com or search URL and go
• 🆕 Sort by Date or Relevance: choose recency or best-match ranking for searches
• 📦 Clean Exports: JSON, CSV and Excel straight from the Apify dataset

---

🎬 Quick Start

Pick a mode, set what you want, press Start. Results stream into the dataset in seconds. Below: pull the 30 top front-page stories.

curl -X POST https://api.apify.com/v2/acts/sian.agency~hacker-news-scraper/runs?token=YOUR_TOKEN \
-H 'Content-Type: application/json' \
-d '{"scrapeMode": "overview", "searchMode": "byList", "list": "top", "maxResults": 30}'

---

🚀 Getting Started (3 Simple Steps)

Step 1: Choose your mode

Pick Overview (fast lists/search), Detail (full text + thread) or User (profiles).

Step 2: Tell it what to scrape

Select a list, type a search query, or paste item IDs / usernames.

Step 3: Press Start

Items stream into the dataset — export to JSON, CSV or Excel.

That's it! In under a minute, you'll have:

• A structured table of stories, comments or profiles
• Scores, authors, comment counts and links
• A dataset ready for analysis, alerts or dashboards

---

📥 Input Configuration

Field	Type	Required	Description
scrapeMode	string	No	overview, detail or user
searchMode	string	No	byList, byQuery, bySearchUrl, byItemUrl, byUser (auto-detected)
list	string	No	Front-page list: top, new, best, ask, show, job
query	string	No	Full-text search query
tags	string	No	story, comment, poll, job, ask_hn, show_hn
byDate	boolean	No	Sort search by recency instead of relevance
searchUrls	array	No	Paste search or front-page URLs
itemUrls	array	No	Detail mode: item IDs or item?id= URLs
usernames	array	No	User mode: usernames or user?id= URLs
withUser	boolean	No	Detail mode: also fetch each author's profile
maxResults	integer	No	Max items per run (FREE: 25, PAID: unlimited)
maxPages	integer	No	Search paging depth
hitsPerPage	integer	No	Search page size

Example — search stories about "rust":

{
  "scrapeMode": "overview",
  "searchMode": "byQuery",
  "query": "rust",
  "tags": "story",
  "maxResults": 100
}

Example — full detail on specific items:

{
  "scrapeMode": "detail",
  "itemUrls": ["48649435", "https://news.ycombinator.com/item?id=37392676"],
  "withUser": true
}

---

📤 Output

Results are saved to the Apify dataset with rich, structured fields including:

Field	Type	Description
id	number	The Hacker News item ID
type	string	story · comment · job · poll
storyTitle	string	The item title
target_url	string	The external link the story points to
score	number	Points / upvotes
author	string	Submitting username
num_comments	number	Total comment count
time	number	Creation time (epoch seconds)
text	string	Body text (Ask HN / comment / text post)
kid_ids	array	Child comment IDs (the thread)
url	string	Hacker News page for the item
karma	number	User mode: account karma
about	string	User mode: profile bio

Example:

{
  "id": 48649435,
  "url": "https://news.ycombinator.com/item?id=48649435",
  "type": "story",
  "storyTitle": "Jerry's Map",
  "target_url": "http://www.jerrysmap.com/the-map",
  "score": 318,
  "author": "turtleyacht",
  "num_comments": 47,
  "time": 1718900000,
  "kid_count": 12
}

---

💼 Use Cases & Examples

1. Tech Trend Tracking

Founders & analysts watching what's gaining traction.

Input: list: top, overview mode Output: ranked front-page stories with scores and comment counts Use: a daily snapshot of what the tech community is reading.

2. Brand & Product Monitoring

Growth teams tracking mentions of their product or competitors.

Input: query: "your product", tags: story,comment Output: every story and comment mentioning the keyword Use: sentiment, alerts and reputation monitoring.

3. Comment & Thread Analysis

Researchers studying discussion structure.

Input: detail mode on a story ID Output: body text plus the full child comment thread Use: discourse analysis, summarization, training data.

4. Hiring & Jobs Intelligence

Recruiters mining "Who is hiring?" and the jobs list.

Input: list: job or query: "who is hiring" Output: job posts with text and links Use: a structured feed of tech roles.

5. User & Community Research

Community managers profiling top contributors.

Input: user mode with a list of usernames Output: karma, account age, bio and submission count Use: influencer mapping and outreach.

6. Dataset Building for AI

ML engineers building tech-news corpora.

Input: search or list mode at scale Output: clean, labeled rows of stories and comments Use: fine-tuning, RAG and benchmarks.

---

🔗 Integration Examples

JavaScript/Node.js

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_TOKEN' });

const run = await client.actor('sian.agency/hacker-news-scraper').call({
  scrapeMode: 'overview',
  searchMode: 'byList',
  list: 'top',
  maxResults: 30,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items[0]);

Python

from apify_client import ApifyClient
client = ApifyClient('YOUR_TOKEN')

run = client.actor('sian.agency/hacker-news-scraper').call(
    run_input={'scrapeMode': 'overview', 'query': 'rust', 'tags': 'story'}
)

for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item)

cURL

curl -X POST 'https://api.apify.com/v2/acts/sian.agency~hacker-news-scraper/runs?token=YOUR_TOKEN' \
-H 'Content-Type: application/json' \
-d '{"scrapeMode": "overview", "list": "best", "maxResults": 50}'

Automation Workflows (N8N / Zapier / Make)

1. Trigger: Schedule (e.g. hourly) or webhook
2. HTTP Request: Call the actor API
3. Process: Handle the JSON results
4. Action: Save to a sheet, send a Slack alert, or feed a model

---

📊 Performance & Pricing

FREE Tier (Try It Now)

• 25 items per run — full feature access, same quality
• No credit card required
• Perfect for testing and small projects

PAID Tier (Production Ready)

• Unlimited items per run
• Pay-per-result: only charged for items you actually extract
• Best for monitoring, dashboards and large datasets

💰 Transparent pay-per-result pricing — you're never charged for an invalid run.

🔗 View current pricing

---

❓ Frequently Asked Questions

Q: How many items can I scrape? A: FREE tier: 25 per run. PAID tier: unlimited.

Q: Can I get the full comment thread of a story? A: Yes — use Detail mode. You get the body text plus the child comment IDs.

Q: Can I search for a keyword across all of Hacker News? A: Yes — use Overview mode with a query and optional tags to restrict to stories, comments, Ask HN, etc.

Q: What output formats are available? A: JSON, CSV and Excel — export directly from the Apify dataset.

Q: Do I need a Hacker News account or API key? A: No. The actor only reads publicly available content — no login required.

Q: Can I get user karma and profile info? A: Yes — use User mode with usernames, or enable author enrichment in Detail mode.

Q: Is this legal? A: Yes — we only extract publicly available data. See the legal section below.

---

🐛 Troubleshooting

No results returned

• In byQuery mode, make sure query is set; in byList mode, make sure list is valid.
• Some niche searches genuinely return few hits — try broader terms or byDate: false.

Hit the free-tier cap

• FREE runs stop at 25 items. Upgrade to PAID for unlimited items per run.

Detail mode returned fewer rows than expected

• Deleted or dead items are skipped automatically — that's expected.

Search isn't recent enough

• Set byDate: true to sort by recency instead of relevance.

---

⚖️ Is it legal to scrape data?

Our actors are ethical and do not extract any private user data, such as email addresses, gender, or location. They only extract what the user has chosen to share publicly. We therefore believe that our actors, when used for ethical purposes by Apify users, are safe.

However, you should be aware that your results could contain personal data. Personal data is protected by the GDPR in the European Union and by other regulations around the world. You should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers.

You can also read Apify's blog post on the legality of web scraping.

---

🤝 Support

Join our active support community

• For issues or questions, open an issue in the actor's repository
• Check SIÁN Agency Store for more automation tools
• 📧 apify@sian-agency.online

---

📌 Disclaimer

This actor is an independent tool and is not affiliated with, endorsed by, or sponsored by Hacker News or Y Combinator. "Hacker News" and "Y Combinator" are trademarks of their respective owners. The actor accesses only publicly available content and is intended for lawful use in line with applicable terms and regulations.

---

Built by SIÁN Agency | More Tools