Hacker News Scraper - Fast & Reliable Data Extraction

Extract stories, discussions, and job postings from Hacker News with this powerful scraper. Get real-time data from multiple categories including front page news, newest submissions, Ask HN, Show HN, and past stories.

🚀 Key Features

• Multi-Category Scraping: Scrape from default news, newest, past news, Ask HN, and Show HN simultaneously
• Asynchronous Processing: Fast concurrent scraping of multiple categories at once
• Structured Data: Get clean, structured JSON data with titles, URLs, and scores
• Real-time Updates: Access the latest Hacker News content as it's published
• No Rate Limiting Issues: Built-in request throttling to respect Hacker News servers
• Easy to Use: Simple configuration with just a few parameters
• Scalable: Handle large-scale data extraction efficiently

📊 What Data Can You Extract?

This Hacker News scraper extracts the following data points from each post:

• Title: Full story or discussion title
• URL: Link to the external article or HN discussion
• Score: Number of upvotes/points
• Category: Source category (default, new, ask, show, past news)

Perfect for:

• 📈 Tech trend analysis and monitoring
• 🔍 Content discovery and curation
• 📰 News aggregation
• 🎯 Competitive intelligence
• 📊 Data analysis and research
• 🤖 Training ML models on tech discussions
• 💼 Job market analysis

🎯 Use Cases

Content Creators & Marketers

• Monitor trending tech topics for content ideas
• Track what stories resonate with the tech community
• Identify popular discussion themes
• Discover viral content patterns

Researchers & Analysts

• Analyze Hacker News trends over time
• Study community interests and engagement patterns
• Extract data for academic research
• Track technology adoption and sentiment

Developers & Startups

• Track Show HN launches and feedback
• Monitor competitor mentions
• Identify emerging technologies and tools
• Analyze community reception of products

Job Seekers & Recruiters

• Access Hacker News job postings
• Monitor hiring trends in tech
• Find companies actively recruiting
• Track in-demand skills and technologies

💡 How to Use

Input Configuration

Configure the scraper with these simple parameters:

{
  "categories": ["default", "show", "ask"],
  "page_count": 2
}

Input Parameters

Parameter	Type	Required	Default	Description
categories	Array	No	["default"]	Categories to scrape from Hacker News
page_count	Integer	No	1	Number of pages per category (1-10)

Available Categories

Choose from these Hacker News sections:

• default: Front page news and popular stories
• new: Newest submissions across all categories
• past news: Previously popular front page stories
• ask: Ask HN discussions and questions
• show: Show HN project launches and demos

Example Configurations

Scrape front page stories (1 page)

{
  "categories": ["default"],
  "page_count": 1
}

Scrape Show HN and Ask HN (3 pages each)

{
  "categories": ["show", "ask"],
  "page_count": 3
}

Comprehensive scrape (all categories, 2 pages each)

{
  "categories": ["default", "new", "past news", "ask", "show"],
  "page_count": 2
}

Maximum data extraction

{
  "categories": ["default", "new", "past news", "ask", "show"],
  "page_count": 10
}

📤 Output Format

Each item in the dataset contains:

{
  "title": "Show HN: I built a tool for analyzing Hacker News trends",
  "url": "https://example.com/hn-analytics",
  "score": "324 points",
  "category": "show"
}

Output Fields

Field	Type	Description
title	String	Story or discussion title
url	String	Link to content or discussion
score	String	Upvotes/points (e.g., "324 points")
category	String	Source category

🎬 Quick Start Guide

Step 1: Open the Actor

Click "Try for free" or "Start" button

Step 2: Configure Input

Select your desired categories and page count:

• Choose one or more categories from the dropdown
• Set pages per category (1-10)

Step 3: Run the Actor

Click "Start" and wait for the scraper to complete

Step 4: Download Your Data

Export your data in multiple formats:

• JSON
• CSV
• Excel
• HTML Table
• RSS Feed

📈 Performance & Limits

• Items per Page: ~30 items
• Pages per Category: 1-10 pages
• Maximum Items per Category: Up to 300 items
• Concurrent Processing: All categories scraped simultaneously
• Average Speed: ~1-2 seconds per page
• Built-in Delays: Automatic rate limiting for reliability

Recommended Settings

Use Case	Categories	Pages	Est. Items
Quick check	["default"]	1	~30
Daily monitoring	["default", "new"]	2	~120
Comprehensive analysis	All categories	5	~750
Maximum extraction	All categories	10	~1,500

🔄 Integration & Automation

Scheduled Runs

Set up automatic scraping:

• Hourly updates for breaking tech news
• Daily digests of top stories
• Weekly trend analysis

Connect to Your Tools

Export data directly to:

• Google Sheets: Automatic spreadsheet updates
• Slack: Get notifications of new content
• Webhooks: Send data to your applications
• Zapier/Make: Integrate with 1000+ apps
• Email: Receive data reports

API Access

Access your scraped data via:

• Apify API
• Direct dataset download
• Webhook delivery
• RSS feed

💎 Pro Tips

Optimize Your Scraping

For Real-time Monitoring

{
  "categories": ["new"],
  "page_count": 1
}

Run every 15-30 minutes to catch new submissions.

For Trend Analysis

{
  "categories": ["default"],
  "page_count": 5
}

Run daily to track what's trending.

For Research Projects

{
  "categories": ["default", "ask", "show"],
  "page_count": 10
}

Run weekly for comprehensive datasets.

For Job Hunting Monitor "Who is hiring?" threads in Ask HN category.

Data Analysis Ideas

• Track keyword frequency in titles
• Analyze score distribution patterns
• Monitor domain popularity
• Identify trending topics over time
• Compare engagement across categories

🔒 Privacy & Compliance

• ✅ Scrapes only publicly available data
• ✅ No authentication required
• ✅ Respects Hacker News rate limits
• ✅ GDPR compliant (no personal data)
• ✅ No cookies or tracking
• ✅ Ethical scraping practices

🐛 Troubleshooting

No Results Returned

Possible causes:

• Category name misspelled (check spelling and case)
• Page count set to 0
• Network connectivity issues

Solution: Verify input parameters and try again

Partial Results

Possible causes:

• Some categories may have fewer items
• Pages beyond available content return empty

Solution: This is normal behavior - you'll get all available items

Slow Performance

Possible causes:

• Scraping many categories with high page counts
• High platform load

Solution: Reduce page_count or split into multiple runs

📊 Sample Use Cases with Code

Analyze Top Stories

Get the top 100 stories and analyze trends:

{
  "categories": ["default"],
  "page_count": 4
}

Monitor Show HN Launches

Track new product launches:

{
  "categories": ["show"],
  "page_count": 3
}

Research Community Discussions

Study Ask HN questions and discussions:

{
  "categories": ["ask"],
  "page_count": 5
}

📚 Related Resources

• Hacker News Official Site
• Hacker News Guidelines
• Y Combinator
• Apify Platform Documentation

🎯 What Makes This Scraper Different?

✨ Multi-Category Support: Scrape multiple sections simultaneously
⚡ Fast & Efficient: Asynchronous processing for speed
🎨 Clean Data: Structured, ready-to-use JSON output
🔄 Reliable: Built-in error handling and retries
📊 Scalable: Handle small to large-scale extraction
🚀 Easy to Use: No coding required, just configure and run

❓ FAQ

Q: How many items can I scrape?
A: Up to 300 items per category (10 pages × ~30 items)

Q: How often can I run this scraper?
A: As often as needed, but we recommend respecting rate limits

Q: Can I scrape historical data?
A: The "past news" category shows previously popular stories

Q: Is this free to use?
A: Free tier available. Check Apify pricing for usage limits

Q: Can I export to CSV/Excel?
A: Yes! Export in JSON, CSV, Excel, HTML, or RSS

Q: How fresh is the data?
A: Data is scraped in real-time when you run the actor

💡 Need Help?

• 📖 Read the documentation above
• 💬 Contact support through Apify platform
• 🐛 Report issues via actor feedback
• ⭐ Rate and review if you find this useful!

🏷️ Keywords

hacker news scraper, hn scraper, tech news extractor, news aggregator, hacker news data, show hn, ask hn, y combinator news, tech trends, startup news, discussion forum scraper, web scraping, data extraction, news api, tech community, developer news, startup stories

---

Ready to extract Hacker News data? Click "Start" to begin scraping!