🚀 Hacker News Scraper

Scrape stories, comments, and user data from Hacker News at scale. Access top stories, new posts, Ask HN, Show HN, jobs, and perform full-text search with advanced filters. Perfect for tech trend monitoring, competitive analysis, recruitment research, and content discovery.

Target Audience: Tech analysts, product managers, recruiters, content creators, academic researchers Primary Use Cases: Tech news monitoring, trend analysis, competitive intelligence, developer community research

What Does Hacker News Scraper Do?

This tool collects data from Hacker News (https://news.ycombinator.com) in real-time. It delivers:

• Story titles, URLs, domains, authors, points, and comment counts
• Full comment threads with nested replies and depth control
• Author profiles with karma scores and account age
• Full-text search with filters for points, comments, and date ranges
• Six feed types: Top, New, Best, Ask HN, Show HN, and Jobs

Business Value: Stay ahead of tech trends by monitoring what the developer community is discussing, sharing, and building.

📊 What You Get

This tool supports 2 scraping modes:

• Feed Mode - Browse curated feeds: Top Stories, New, Best, Ask HN, Show HN, or Jobs
• Search Mode - Full-text search across all Hacker News content with advanced filtering

Input

To start collecting data, fill in the input form. You can configure:

• mode - Choose between feed or search
• maxItems - How many items to collect (free users get up to 100)
• feedType - Which feed to browse (top, new, best, ask, show, job)
• searchQuery - Keywords to search for

Here's what the input configuration looks like in JSON:

{
    "mode": "feed",
    "feedType": "top",
    "maxItems": 10
}

Search example:

{
    "mode": "search",
    "searchQuery": "web scraping",
    "contentType": "story",
    "sortBy": "relevance",
    "minPoints": 50,
    "maxItems": 25
}

Output

After the Actor finishes its run, you'll get a dataset with the output. You can download those results as an Excel, HTML, XML, JSON, and CSV document.

Here's an example of scraped data:

{
    "title": "Show HN: I built an open-source AI code editor",
    "url": "https://example.com/project",
    "domain": "example.com",
    "author": "username",
    "points": 342,
    "numComments": 89,
    "type": "story",
    "storyText": null,
    "createdAt": "2025-01-15T14:30:00.000Z",
    "hnUrl": "https://news.ycombinator.com/item?id=12345678",
    "id": 12345678,
    "authorKarma": 15420,
    "authorCreatedAt": "2018-03-20T10:00:00.000Z",
    "comments": [
        {
            "id": 12345679,
            "author": "commenter",
            "text": "Great project!",
            "createdAt": "2025-01-15T15:00:00.000Z",
            "replies": []
        }
    ]
}

What You Get: Story metadata (title, URL, domain, type), engagement data (points, comment count), author info (karma, account age), and full comment threads when enabled. Download Options: CSV, Excel, or JSON formats for easy analysis.

⚡ Why Choose Hacker News Scraper?

• 📰 Two powerful modes - Browse 6 different feeds or search with full-text queries
• 💬 Nested comment threads - Fetch replies with configurable depth (up to 10 levels)
• 👤 Author enrichment - Get karma scores and account creation dates
• 🎛️ Advanced search filters - Filter by minimum points, minimum comments, and date ranges
• ⏰ Real-time data - All data collected directly from Hacker News in real-time

Time Savings: What would take hours of manual browsing and copy-pasting takes just minutes. Efficiency: Collect thousands of stories with a single click instead of manual research.

🏆 How We Compare

Our Hacker News Scraper is built for speed and flexibility. Unlike other scrapers that require a startUrl and only support one mode, ours offers both Feed Mode (6 feed types) and Search Mode with advanced filters - all without needing any URLs. It provides 12 configurable input fields including minimum points, minimum comments, date ranges, comment depth control, and author enrichment. The result is a faster, more complete scraping experience with richer data output.

Feature	Hacker News Scraper (Ours)	Hacker News Data Scraper
Pricing	$10/month + usage	$10/month + usage
Speed	100 items with comments in ~12 seconds	~50 seconds for 100 items with comments
Modes	Feed Mode + Search Mode (no startUrl needed)	Requires startUrl
Inputs	12 configurable fields (filters, sorting, depth)	6 basic fields
Comments	Nested replies with configurable depth (up to 10 levels) and comments per story	Basic comment hierarchy

🔧 How to Use

1. Sign Up: Create a free account w/ $5 credit (takes 2 minutes)
2. Find the Scraper: Visit the Hacker News Scraper page on Apify
3. Set Input: Choose your mode (feed or search) and configure parameters
4. Run It: Click "Start" and let it collect your data
5. Download Data: Get your results in the "Dataset" tab as CSV, Excel, or JSON

No Technical Skills Required: Everything is point-and-click.

Business Use Cases

Product Managers:

• Track discussions about your product or competitors
• Monitor developer sentiment around new features
• Identify pain points and feature requests from the tech community

Recruiters:

• Find active developers discussing relevant technologies
• Research candidates by analyzing their HN contributions
• Identify trending skills and technologies

Content Creators:

• Discover trending topics in the tech community
• Find popular content formats that resonate with developers
• Research what gets upvoted and discussed

Analysts:

• Monitor startup launches and funding discussions
• Track technology adoption trends over time
• Study community engagement patterns and sentiment

💬 About Comment Enrichment

When you enable "Include Comments", the scraper fetches comment threads for each story. Keep in mind:

• Each story requires separate requests for comments
• You can control depth (up to 10 levels) and number of comments per story
• Comment fetching adds processing time proportional to the number of stories
• Nested replies are included with full author and timestamp data

Using Hacker News Scraper with the Apify API

For advanced users who want to automate this process, you can control the scraper programmatically with the Apify API. This allows you to schedule regular data collection and integrate with your existing business tools.

• Node.js: Install the apify-client NPM package
• Python: Use the apify-client PyPI package
• See the Apify API reference for full details

Frequently Asked Questions

Q: How does it work? A: Hacker News Scraper collects data directly from Hacker News. Choose feed mode to browse curated feeds, or search mode to find specific content. No coding required.

Q: How fast is it? A: Feed mode fetches items in parallel. A typical run of 100 stories completes in under 10 seconds. Adding comments or author info increases run time proportionally.

Q: Can I schedule regular runs? A: Yes! You can set up automated schedules on Apify to run the scraper daily, weekly, or at any custom interval.

Q: What about rate limits? A: The scraper has built-in rate limiting and error handling to ensure reliable data collection.

Q: What if I need help? A: Contact us through the Apify platform. We're here to help you get the most out of this tool.

Integrate Hacker News Scraper with any app and automate your workflow

Last but not least, Hacker News Scraper can be connected with almost any cloud service or web app thanks to integrations on the Apify platform.

These include:

• Make
• Zapier
• Slack
• Airbyte
• GitHub
• Google Drive
• and much more.

Alternatively, you can use webhooks to carry out an action whenever an event occurs, e.g. get a notification whenever Hacker News Scraper successfully finishes a run.

🔗 Recommended Actors

Looking for more data collection tools? Check out these related actors:

Actor	Description	Link
PR Newswire Scraper	Collect press releases and news articles from PR Newswire	https://apify.com/parseforge/pr-newswire-scraper
Hugging Face Model Scraper	Extract AI model data and metadata from Hugging Face	https://apify.com/parseforge/hugging-face-model-scraper
Smart Apify Actor Scraper	Collect actor data and quality metrics from Apify Store	https://apify.com/parseforge/smart-apify-actor-scraper
HTML to JSON Smart Parser	Convert any webpage HTML into structured JSON data	https://apify.com/parseforge/html-to-json-smart-parser
Pitchbook Investors Scraper	Research investors and funding data from Pitchbook	https://apify.com/parseforge/pitchbook-investors-scraper

Pro Tip: 💡 Browse our complete collection of data collection actors to find the perfect tool for your business needs.

Need Help? Our support team is here to help you get the most out of this tool.

---

⚠️ Disclaimer: This Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by Y Combinator or Hacker News. All trademarks mentioned are the property of their respective owners.