Hacker News Scraper Pro - Web Scraping, Tech News Extraction, Startup Intelligence

Web scraping tool for extracting Hacker News stories, Ask HN threads, job postings, and user profiles. No proxy required. Data extraction includes points, comment counts, authors, URLs, and submission metadata for tech trend analysis, startup discovery, and developer sentiment analysis.

Use Cases

• Tech News Monitoring: Track frontpage stories and emerging trends in real-time for news aggregation and competitive intelligence. Identify which technologies and products are generating the most discussion.

• Startup Deal Flow: Identify new product launches on Show HN, YC companies posting, and early-stage startup traction signals. Monitor the frontpage for investment opportunities.

• Developer Sentiment Analysis: Analyze Ask HN threads and comments to gauge community opinion on technologies, products, and industry trends. Understand what developers think about specific tools or approaches.

• VC Intelligence: Monitor Hacker News for investment opportunities. Track poster karma as a proxy for founder credibility and activity. Identify repeat founders and active community members.

• Job Market Analysis: Extract job postings to understand hiring trends, salary discussions, and in-demand skills in the tech industry. Track which companies are actively recruiting.

MCP Tools Reference

This actor exposes the following tools via the Apify MCP protocol. AI agents can invoke these tools directly to integrate with workflows:

Tool	Input	Output
scrapeFrontpage	maxItems, includeComments	Top frontpage stories with metadata
scrapeCategory	mode (frontpage|newest|ask|show|jobs), maxItems	Category-specific stories or posts
scrapeStory	HN story URL, includeComments	Full story with all comments tree
getUserProfile	HN username	User karma, join date, about text, submission history

Input Parameters

Parameter	Type	Description	Default
startUrls	array	Hacker News URLs to scrape (stories, lists, specific pages)	-
mode	string	Section: frontpage, newest, ask, show, jobs	frontpage
users	array	Username list to extract profile data	-
maxItems	integer	Maximum items to extract (1-1000)	100
includeComments	boolean	Extract full comment tree (increases runtime significantly)	false
proxyType	string	none (free), datacenter, residential	none

Pricing

This actor uses Apify's Pay-Per-Event (PPE) pricing model. You are charged per successful result returned. No charge for queries that return zero results.

• No Proxy Required: Hacker News allows direct scraping without proxy infrastructure, making this one of the most cost-effective data sources available.

• Datacenter Proxy (if needed for high concurrency): $0.60–$7 per IP address

• Residential Proxy (rarely needed): $7–$8 per GB. HN rarely blocks residential IPs.

Example Output

{
  "results": [
    {
      "type": "story",
      "id": "38618262",
      "title": "Ask HN: What's Your Emergency Fund Strategy?",
      "url": "https://example.com/financial-strategy",
      "points": 847,
      "author": "throwaway59724",
      "commentCount": 492,
      "publishedAt": "2026-04-11T08:15:00.000Z",
      "rank": 1,
      "cleanMarkdown": "## Discussion\n\nI've been thinking about emergency funds..."
    },
    {
      "type": "job",
      "id": "38618100",
      "title": "Y Combinator is hiring a Senior Software Engineer",
      "company": "Y Combinator",
      "url": "https://ycombinator.com/jobs",
      "points": 45,
      "author": "ycombinator_hr",
      "commentCount": 12,
      "publishedAt": "2026-04-11T07:30:00.000Z"
    }
  ]
}

Quick Start

# Run via Apify CLI
apify run -a red.cars/hackernews-scraper-pro

# Or configure input and run at:
# https://apify.com/red.cars/hackernews-scraper-pro

How It Works

The actor connects directly to Hacker News (news.ycombinator.com) without requiring proxy infrastructure. It parses the Firebase API backend that HN uses for its frontend, extracting structured data including story metadata, user profiles, and comment trees. No API key required.

The extraction process:

1. Queries HN's Firebase API for the requested category (frontpage, newest, ask, show, jobs)
2. Retrieves story metadata including points, author, timestamp, and URLs
3. Optionally fetches the full comment tree for stories
4. Returns structured JSON data with all story and user information

Data Fields Returned

Story/Ask/Show Results:

• type — story, ask, show, or job
• id, title, url — Story identifiers and link
• points — Score/votes from community
• author — Poster's HN username
• commentCount — Number of comments
• publishedAt — Submission timestamp
• rank — Position on the list
• cleanMarkdown — Rendered content in markdown format (if requested)

User Profile Results:

• username — HN username
• karma — User's karma score (proxy for reputation)
• joinedAt — Account creation date
• about — User bio/about text (may be empty)
• submissionCount — Total submissions to HN

Known Limitations

• HN's Firebase API may return rate limit errors with very high concurrency; implement exponential backoff
• Job postings expire quickly and are removed from HN; historical job data is limited
• Private/flagged comments and submissions are not accessible
• Some external URLs in stories may be dead or redirected since posting
• Deleted accounts show minimal profile information

Legal

Data source is news.ycombinator.com. This actor must be used in compliance with Y Combinator's Terms of Service. Do not use for unauthorized data collection or privacy-violating activities.