Hacker News Scraper — Stories, Comments & Search for RAG

Fetch Hacker News stories, comment threads, and search results as clean structured data with markdown-formatted comments. Search by keyword, grab front page stories, or get full comment trees with depth tracking.

Perfect for RAG pipelines, trend monitoring, startup research, and AI agents.

Features

• Keyword search — search all HN stories and comments via the Algolia API
• Front page — get current top stories with points, authors, and metadata
• Full comment trees — recursive comment fetching with depth tracking and parent IDs
• User profiles — fetch user karma, about, and submission history
• Clean markdown — all HTML stripped, comments converted to readable markdown
• Dual-mode — run as a normal Apify Actor OR connect as an MCP server for Claude, Cursor, and other AI agents
• No browser required — uses free public APIs (Algolia + Firebase), no proxies needed

Use Cases

• RAG pipelines — feed HN discussions into vector databases for Q&A over tech community knowledge
• Trend monitoring — track trending topics, startups, and technologies on HN
• Startup research — search for mentions of companies, products, or technologies
• AI agents — let Claude/Cursor search HN via MCP tools
• Sentiment analysis — analyze community reactions to product launches and news

Input (Normal Actor Mode)

Field	Type	Description
mode	enum	search, front_page, story, or user
query	string	Search keyword (mode=search)
storyId	integer	HN story ID (mode=story)
username	string	HN username (mode=user)
maxStories	integer	Max stories to return (default 20)
includeComments	boolean	Fetch comment threads (default true)
maxCommentsPerStory	integer	Max comments per story (default 30)

Output

{
  "id": 38309611,
  "title": "OpenAI's board has fired Sam Altman",
  "url": "https://openai.com/blog/...",
  "hnUrl": "https://news.ycombinator.com/item?id=38309611",
  "author": "davidbarker",
  "points": 5710,
  "numComments": 2530,
  "createdAt": "2023-11-17T20:28:50Z",
  "comments": [
    {
      "id": 15,
      "author": "sama",
      "textMarkdown": "Clean markdown comment text...",
      "parent": 38309611,
      "depth": 0
    }
  ]
}

MCP Tools

When connected as an MCP server, the following tools are available:

Tool	Description
search_hn	Search HN stories by keyword. Returns title, points, author, URL.
get_story	Fetch a full story with optional comment thread.
get_front_page	Get current top stories from the HN front page.

Pricing

Pay-per-event. First results are free; subsequent events billed at:

Event	Price
Story fetched	$0.002
Comment thread fetched	$0.001
MCP tool call	$0.005
Actor start	$0.00005

Volume discounts available at higher tiers.