Hacker News Top Stories Scraper

Scrape Hacker News story feeds and get structured data — titles, URLs, points, comment counts, authors, timestamps, and optionally the text of top-level comments — all delivered to an Apify dataset ready for export or downstream processing.

---

What It Does

This actor calls the official Hacker News Firebase API (free, no authentication required) and:

1. Fetches the list of story IDs from the requested feed (top, new, best, ask, show, or job).
2. Resolves each story's details concurrently (up to 20 requests in parallel).
3. Applies optional point and comment-count filters.
4. Optionally fetches the text of the top 10 top-level comments per story.
5. Pushes each story as a structured record to the Apify default dataset.

---

Input Parameters

Parameter	Type	Default	Description
story_type	enum	top	Feed to scrape: top, new, best, ask, show, job
max_stories	integer	30	Maximum stories to return (1–500)
min_points	integer	0	Skip stories below this point threshold
min_comments	integer	0	Skip stories below this comment count
include_comment_text	boolean	false	Fetch and include top-level comment text
output_format	enum	full	full (all fields) or compact (core fields only)

Example Input

{
  "story_type": "top",
  "max_stories": 50,
  "min_points": 100,
  "min_comments": 10,
  "include_comment_text": false,
  "output_format": "full"
}

---

Output Format

Full record (output_format: "full")

{
  "rank": 1,
  "id": 41234567,
  "title": "Ask HN: What tools do you use for local AI development?",
  "url": "https://news.ycombinator.com/item?id=41234567",
  "hn_url": "https://news.ycombinator.com/item?id=41234567",
  "author": "jsmith",
  "time": 1727900400,
  "time_ago": "3 hours ago",
  "time_iso": "2026-10-02T15:00:00+00:00",
  "points": 342,
  "num_comments": 87,
  "type": "story",
  "story_text": null,
  "kids": [41234600, 41234601, 41234602]
}

Compact record (output_format: "compact")

{
  "rank": 1,
  "id": 41234567,
  "title": "Ask HN: What tools do you use for local AI development?",
  "url": "https://news.ycombinator.com/item?id=41234567",
  "hn_url": "https://news.ycombinator.com/item?id=41234567",
  "author": "jsmith",
  "time": 1727900400,
  "time_ago": "3 hours ago",
  "points": 342,
  "num_comments": 87,
  "type": "story"
}

Comment object (when include_comment_text: true)

{
  "top_comments": [
    {
      "id": 41234600,
      "author": "alice",
      "text": "I've been using Ollama with a local Llama model...",
      "time": 1727901000,
      "time_ago": "2 hours ago"
    }
  ]
}

---

Use Cases

Content Curation

Pull the day's top 30 stories with more than 200 points to build a daily digest newsletter or Slack feed. Filter by min_points to ensure only high-signal content is included.

Trend Monitoring

Schedule the actor to run hourly on the new feed and track velocity — stories that accumulate points quickly often signal emerging trends in technology, startups, or research.

Competitive Research

Monitor show and ask feeds to discover what products and ideas the HN community is building and discussing. Combine with comment text to understand sentiment.

Data Analysis & ML

Export historical snapshots to CSV or JSON for time-series analysis of topic popularity, point distributions, or community engagement patterns. The structured output is ready for pandas, BigQuery, or any data warehouse.

Job Market Intelligence

Use the job feed to track which companies are hiring on HN and what roles are in demand, without needing to scrape the full HN website.

---

Rate Limiting & Performance

• The Hacker News Firebase API is public and unauthenticated — no API key required.
• The actor fetches item details with 20 concurrent requests by default, balancing speed with politeness.
• Transient failures are retried up to 3 times with exponential backoff before a story is skipped.
• For max_stories: 500 with include_comment_text: true, expect a runtime of roughly 2–5 minutes depending on comment counts.
• HN does not publish an explicit rate limit, but the actor is designed to be a considerate API consumer.

---

Technical Details

• Language: Python 3.12
• HTTP client: httpx (async)
• Apify SDK: apify >= 2.0.0
• Concurrency model: asyncio with a Semaphore-bounded task pool
• API: Hacker News Firebase API v0
• Docker base image: apify/actor-python:3.12

Project Structure

hn-top-stories/
├── .actor/
│   ├── actor.json          # Actor metadata and dataset view configuration
│   └── INPUT_SCHEMA.json   # Input parameter schema
├── src/
│   ├── __init__.py
│   └── main.py             # Actor entry point and all scraping logic
├── Dockerfile
├── requirements.txt
└── README.md