# Hacker News Top Stories Scraper (`emperor1978/hn-top-stories-scraper`) Actor

Extract top stories from Hacker News (news.ycombinator.com). Get titles, URLs, scores, authors, comment counts, and timestamps. Perfect for trend monitoring, content curation, and tech news analysis. Fast, reliable, no browser needed.

- **URL**: https://apify.com/emperor1978/hn-top-stories-scraper.md
- **Developed by:** [The Emperor](https://apify.com/emperor1978) (community)
- **Categories:** News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Hacker News Top Stories Scraper

Scrape Hacker News story feeds and get structured data — titles, URLs, points, comment counts, authors, timestamps, and optionally the text of top-level comments — all delivered to an Apify dataset ready for export or downstream processing.

***

### What It Does

This actor calls the [official Hacker News Firebase API](https://github.com/HackerNews/API) (free, no authentication required) and:

1. Fetches the list of story IDs from the requested feed (top, new, best, ask, show, or job).
2. Resolves each story's details concurrently (up to 20 requests in parallel).
3. Applies optional point and comment-count filters.
4. Optionally fetches the text of the top 10 top-level comments per story.
5. Pushes each story as a structured record to the Apify default dataset.

***

### Input Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `story_type` | enum | `top` | Feed to scrape: `top`, `new`, `best`, `ask`, `show`, `job` |
| `max_stories` | integer | `30` | Maximum stories to return (1–500) |
| `min_points` | integer | `0` | Skip stories below this point threshold |
| `min_comments` | integer | `0` | Skip stories below this comment count |
| `include_comment_text` | boolean | `false` | Fetch and include top-level comment text |
| `output_format` | enum | `full` | `full` (all fields) or `compact` (core fields only) |

#### Example Input

```json
{
  "story_type": "top",
  "max_stories": 50,
  "min_points": 100,
  "min_comments": 10,
  "include_comment_text": false,
  "output_format": "full"
}
```

***

### Output Format

#### Full record (output_format: "full")

```json
{
  "rank": 1,
  "id": 41234567,
  "title": "Ask HN: What tools do you use for local AI development?",
  "url": "https://news.ycombinator.com/item?id=41234567",
  "hn_url": "https://news.ycombinator.com/item?id=41234567",
  "author": "jsmith",
  "time": 1727900400,
  "time_ago": "3 hours ago",
  "time_iso": "2026-10-02T15:00:00+00:00",
  "points": 342,
  "num_comments": 87,
  "type": "story",
  "story_text": null,
  "kids": [41234600, 41234601, 41234602]
}
```

#### Compact record (output_format: "compact")

```json
{
  "rank": 1,
  "id": 41234567,
  "title": "Ask HN: What tools do you use for local AI development?",
  "url": "https://news.ycombinator.com/item?id=41234567",
  "hn_url": "https://news.ycombinator.com/item?id=41234567",
  "author": "jsmith",
  "time": 1727900400,
  "time_ago": "3 hours ago",
  "points": 342,
  "num_comments": 87,
  "type": "story"
}
```

#### Comment object (when include_comment_text: true)

```json
{
  "top_comments": [
    {
      "id": 41234600,
      "author": "alice",
      "text": "I've been using Ollama with a local Llama model...",
      "time": 1727901000,
      "time_ago": "2 hours ago"
    }
  ]
}
```

***

### Use Cases

#### Content Curation

Pull the day's top 30 stories with more than 200 points to build a daily digest newsletter or Slack feed. Filter by `min_points` to ensure only high-signal content is included.

#### Trend Monitoring

Schedule the actor to run hourly on the `new` feed and track velocity — stories that accumulate points quickly often signal emerging trends in technology, startups, or research.

#### Competitive Research

Monitor `show` and `ask` feeds to discover what products and ideas the HN community is building and discussing. Combine with comment text to understand sentiment.

#### Data Analysis & ML

Export historical snapshots to CSV or JSON for time-series analysis of topic popularity, point distributions, or community engagement patterns. The structured output is ready for pandas, BigQuery, or any data warehouse.

#### Job Market Intelligence

Use the `job` feed to track which companies are hiring on HN and what roles are in demand, without needing to scrape the full HN website.

***

### Rate Limiting & Performance

- The Hacker News Firebase API is **public and unauthenticated** — no API key required.
- The actor fetches item details with **20 concurrent requests** by default, balancing speed with politeness.
- Transient failures are retried up to **3 times** with exponential backoff before a story is skipped.
- For `max_stories: 500` with `include_comment_text: true`, expect a runtime of roughly 2–5 minutes depending on comment counts.
- HN does not publish an explicit rate limit, but the actor is designed to be a considerate API consumer.

***

### Technical Details

- **Language**: Python 3.12
- **HTTP client**: [httpx](https://www.python-httpx.org/) (async)
- **Apify SDK**: [apify](https://pypi.org/project/apify/) >= 2.0.0
- **Concurrency model**: `asyncio` with a `Semaphore`-bounded task pool
- **API**: [Hacker News Firebase API v0](https://github.com/HackerNews/API)
- **Docker base image**: `apify/actor-python:3.12`

#### Project Structure

```
hn-top-stories/
├── .actor/
│   ├── actor.json          # Actor metadata and dataset view configuration
│   └── INPUT_SCHEMA.json   # Input parameter schema
├── src/
│   ├── __init__.py
│   └── main.py             # Actor entry point and all scraping logic
├── Dockerfile
├── requirements.txt
└── README.md
```

# Actor input Schema

## `story_type` (type: `string`):

Which feed to scrape from Hacker News.

## `max_stories` (type: `integer`):

Maximum number of stories to return. The HN API returns up to 500 IDs per feed; this caps how many are fetched and processed.

## `min_points` (type: `integer`):

Exclude stories with fewer points than this value. Set to 0 to include all stories.

## `min_comments` (type: `integer`):

Exclude stories with fewer comments than this value. Set to 0 to include all stories.

## `include_comment_text` (type: `boolean`):

If enabled, the actor will also fetch the text of top-level comments for each story. This significantly increases the number of API requests.

## `output_format` (type: `string`):

Full includes all fields including metadata and comment text. Compact omits comment text and some metadata fields to reduce output size.

## Actor input object example

```json
{
  "story_type": "top",
  "max_stories": 30,
  "min_points": 0,
  "min_comments": 0,
  "include_comment_text": false,
  "output_format": "full"
}
```

# Actor output Schema

## `results` (type: `string`):

Scraped results stored in the default dataset

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("emperor1978/hn-top-stories-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("emperor1978/hn-top-stories-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call emperor1978/hn-top-stories-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,emperor1978/hn-top-stories-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/1A6iJU4iN6wnmNWks/builds/TveI7V2P9VPGgioBc/openapi.json