# Hacker News Scraper — Search, Front Page, Users & Comments (`hipersoft/hackernews-scraper`) Actor

Scrape Hacker News: keyword search across all stories & comments, live Top/New/Best/Ask/Show/Job front pages, user profiles and full comment threads. Filter by points, date and comment count. One clean row per item as JSON, CSV or Excel.

- **URL**: https://apify.com/hipersoft/hackernews-scraper.md
- **Developed by:** [hiper soft](https://apify.com/hipersoft) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.0015 / item scraped

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Hacker News Scraper — Search, Front Page, Users & Comments

Turn **Hacker News** into clean, structured data. Run a **keyword search** across every story and comment, pull the live **Top / New / Best / Ask HN / Show HN / Jobs** front pages, look up **user profiles**, or grab a **story with its full comment thread** — and get one tidy row per item. Export to **JSON, CSV or Excel**, or pull straight from the API.

Perfect for tech trend monitoring, startup and product research, developer sentiment analysis, building datasets for LLMs, tracking mentions of your company or competitors, lead lists from "Who is hiring?" threads, and feeding Hacker News data into your own app, dashboard or workflow.

***

### What you get

One flat, ready-to-use record per item — story, comment, job or user:

| Field | Description |
|---|---|
| `id` | Hacker News item ID |
| `type` | `story`, `comment`, `job`, `poll` or `user` |
| `title` | Story / job title |
| `text` | Body text (Ask HN posts, comments, job posts) |
| `url` | The story's outbound link |
| `hnUrl` | Direct link to the item on Hacker News |
| `author` | Username of the poster |
| `points` | Score / upvotes |
| `numComments` | Number of comments |
| `createdAt` / `time` | When it was posted (ISO 8601 + unix) |
| `parentId` / `storyId` | Parent comment / owning story (for comments) |
| `depth` | Reply depth within a thread (for comments) |
| `karma` / `about` / `submittedCount` | Profile stats (for users) |
| `source` | Which feed / mode the row came from |
| `collectedAt` | When the row was scraped |

***

### How to use

#### 1. Search by keyword

Add one or more keywords and the scraper searches across all of Hacker News. Narrow it down with type, score, comment count and date filters.

```json
{
  "searchQueries": ["openai", "rust"],
  "tags": "story",
  "minPoints": 100,
  "sortByDate": false,
  "maxItems": 200
}
```

- **Search: item type** — Stories, Comments, Ask HN, Show HN, Polls, or Stories + comments.
- **Search: min points / min comments** — only keep items above a threshold.
- **Search: from / to date** — restrict to a date range (`YYYY-MM-DD`).
- **Search: newest first** — sort chronologically instead of by relevance.

#### 2. Front-page feeds

Scrape the live front pages — pick any combination:

```json
{
  "feeds": ["top", "ask", "show"],
  "maxItems": 100
}
```

Top stories, New stories, Best stories, Ask HN, Show HN and Jobs are all supported.

#### 3. User profiles

Look up any Hacker News user — karma, account age, about text and submission count:

```json
{
  "usernames": ["pg", "dang"]
}
```

#### 4. Specific stories & full comment threads

Paste story URLs or IDs and optionally pull every comment:

```json
{
  "itemUrls": ["https://news.ycombinator.com/item?id=1"],
  "includeComments": true,
  "maxCommentsPerItem": 300,
  "maxCommentDepth": 10
}
```

Turn on **Include comments** with any front-page feed or specific item to add each comment as its own row, with reply depth preserved.

***

### Example output

```json
{
  "id": 1,
  "type": "story",
  "title": "Y Combinator",
  "url": "http://ycombinator.com",
  "hnUrl": "https://news.ycombinator.com/item?id=1",
  "author": "pg",
  "points": 57,
  "numComments": 3,
  "createdAt": "2006-10-09T18:21:51.000Z",
  "source": "feed:top",
  "collectedAt": "2026-09-20T01:30:00.000Z"
}
```

***

### Common use cases

- **Tech trend & topic monitoring** — track how often a technology, framework or company is discussed and how sentiment shifts over time.
- **Startup & product research** — mine Show HN launches, Ask HN threads and comment discussions.
- **Hiring intelligence** — extract companies and roles from the monthly "Who is hiring?" threads.
- **LLM & research datasets** — build clean corpora of stories and comments with points, authors and timestamps.
- **Brand & competitor tracking** — get alerted to mentions and read the full discussion around them.
- **Newsletters & dashboards** — pipe the daily Top or Best stories straight into your own product.

***

### Frequently asked questions

**Do I need any login, cookies or tokens?** No. Just set your inputs and run.

**How many results can I get?** Set **Max items** to whatever you need (0 = no limit). Keyword search paginates through all matches; front-page feeds cover up to 500 stories each.

**Can I get every comment on a story?** Yes — enable **Include comments** and set the per-item cap and depth. Comments come back as individual rows with `parentId`, `storyId` and `depth`.

**What format is the data?** A flat table you can export as JSON, CSV, or Excel, or fetch from the Apify API.

**Is the data live?** Yes — front-page feeds and items reflect Hacker News in real time; search covers the full history.

***

### Related scrapers

Looking for more data sources? Check out our other scrapers on Apify Store for news, tech communities, job boards and developer platforms — all with the same clean, one-row-per-item output.

# Actor input Schema

## `searchQueries` (type: `array`):

Keywords to search across Hacker News stories and comments — e.g. 'openai', 'rust', 'yc s24'. Each keyword runs its own search and the results are combined. Leave empty if you only want front-page feeds, users or specific items below.

## `feeds` (type: `array`):

Scrape live Hacker News front pages. Pick any combination of Top, New, Best, Ask HN, Show HN and Jobs.

## `tags` (type: `string`):

For keyword search — restrict to stories, comments, Ask HN, Show HN, polls, or both stories and comments.

## `sortByDate` (type: `boolean`):

Sort keyword-search results by date (newest first) instead of by relevance.

## `minPoints` (type: `integer`):

Only return search results with at least this many points (0 = no minimum).

## `minComments` (type: `integer`):

Only return search results with at least this many comments (0 = no minimum).

## `dateFrom` (type: `string`):

Only return search results created on or after this date (YYYY-MM-DD). Leave empty for no lower bound.

## `dateTo` (type: `string`):

Only return search results created on or before this date (YYYY-MM-DD). Leave empty for no upper bound.

## `usernames` (type: `array`):

Hacker News usernames (or profile URLs) to fetch — returns karma, account age, about text and submission count for each.

## `itemUrls` (type: `array`):

Specific Hacker News items to fetch — paste story URLs (https://news.ycombinator.com/item?id=…) or bare item IDs. Enable 'Include comments' to also pull each item's full comment thread.

## `includeComments` (type: `boolean`):

For front-page stories and the specific items above, also scrape the full comment thread (each comment is one extra row).

## `maxCommentsPerItem` (type: `integer`):

Cap on how many comments to collect per story when 'Include comments' is on.

## `maxCommentDepth` (type: `integer`):

How deep to follow reply chains when collecting comments (1 = top-level replies only).

## `maxItems` (type: `integer`):

Maximum number of rows to collect across everything (0 = no limit).

## Actor input object example

```json
{
  "searchQueries": [
    "openai"
  ],
  "feeds": [],
  "tags": "story",
  "sortByDate": false,
  "minPoints": 0,
  "minComments": 0,
  "includeComments": false,
  "maxCommentsPerItem": 200,
  "maxCommentDepth": 10,
  "maxItems": 100
}
```

# Actor output Schema

## `results` (type: `string`):

The results as dataset items.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "openai"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("hipersoft/hackernews-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "searchQueries": ["openai"] }

# Run the Actor and wait for it to finish
run = client.actor("hipersoft/hackernews-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "openai"
  ]
}' |
apify call hipersoft/hackernews-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,hipersoft/hackernews-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/bSrdxGc8buTnimfQD/builds/tJ0bx4SNl1WZjdcgx/openapi.json