# Hacker News Story, Comment & Karma Scraper (`dataspryva/hacker-news-story-comment-and-karma-scraper`) Actor

Pull Hacker News front-page, newest, best, Ask HN, Show HN, or job stories, look up specific items or user profiles, or run a full-text search across HN history. Filter by score, domain, date range, and comment count or author.

- **URL**: https://apify.com/dataspryva/hacker-news-story-comment-and-karma-scraper.md
- **Developed by:** [Dataspryva](https://apify.com/dataspryva) (community)
- **Categories:** Developer tools, News, Social media
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.49 / 1,000 results

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Hacker News Archive: Stories, Comments & Hiring

Hacker News is a living archive of technology discussion: front-page links, Ask HN and Show HN threads, job ads, polls, user karma, and years of comments. This actor collects all of it from the public Hacker News surfaces — live feeds, keyword search, historical front pages, individual items, user profiles, and the monthly “Who is hiring?” threads. No login, cookies, API key, or proxy setup is needed.

### What it does

- **11 modes** — six live feeds, item lookup, user profiles, keyword search, historical front pages, and monthly hiring threads.
- **Complete comment trees** — flat rows or nested `comments`, with no duplicated replies and a correct `maxItems` cap.
- **Deep search** — relevance or newest-first ordering, tags, author, story, date range, points, and comment filters.
- **Search past 1,000 results** — adaptive date-window splitting keeps collecting when a query is larger than the search API’s per-query ceiling.
- **Typed records** — stories, comments, jobs, polls with vote options, poll options, user profiles, and parsed job postings.
- **Historical front pages** — read any day back to 2007 with its original ranking, score, author, and comment count.

### Services

The actor reads three public Hacker News services:

| Service | What it provides |
|---|---|
| Firebase Hacker News API | Live feeds, individual items, user profiles, scores, comment counts, and exact deleted/dead flags |
| Algolia Hacker News Search | Keyword search, date ordering, tag and author filters, and fast comment-tree reads |
| Hacker News HTML | Historical `/front?day=` pages and their original story rankings |

All three are reachable without credentials.

### How it works

Pick a mode and the actor chooses the right source for it:

- **Feeds** read the complete live list for the chosen section, then fetch item details and apply your filters. It keeps walking until the feed ends or `maxItems` is reached, so filtered-out stories do not silently shorten the result.
- **Search** sends your query, tags, author, story, and date range to Hacker News search. With `exhaustiveSearch` enabled it splits the date range whenever a window would exceed 1,000 hits, then merges the windows without duplicates.
- **Comments** use the fast threaded read first. When a thread is missing live replies, the actor checks the thread’s comment search and merges the gap. Asking for dead or deleted content switches to the exact live item walk.
- **Historical front pages** read page after page of the chosen day, then deduplicate stories that appear twice.
- **Who is hiring?** finds the requested number of monthly threads, reads their top-level posts, and extracts the common hiring fields without inventing missing details.

Every mode is filter-aware: a story is checked before its comments are fetched, so filtered-out stories cost nothing extra.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `mode` | string | `topStories` | `topStories` / `newStories` / `bestStories` / `askStories` / `showStories` / `jobStories` / `item` / `user` / `search` / `frontPage` / `whoIsHiring` |
| `itemIds` | array | `[]` | Item IDs for `mode=item` |
| `usernames` | array | `[]` | Usernames for `mode=user` |
| `searchQueries` | array | `[]` | Keywords for `mode=search`; several queries are merged |
| `searchSort` | string | `relevance` | `relevance` or `date` |
| `contentTypes` | array | `["story"]` | `story`, `comment`, `poll`, `pollopt`, `job`, `show_hn`, `ask_hn`, `front_page` |
| `author` | string | – | Restrict search to one username |
| `storyId` | string | – | Restrict search to one story |
| `exhaustiveSearch` | boolean | `false` | Date-slice past the 1,000-hit search ceiling |
| `dateRangeFrom` | string | – | Earliest date (`YYYY-MM-DD`, UTC) |
| `dateRangeTo` | string | – | Latest date (`YYYY-MM-DD`, UTC) |
| `minScore` | integer | – | Minimum points for stories, jobs, and polls |
| `minCommentCount` | integer | – | Minimum comment count for stories and polls |
| `domainAllowlist` | array | `[]` | Keep only these link-host fragments |
| `domainBlocklist` | array | `[]` | Drop these link-host fragments |
| `excludeDeadOrDeleted` | boolean | `true` | Drop dead/deleted items |
| `commentAuthorFilter` | array | `[]` | Keep only comments by these authors |
| `commentSort` | string | `rank` | `rank` or `newest` sibling order |
| `maxComments` | integer | `0` | Comments collected per story; 0 skips comments |
| `maxDepth` | integer | `5` | Maximum reply depth; roots are depth 0 |
| `enableCommentHierarchy` | boolean | `false` | Nested `comments` instead of flat comment rows |
| `includeSubmissions` | boolean | `false` | Also emit a user’s submissions |
| `submissionTypes` | string | `all` | `all`, `story`, or `comment` |
| `frontPageDate` | string | – | Day for `mode=frontPage`; empty means yesterday |
| `hiringMonths` | integer | `1` | Recent monthly hiring threads to parse |
| `hiringThreadIds` | array | `[]` | Specific hiring threads, overriding the month count |
| `hiringKeywords` | array | `[]` | Keep matching hiring posts |
| `remoteOnly` | boolean | `false` | Keep hiring posts that mention remote work |
| `maxItems` | integer | `50` | Total records emitted; nested comments count too |
| `includeAuthorProfiles` | boolean | `false` | Add author karma and account creation date |
| `startUrls` | array | `[]` | Item, user, front-page, or hiring-thread links |

Empty fields are omitted from the output — there are no nulls or blank strings.

#### Example: search with filters

```json
{
  "mode": "search",
  "searchQueries": ["rust async runtime", "tokio"],
  "searchSort": "date",
  "contentTypes": ["story", "comment"],
  "dateRangeFrom": "2025-01-01",
  "minScore": 50,
  "maxItems": 200
}
```

#### Example: full comment tree

```json
{
  "mode": "item",
  "itemIds": ["49922569"],
  "maxComments": 300,
  "maxDepth": 10,
  "enableCommentHierarchy": true,
  "commentSort": "newest"
}
```

#### Example: a historical front page

```json
{
  "mode": "frontPage",
  "frontPageDate": "2024-01-01",
  "maxItems": 500
}
```

#### Example: this month’s hiring posts

```json
{
  "mode": "whoIsHiring",
  "hiringMonths": 1,
  "hiringKeywords": ["python", "backend"],
  "remoteOnly": true,
  "maxItems": 100
}
```

### Output per story, job, or poll

- `id`, `hnUrl`, `type` (`story`, `job`, or `poll`), `title`
- `url`, `domain` for link posts
- `score`, `author`, `numComments`
- `createdAt` (ISO-8601 UTC), `createdAtEpoch`, `ageHours`
- `text` for Ask HN, Show HN, jobs, and polls
- `kids` — top-level comment IDs
- `rank` — feed, search, or front-page position
- `frontPageDate` for historical rows
- `pollOptions[]` — `id`, `text`, `votes`, `author`, `createdAt`, `hnUrl`
- `fetchedCommentCount` when comments were read
- `comments[]` — nested replies when hierarchy is enabled
- `authorKarma`, `authorCreatedAt` when author enrichment is enabled
- `dead` / `deleted` only when true
- `recordType: "story"`, `scrapedAt`

### Output per comment

- `id`, `hnUrl`, `type`, `author`, `text`
- `storyId`, `parentId`, `depth`
- `createdAt`, `createdAtEpoch`, `ageHours`
- `kids` — reply IDs
- `storyTitle`, `storyUrl` when a search hit supplies them
- `replies[]` in nested mode only
- `dead` / `deleted` only when true
- `recordType: "comment"`, `scrapedAt`

### Output per user

- `id`, `profileUrl`
- `karma`, `createdAt`, `createdAtEpoch`, `about`
- `submittedCount`, `submitted` (first 50 recent IDs)
- `recordType: "user"`, `scrapedAt`

With `includeSubmissions`, normal story and comment rows follow the profile.

### Output per poll option

- `id`, `hnUrl`, `type: "pollopt"`, `text`, `votes`, `pollId`
- `author`, `createdAt`, `createdAtEpoch`, `ageHours`
- `recordType: "pollOption"`, `scrapedAt`

### Output per hiring post

- `id`, `hnUrl`, `threadId`, `threadTitle`, `threadMonth`
- `author`, `company`, `role`, `location`, `remote`, `salary`
- `techStack`, `visa`, `email`, `applyUrl`
- `text`, `postedAt`
- `recordType: "jobPosting"`, `scrapedAt`

Parsed hiring fields are omitted when the post does not state them.

### Usage cost

There is no paid data source, proxy, or credential in this actor. Usage is ordinary Apify compute plus the records you choose to collect. The cheapest useful run is the default prefill: 20 top stories with comments and author enrichment off. Cost grows with `maxItems`, `maxComments`, `maxDepth`, multiple search queries, `exhaustiveSearch`, and `includeAuthorProfiles` because each adds fetches. Small `maxItems` values keep runs short and inexpensive.

### FAQ

**Do I need an account, cookie, or API key?** No. Every mode reads public Hacker News data.

**Do I need a proxy?** No. The actor connects directly and needs no proxy configuration.

**Why is a field missing from some records?** Hacker News does not store that field for every item. For example, self posts have text but no link URL, and many users have no “about” text. Missing fields are omitted instead of filled with a placeholder.

**Why do comment scores never appear?** Hacker News does not publish per-comment scores through its feeds, search, or HTML. Stories, polls, jobs, and poll options do have scores.

**Why did a search stop at 1,000 results?** Hacker News search returns at most 1,000 hits per query window. Turn on `exhaustiveSearch` to split the date range into smaller windows and collect deeper results.

**Why is relevance order different when `exhaustiveSearch` is on?** The deep search collects several date windows. Each window is ranked by relevance, so the final list follows window order rather than one global relevance ordering. Use `searchSort: "date"` for a stable chronological order.

**How fresh is the data?** Feeds and item lookups are live. Search and comment search can lag the live site by a short period, so brand-new comments or score changes may appear slightly later.

**What is the difference between `maxItems` and `maxComments`?** `maxItems` caps every emitted record, including nested comments. `maxComments` caps how many comments are read for each story before depth filtering.

**Can I read a historical best, Ask HN, or Show HN list?** No. Hacker News archives only the historical front page by date. Live Ask HN, Show HN, best, and job lists are available through their normal modes.

**Is this affiliated with Hacker News or Y Combinator?** No. This is an independent actor that reads Hacker News’ public data.

### Notes

- All timestamps are UTC and ISO-8601, with a matching Unix epoch.
- Records are snapshots: scores, comment counts, and comment trees can change after a run.
- Flat comment output contains one row per comment and never repeats a reply inside another row.
- Nested comment output is one story row; every reply carries its own type and scrape timestamp.
- `item` returns a comment row for a comment ID, a story-style row for a story, job, or poll, and a poll-option row for a poll-option ID.
- `startUrls` accepts item, user, historical front-page, and hiring-thread links alongside the mode-specific lists.
- Historical front pages are deduplicated when a story appears twice on the same day.

### Limitations

- **No comment scores or flags.** Hacker News does not expose per-comment scores, moderation flags, or edit history publicly.
- **Search ceiling.** Each Hacker News search window returns at most 1,000 hits. `exhaustiveSearch` works around this by date slicing, but a single second with more than 1,000 hits cannot be split further.
- **Historical lists.** Only the front page is archived by day. Historical best, Ask HN, Show HN, and jobs pages are not available.
- **Feed caps.** Hacker News itself limits the live feeds: top and new stories usually expose 500 IDs, best about 200, Ask HN 20, Show HN up to about 150, and jobs about 30. The actor reports when a feed is exhausted.
- **Comment-tree lag.** The fast tree can trail the live thread by a short period, and deleted branches may be absent. The actor merges a comment-search fallback when live replies are missing; turning off `excludeDeadOrDeleted` uses the exact slower walk.
- **Hiring parsing is heuristic.** Company, role, location, salary, tech stack, visa, email, and apply link are extracted from free-form posts. Fields the post does not state are omitted; no value is invented.
- **Very large user histories.** A user can have tens of thousands of submissions. `maxItems` and `submissionTypes` keep the run bounded.
- **Relevance is not deterministic.** Search relevance can shift between runs as the index changes; date order is stable.

# Actor input Schema

## `mode` (type: `string`):

What to scrape. Feeds walk the live Hacker News lists. `item` looks up specific IDs, `user` loads profiles, `search` queries Hacker News history, `frontPage` reads a historical day, and `whoIsHiring` parses monthly hiring threads.

## `itemIds` (type: `array`):

Hacker News item IDs to look up, as numbers or numeric strings. Stories, comments, jobs, polls, and poll options are all accepted.

## `usernames` (type: `array`):

Hacker News usernames whose profiles and, optionally, submissions you want.

## `searchQueries` (type: `array`):

One or more keyword queries. Every query runs independently and the results are merged without duplicates. Leave empty to browse by tags, author, or story instead.

## `searchSort` (type: `string`):

`relevance` ranks by keyword match. `date` returns newest first and is more reproducible.

## `contentTypes` (type: `array`):

Which Hacker News record types search should return. Selecting several types returns any of them. `front_page` covers only the current front page.

## `author` (type: `string`):

Only return content written by this Hacker News username.

## `storyId` (type: `string`):

Only return content attached to this story ID. Useful with `contentTypes` set to `comment` to search inside one thread.

## `exhaustiveSearch` (type: `boolean`):

Hacker News search exposes at most 1,000 hits per query. When enabled, the actor splits the requested date range into smaller windows so deeper result sets can be collected. Relevance order then follows each date window rather than one global ranking.

## `dateRangeFrom` (type: `string`):

Drop items posted before this date (UTC, `YYYY-MM-DD`). Empty means no lower bound.

## `dateRangeTo` (type: `string`):

Drop items posted after this date (UTC, `YYYY-MM-DD`). Empty means no upper bound.

## `minScore` (type: `integer`):

Drop stories, jobs, and polls below this point score. Comments are unaffected because Hacker News does not publish comment scores.

## `minCommentCount` (type: `integer`):

Drop stories and polls with fewer than this many comments.

## `domainAllowlist` (type: `array`):

Keep only stories whose link host contains one of these text fragments. For example, `github.com` or `arxiv.org`.

## `domainBlocklist` (type: `array`):

Drop stories whose link host contains one of these text fragments. For example, `twitter.com` or `x.com`.

## `excludeDeadOrDeleted` (type: `boolean`):

Drop items marked dead or deleted by Hacker News. Turning this off uses the slower exact comment path so those records can be included.

## `commentAuthorFilter` (type: `array`):

Only keep comments written by these usernames. Matching is case-insensitive and accepts partial names.

## `commentSort` (type: `string`):

`rank` keeps Hacker News display order. `newest` orders sibling replies newest first.

## `maxComments` (type: `integer`):

How many comments to collect for each story. 0 skips comment collection. This is applied after a story passes your filters.

## `maxDepth` (type: `integer`):

Deepest reply level to keep when comments are collected. Root comments are depth 0.

## `enableCommentHierarchy` (type: `boolean`):

When true, each story carries a nested `comments` tree. When false, comments are separate rows and never contain duplicate reply rows.

## `includeSubmissions` (type: `boolean`):

For mode `user`, also emit the user's recent stories and comments as normal records after the profile.

## `submissionTypes` (type: `string`):

Which submissions to emit when `includeSubmissions` is enabled.

## `frontPageDate` (type: `string`):

Historical day to read in mode `frontPage`, as `YYYY-MM-DD`. Empty uses yesterday (UTC).

## `hiringMonths` (type: `integer`):

How many recent monthly Who is hiring? threads to parse. Ignored when specific thread IDs are supplied.

## `hiringThreadIds` (type: `array`):

Specific Who is hiring? thread IDs to parse. Overrides the automatic monthly selection.

## `hiringKeywords` (type: `array`):

Keep only job posts whose text, company, role, location, or tech stack contains one of these terms.

## `remoteOnly` (type: `boolean`):

Keep only hiring posts that explicitly mention remote work.

## `maxItems` (type: `integer`):

Hard cap on emitted records across all types. Nested comments also count toward this cap.

## `includeAuthorProfiles` (type: `boolean`):

Add `authorKarma` and `authorCreatedAt` to stories and comments. This costs one extra lookup per unique author.

## `startUrls` (type: `array`):

Hacker News links to read directly: item, user, historical front-page, and hiring-thread URLs. IDs and URLs are merged with the mode-specific lists.

## Actor input object example

```json
{
  "mode": "topStories",
  "itemIds": [],
  "usernames": [],
  "searchQueries": [],
  "searchSort": "relevance",
  "contentTypes": [
    "story"
  ],
  "exhaustiveSearch": false,
  "domainAllowlist": [],
  "domainBlocklist": [],
  "excludeDeadOrDeleted": true,
  "commentAuthorFilter": [],
  "commentSort": "rank",
  "maxComments": 0,
  "maxDepth": 5,
  "enableCommentHierarchy": false,
  "includeSubmissions": false,
  "submissionTypes": "all",
  "hiringMonths": 1,
  "hiringThreadIds": [],
  "hiringKeywords": [],
  "remoteOnly": false,
  "maxItems": 20,
  "includeAuthorProfiles": false,
  "startUrls": []
}
```

# Actor output Schema

## `items` (type: `string`):

Dataset containing every scraped record. `recordType` identifies stories, comments, users, poll options, and hiring posts. Story rows include id, title, url, domain, score, author, numComments, rank, timestamps, text, pollOptions, and optional nested comments. Comment rows include id, storyId, parentId, depth, author, text, and story context. User rows include karma, account age, about text, and submission counts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "topStories",
    "itemIds": [],
    "usernames": [],
    "searchQueries": [],
    "searchSort": "relevance",
    "contentTypes": [
        "story"
    ],
    "exhaustiveSearch": false,
    "domainAllowlist": [],
    "domainBlocklist": [],
    "excludeDeadOrDeleted": true,
    "commentAuthorFilter": [],
    "commentSort": "rank",
    "maxComments": 0,
    "maxDepth": 5,
    "enableCommentHierarchy": false,
    "includeSubmissions": false,
    "submissionTypes": "all",
    "hiringMonths": 1,
    "hiringThreadIds": [],
    "hiringKeywords": [],
    "remoteOnly": false,
    "maxItems": 20,
    "includeAuthorProfiles": false,
    "startUrls": []
};

// Run the Actor and wait for it to finish
const run = await client.actor("dataspryva/hacker-news-story-comment-and-karma-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "topStories",
    "itemIds": [],
    "usernames": [],
    "searchQueries": [],
    "searchSort": "relevance",
    "contentTypes": ["story"],
    "exhaustiveSearch": False,
    "domainAllowlist": [],
    "domainBlocklist": [],
    "excludeDeadOrDeleted": True,
    "commentAuthorFilter": [],
    "commentSort": "rank",
    "maxComments": 0,
    "maxDepth": 5,
    "enableCommentHierarchy": False,
    "includeSubmissions": False,
    "submissionTypes": "all",
    "hiringMonths": 1,
    "hiringThreadIds": [],
    "hiringKeywords": [],
    "remoteOnly": False,
    "maxItems": 20,
    "includeAuthorProfiles": False,
    "startUrls": [],
}

# Run the Actor and wait for it to finish
run = client.actor("dataspryva/hacker-news-story-comment-and-karma-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "topStories",
  "itemIds": [],
  "usernames": [],
  "searchQueries": [],
  "searchSort": "relevance",
  "contentTypes": [
    "story"
  ],
  "exhaustiveSearch": false,
  "domainAllowlist": [],
  "domainBlocklist": [],
  "excludeDeadOrDeleted": true,
  "commentAuthorFilter": [],
  "commentSort": "rank",
  "maxComments": 0,
  "maxDepth": 5,
  "enableCommentHierarchy": false,
  "includeSubmissions": false,
  "submissionTypes": "all",
  "hiringMonths": 1,
  "hiringThreadIds": [],
  "hiringKeywords": [],
  "remoteOnly": false,
  "maxItems": 20,
  "includeAuthorProfiles": false,
  "startUrls": []
}' |
apify call dataspryva/hacker-news-story-comment-and-karma-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,dataspryva/hacker-news-story-comment-and-karma-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/shC9wAKWFYbt1zZYs/builds/ajANmalEbfWPogs9n/openapi.json