# Hacker News Scraper: Stories, Comments, Who's Hiring (`grit-77/hacker-news-scraper`) Actor

Hacker News scraper for story, comment and hiring-post research. Select a mode to export stories, comments, search results or parsed hiring-post rows as JSON.

- **URL**: https://apify.com/grit-77/hacker-news-scraper.md
- **Developed by:** [Grit](https://apify.com/grit-77) (community)
- **Categories:** News, Jobs, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Hacker News Scraper: Stories, Comments, Who's Hiring

Hacker News scraper returns public story, comment, search, or parsed hiring-post rows, depending on the selected mode.

Hacker News scraper uses the public source named below and writes its results to the Actor dataset.

### What you can do

| Mode | What it returns | Source |
|---|---|---|
| `topstories`, `newstories`, `beststories`, `askstories`, `showstories`, `jobstories` | The current list, in HN rank order (`rank` field) | Official Firebase API |
| `search` | Stories, comments, polls, Ask/Show HN matching a query, with date range, minimum points, minimum comments, type and author filters; relevance or newest-first | Algolia HN Search |
| `comments` | The whole comment tree of each story id or link you give, flattened to rows with `depth`, `parent_id`, `reply_count` | Algolia items endpoint (one request per tree) |
| `whoishiring` | One parsed row per job post in the latest (or any named) monthly thread, with the raw text kept next to the parsed fields | Algolia items endpoint |

### Output fields

Fields vary by result type and optional enrichment. The examples below are from `sample_output.json`.

| Output field | Type | Example value or excerpt from saved sample |
|---|---|---|
| `kind` | string | `"item"` |
| `id` | integer | `49962012` |
| `type` | string | `"story"` |
| `title` | string or null | `"Denmark Data Breach Exposes 8.8M People's Personal Data"` |
| `url` | string or null | `"https://www.cpr.dk/cpr-nyt/nyhedsarkiv/2026/okt/omfattende-uautoriseret-adgang-til-borgeres-cpr-oplysninger"` |
| `domain` | string or null | `"cpr.dk"` |
| `text` | string or null | `"oh, and a mac port is coming :)"` |
| `author` | string | `"clan"` |
| `points` | integer or null | `198` |
| `num_comments` | integer or null | `139` |
| `created_at` | string | `"2026-10-05T08:09:36Z"` |
| `created_at_i` | integer | `1791187776` |
| `parent_id` | integer or null | `8863` |
| `story_id` | integer | `49962012` |
| `story_title` | string or null | `"My YC app: Dropbox - Throw away your USB drive"` |
| `depth` | integer or null | `0` |
| `reply_count` | integer or null | `32` |
| `rank` | integer or null | `1` |
| `hn_url` | string | `"https://news.ycombinator.com/item?id=49962012"` |
| `error` | null | `null` |
| `company` | string | `"PrairieLearn (Remote US"` |
| `role` | string or null | `"Full-Stack Software Engineer"` |
| `location` | string or null | `"San Francisco, CA & New York City, NY"` |
| `remote` | boolean | `true` |
| `work_mode` | string | `"remote"` |
| `salary_text` | string or null | `"$100k-$180k"` |
| `visa` | string or null | `"(We’re not able to sponsor visas at this time.)"` |
| `visa_sponsorship` | boolean or null | `false` |
| `employment_type` | array or null | `["contract"]` |
| `apply_email` | string or null | `"eric+hn@sidekicklabs.com"` |
| `all_urls` | array | `["https://www.prairielearn.com"]` |
| `header` | string | `"PrairieLearn (Remote US) — Full-Stack Software Engineer — TypeScript / Postgres / React / AI"` |
| `header_structured` | boolean | `false` |
| `unclassified_segments` | array or null | `["TypeScript / Postgres / React / AI"]` |
| `raw_text` | string | `"PrairieLearn (Remote US) — Full-Stack Software Engineer — TypeScript / Postgres / React / AI\nPrair…` |
| `posted_at` | string | `"2026-10-01T15:02:37Z"` |
| `thread_id` | integer | `49922569` |
| `thread_title` | string | `"Ask HN: Who is hiring? (October 2026)"` |

Story, comment and search rows (`kind: "item"`, event `item`):

| Field | Meaning |
|---|---|
| `id`, `type`, `hn_url` | Item id, `story` / `comment` / `poll` / `job`, link to the HN page |
| `title`, `url`, `domain` | Story title, external link and its domain |
| `text` | Plain text of a comment, Ask HN or job body (`text_html` too if `includeHtml`) |
| `author`, `points`, `num_comments` | As shown on HN |
| `created_at`, `created_at_i` | UTC ISO time and Unix seconds |
| `parent_id`, `story_id`, `story_title` | Where a comment sits |
| `depth`, `reply_count` | Comment trees: 0 = top-level comment; `null` on the story row |
| `rank` | Position in a list mode |
| `error` | Set only on error rows, which are never charged |

Who-is-hiring rows (`kind: "job"`, event `job`):

| Field | Meaning |
|---|---|
| `company`, `role` | From the post header (`Company \| Role \| Location \| ...`) |
| `location`, `remote`, `work_mode` | `work_mode` is `remote`, `hybrid`, `onsite` or `remote_or_onsite` when the header says so |
| `salary_text` | The salary as written, for example `$180k-$250k` or `100k-160k GBP` |
| `visa`, `visa_sponsorship` | The sentence(s) that mention visa / sponsorship / relocation; `true` / `false` / `null` when unclear |
| `employment_type` | `full-time`, `part-time`, `intern`, `contract` when named in the header |
| `url`, `all_urls`, `apply_email` | First non-HN link, other links, and the e-mail the post publishes for applying |
| `header`, `header_structured`, `unclassified_segments` | The first line, whether it had the usual pipe layout, header parts the parser could not place |
| `raw_text` | Full post text, always kept |
| `author`, `posted_at`, `hn_url`, `thread_id`, `thread_title` | Source of the post |

The parsing is heuristic: HN does not force a format, so some fields can be empty or wrong. On the October 2026 thread (194 job posts returned) the fields were filled like this: company 100%, location 88%, URL 86%, role 72%, work mode 95%, salary 38%, visa mention 7%, apply e-mail 27%. Always check `raw_text` when a field matters. Off-topic top-level comments (replies, meta) are skipped and not charged.

### Input example

```json
{
  "mode": "search",
  "query": "sqlite",
  "type": "story",
  "searchSort": "date",
  "dateFrom": "2026-01-01",
  "minPoints": 100,
  "maxItems": 200
}
```

Who is hiring, remote Rust jobs from two months:

```json
{ "mode": "whoishiring", "hiringMonths": ["latest", "2026-09"], "hiringRemoteOnly": true, "hiringKeywords": ["rust"] }
```

Full comment tree of a story:

```json
{ "mode": "comments", "storyIds": ["8863", "https://news.ycombinator.com/item?id=49962012"], "maxCommentDepth": 0 }
```

### Output example

```json
{
  "kind": "job",
  "id": 49922693,
  "company": "Planlab.ai",
  "role": "Product Engineer (Design) & Platform Engineer",
  "location": "ONSITE in London, UK",
  "remote": false,
  "work_mode": "onsite",
  "salary_text": "100k-160k GBP",
  "visa_sponsorship": true,
  "url": "https://planlab.ai/jobs",
  "posted_at": "2026-10-01T15:08:44Z",
  "hn_url": "https://news.ycombinator.com/item?id=49922693"
}
```

A real 20-row sample (5 front-page stories, 4 comment rows, 3 search hits, 8 hiring posts) is in `sample_output.json`.

### Pricing (pay per event)

| Event | Price |
|---|---|
| `item` (story, comment or search row) | 0.00025 USD |
| `job` (parsed Who-is-hiring post) | 0.002 USD |

No start fee. Error rows are free. Set `maxItems` and the run's maximum cost to cap spend. See `PRICING.md` for the comparison with other Hacker News actors.

### Measured speed (local run, public APIs)

- Front page, 30 rows: 2.2 s; 100 rows: 2.8 s.
- Search, 1,200 newest-first stories across two Algolia windows: 4.2 s, no duplicates.
- Comment tree: 72 rows (the 2007 Dropbox thread, matches HN's own count) in 2.3 s; a 136-row tree in 1.9 s.
- Who is hiring, latest thread (October 2026): 194 job rows in 1.7 s.

### Limits

- Algolia returns at most 1,000 hits per query. With `searchSort: "date"` the actor keeps going by moving the date window; relevance search stops at 1,000 and logs a warning.
- Hacker News data is what is public on the site; dead and deleted items are skipped.
- `minPoints` / `minComments` apply to search and list modes; they do not apply to comment trees or hiring posts.
- Comment trees of very large threads (thousands of comments) are one big response; keep memory at 512 MB or more for those.
- Requests are retried with exponential backoff. If an item still fails you get an error row instead of a crashed run.

### FAQ

**What limits the output?** `maxItems` bounds saved rows. Comment depth and hiring month inputs further narrow the selected mode.

**Do I need a proxy, login, or API key?** No. It reads the public Hacker News Firebase and Algolia search APIs.

**How is it priced?** The proposed pay-per-event model charges `item` for story, comment, or search-hit rows and `job` for parsed hiring rows. Error rows have no custom event charge.

**How are rate limits handled?** The API client retries temporary failures with backoff. Source throttling or unavailable items can leave fewer rows than the requested cap.

**How fresh are posts?** The actor fetches public API data during each run. `created_at` or `posted_at` is the source posting time, not a guarantee that a listing remains open.

**Why is a hiring field empty?** Hiring headers vary; inspect `raw_text` before relying on parsed job fields.

### Use with AI agents / MCP

Every mode is one JSON input and a flat JSON dataset, so an agent can call it as a tool: through the Apify MCP server (`mcp.apify.com`) add this actor, then ask for "the latest Who is hiring posts that mention Rust and are remote" or "all comments on this HN story". The input schema documents each field, and `hiringKeywords`, `hiringRemoteOnly` and `maxItems` keep the result small enough for a model context.

# Actor input Schema

## `mode` (type: `string`):

What to collect. Front page / new / best / ask / show / jobs lists come from the official Hacker News API. 'search' uses Algolia HN Search. 'comments' returns the full comment tree of the stories you name as flat rows. 'whoishiring' parses the monthly 'Ask HN: Who is hiring?' thread into one structured row per job post.

## `maxItems` (type: `integer`):

Hard cap on rows returned (and charged) in this run.

## `query` (type: `string`):

Keywords to search for, for example `rust async` or `"show hn" sqlite`. May be empty if you set an author, date range or minimum points.

## `type` (type: `string`):

Restrict search to one kind of item. `ask_hn`, `show_hn` and `front_page` match Algolia tags.

## `searchSort` (type: `string`):

Relevance is capped at 1,000 hits by Algolia. Newest first has no practical cap: the actor keeps narrowing the date window.

## `author` (type: `string`):

Only items posted by this HN username.

## `dateFrom` (type: `string`):

UTC date `YYYY-MM-DD` (or ISO timestamp). Inclusive.

## `dateTo` (type: `string`):

UTC date `YYYY-MM-DD` (or ISO timestamp). A bare date includes that whole day.

## `minPoints` (type: `integer`):

Keep only items with at least this many points. Works in search mode and in the list modes.

## `minComments` (type: `integer`):

Keep only items with at least this many comments. Works in search mode and in the list modes.

## `storyIds` (type: `array`):

Item ids or `https://news.ycombinator.com/item?id=...` links. One request returns a whole tree, however large.

## `includeStory` (type: `boolean`):

Add the story itself as the first row of each tree. It is charged like any other row.

## `maxCommentDepth` (type: `integer`):

0 = whole tree. 1 = top-level comments only, 2 = top-level plus direct replies, and so on.

## `hiringMonths` (type: `array`):

`latest` finds the newest 'Ask HN: Who is hiring?' thread. Or name months such as `2026-09` or `September 2026`. Each thread is one request and typically holds several hundred posts.

## `hiringRemoteOnly` (type: `boolean`):

Keep only posts whose header mentions remote work.

## `hiringKeywords` (type: `array`):

Keep only posts whose text contains ANY of these words (case-insensitive), for example `rust`, `python`, `machine learning`.

## `includeHtml` (type: `boolean`):

Add `text_html` (the original HN markup) next to the plain `text` field.

## `maxConcurrency` (type: `integer`):

Parallel requests to the Hacker News API. The default is polite; raise it for large list runs.

## Actor input object example

```json
{
  "mode": "topstories",
  "maxItems": 30,
  "type": "story",
  "searchSort": "relevance",
  "minPoints": 0,
  "minComments": 0,
  "storyIds": [
    "8863"
  ],
  "includeStory": true,
  "maxCommentDepth": 0,
  "hiringMonths": [
    "latest"
  ],
  "hiringRemoteOnly": false,
  "includeHtml": false,
  "maxConcurrency": 10
}
```

# Actor output Schema

## `dataset` (type: `string`):

Dataset with all result rows (stories, comments or parsed hiring posts)

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "storyIds": [
        "8863"
    ],
    "hiringMonths": [
        "latest"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("grit-77/hacker-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "storyIds": ["8863"],
    "hiringMonths": ["latest"],
}

# Run the Actor and wait for it to finish
run = client.actor("grit-77/hacker-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "storyIds": [
    "8863"
  ],
  "hiringMonths": [
    "latest"
  ]
}' |
apify call grit-77/hacker-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,grit-77/hacker-news-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/G8MkphCPDC5AvqUIC/builds/8a6x5tU6iUMyvLL1y/openapi.json