# Hacker News Scraper — Stories, Comments & Hiring Jobs (`pixflor/hacker-news-scraper`) Actor

Scrape Hacker News: front page in real rank order, full-text search past Algolia's 1,000-result limit, comment trees, user profiles, and monthly "Who is hiring" threads parsed into structured jobs with company, role, location, salary and contact email.

- **URL**: https://apify.com/pixflor/hacker-news-scraper.md
- **Developed by:** [Pixflor](https://apify.com/pixflor) (community)
- **Categories:** Jobs, Lead generation, MCP servers
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 row scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Hacker News Scraper — Stories, Comments & Hiring Jobs

Hacker News as clean rows: the front page in **real rank order**, full-text search that
goes **past the 1,000-result wall**, whole comment trees, user profiles, and the monthly
**"Who is hiring"** threads parsed into structured job records.

No API key. No login. Pick a mode, press start, export CSV or JSON.

***

### Three things this does that other HN scrapers don't

#### 1. It gets past the 1,000-result wall

The Algolia index behind Hacker News search refuses to page beyond 1,000 results. It does
not say so — page 20 comes back **HTTP 200 with an empty list**, so a page-based scraper
stops at exactly 1,000 rows and reports success. If you asked for a month of stories, you
silently got a few days.

This Actor walks a timestamp cursor instead of paging.

| Approach | Unique stories retrieved |
|---|---|
| Page-based (every other HN Actor) | **1,000** — hard stop |
| Cursor walk (this Actor) | **6,000 in 6 requests**, and it keeps going |

#### 2. Its front page is the actual front page

The obvious shortcut is Algolia's `front_page` tag. Measured, that tag returns **171 items**
— not the 30 on the front page — sorted by relevance rather than rank, and **51 of the first
100 are YC job ads with `points: null` and `num_comments: null`.** Half your paid rows come
back empty, in the wrong order, with no rank.

This Actor reads Hacker News' own ranked lists, so every row carries a real `rank` field,
1…N, and a real score. All six lists are available: top, new, best, ask, show, jobs.

#### 3. It turns "Who is hiring" into a jobs database

"Ask HN: Who is hiring?" runs on the first of every month and is one of the best sources of
hiring data on the internet — hundreds of companies, written by the people doing the hiring,
with contact addresses. Existing scrapers hand you the raw comment text.

This one parses it. Measured on the July 2026 thread: **276 posts, 89.9% resolved into
fields**, 268 companies, 232 apply links, 250 workplace classifications, 83 contact emails.

Posts that don't match a known format still return their full text — you are never billed
for an empty row.

***

### What you get

**Stories and comments**

| Column | Example |
|---|---|
| `rank` | `1` (front-page modes — real HN rank) |
| `title` | Elevators |
| `url` / `domain` | link target · `acme.com` |
| `author` | `pg` |
| `points` / `numComments` | `1097` / `262` |
| `createdAt` | ISO 8601 |
| `text` | self-post body, HTML stripped and entities decoded |
| `isAskHN` / `isShowHN` | `true` / `false` |
| `hnUrl` | permalink to the item |

**Who is hiring / Who wants to be hired**

| Column | Example |
|---|---|
| `company` | WireScreen |
| `title` | Senior Software Engineer (Frontend) |
| `location` | NYC - hybrid |
| `workplace` | `REMOTE` · `HYBRID` · `ONSITE` |
| `employmentType` | full-time · contract · internship |
| `salaryMin` / `salaryMax` / `salaryCurrency` | `170000` / `220000` / `USD` |
| `techTags` | `["typescript","postgres","aws"]` |
| `emails` | `["jobs@acme.com"]` — obfuscated forms decoded |
| `applyUrl` / `urls` | direct link · every link in the post |
| `visa` | `mentioned` · `no` |
| `text` | the full original post, always |
| `commentUrl` / `threadTitle` | permalink · "Ask HN: Who is hiring? (July 2026)" |

***

### Modes

| Mode | What it returns |
|---|---|
| **Search stories** | Keyword, author, domain, points, comment count, date range |
| **Front page / ranked lists** | Top, new, best, ask, show or jobs — in rank order |
| **Search comments** | Full-text across every HN comment |
| **One story + all its comments** | The whole tree, flattened, with reply depth |
| **User profile** | Karma, bio, and their recent submissions |
| **Who is hiring** | Structured jobs from the monthly thread |
| **Who wants to be hired** | Structured candidate posts from the companion thread |

***

### Typical uses

- **Recruiting and lead gen.** Every company hiring this month, with contact emails, filtered
  to remote-only or to one technology. Companies hiring are companies spending.
- **Salary benchmarking.** Pull twelve months of hiring threads and read the bands.
- **Trend tracking.** Which technologies appear in job posts, month over month.
- **Newsletters and aggregators.** Schedule the front page daily and publish it.
- **Sentiment and market research.** Search comments for what people say about a product.
- **Feeding an LLM or agent.** One row per item, consistent field names, HTML already
  stripped and entities decoded.

***

### A note on "Ask HN"

Hacker News files **every text-only submission** under "ask" — including "Tell HN" and plain
untitled discussion. Neither the ask feed nor the `ask_hn` tag gives you only posts titled
"Ask HN:" (measured: 2/10 and 3/10 respectively).

If you want real Ask HN posts, set **Title must contain** to `Ask HN`. It's applied before
billing, so you don't pay for the rows it removes. `Show HN` has no such problem — that feed
is clean.

***

### Example input

Remote Python jobs with a stated salary, from the latest hiring thread:

```json
{
  "mode": "whoIsHiring",
  "workplace": "REMOTE",
  "tech": "python",
  "withSalaryOnly": true,
  "sortBy": "salary",
  "maxItems": 200
}
```

Stories about Rust that actually got traction, last 30 days:

```json
{
  "mode": "search",
  "searchQuery": "rust",
  "minPoints": 100,
  "postedWithinDays": 30,
  "sortBy": "points",
  "maxItems": 200
}
```

Real "Ask HN" posts — see the note above on why `titleContains` is the selector:

```json
{ "mode": "search", "storyType": "ask_hn", "titleContains": "Ask HN", "maxItems": 100 }
```

***

### Settings that matter

- **`titleContains`** — the only reliable way to select genuine "Ask HN:" posts, and it runs
  *before* rows are billed, so you don't pay for the ones you'd throw away.
- **`hiringThreads`** — 1 gives you the current month. Raise it to build a hiring time series
  across several months in one run.
- **`withSalaryOnly` / `withEmailOnly`** — the two filters that turn a hiring thread from
  reading material into a usable list. Both applied before billing.
- **`minPoints` / `postedWithinDays`** — HN search returns everything ever posted by default;
  these are what make a search query useful.
- **`allowTypos`** — off by default. Turn it on for name searches, off for exact technical terms
  where a fuzzy match just adds noise.
- **`maxItems`** — your cost ceiling. Set it deliberately.

***

### Reliability

Both sources are official public APIs — the Algolia HN index and the Hacker News Firebase
API. There is no HTML parsing and no bot-detection workaround anywhere in this Actor, which
is why it doesn't quietly break when a page layout changes.

A 126-assertion live test suite runs against the real APIs before every release, including
assertions that the 1,000-result wall is still there, that numeric filters are actually
applied rather than silently ignored, and that every extracted email is a valid address.

***

### Pricing

Pay per event. You are charged for the run start and for each row returned — nothing else,
and platform compute is included free. Filters that can be pushed down to the API (points,
comment count, date range) are, so they don't consume your row budget.

***

### Questions or a bug?

Open an issue on the **Issues** tab of this Actor. Include your input JSON and the run ID —
that's usually enough to reproduce it immediately.

# Actor input Schema

## `mode` (type: `string`):

Pick one. 'search' = stories by keyword/date/points. 'frontPage' = HN's ranked lists in real rank order. 'comments' = search comments. 'item' = one story plus its whole comment tree. 'user' = a profile and their recent stories. 'whoIsHiring' / 'whoWantsToBeHired' = the monthly threads parsed into structured job rows.

## `searchQuery` (type: `string`):

Full-text query for 'search' and 'comments' modes. Leave empty to get everything in date order. In other modes this filters the returned rows.

## `allowTypos` (type: `boolean`):

Off by default, and you almost certainly want it off. Hacker News search fuzzy-matches by default: measured, a search for "rust" returned 66,588 hits of which only 1 in 8 actually contained the word — the rest matched "just", "must" and "trust". Since you pay per row, that is seven wasted rows in eight. Turn this on only if you want deliberately fuzzy matching.

## `titleContains` (type: `string`):

Keep only rows whose title contains this text. Use "Ask HN" or "Show HN" here: Hacker News files EVERY text-only post under 'ask', so neither the ask feed nor the ask\_hn tag returns only real Ask HN posts. Applied before billing, so you are not charged for rows you would discard.

## `rankedList` (type: `string`):

Only used by 'frontPage' mode. Which of HN's six official lists to return, in true rank order. Note 'ask' is HN's Ask page (all self posts, including Tell HN), not only posts titled "Ask HN:".

## `storyType` (type: `string`):

Only used by 'search' mode. Which kind of submission to search. 'story' covers everything.

## `sortBy` (type: `string`):

'default' keeps the natural order for the mode — real HN rank for frontPage, newest-first for search. Choose 'relevance' only with a search text; it is the one option that cannot page past 1,000 results.

## `author` (type: `string`):

Only items submitted by this Hacker News username. Case-sensitive, exactly as it appears on HN.

## `domain` (type: `string`):

Keep only stories linking to this domain, e.g. github.com or nytimes.com. Partial match.

## `minPoints` (type: `integer`):

Keep only stories with at least this many points. Applied by the API itself, so it does not waste your row budget.

## `minComments` (type: `integer`):

Keep only stories with at least this many comments. Applied by the API itself.

## `postedWithinDays` (type: `integer`):

Only items from the last N days. 0 means no limit. Combine with a high maxItems to sweep a whole week or month.

## `itemId` (type: `string`):

Only used by 'item' mode. A numeric HN id, or a full https://news.ycombinator.com/item?id=... URL.

## `username` (type: `string`):

Only used by 'user' mode. The HN account to look up.

## `hiringMonth` (type: `string`):

Only used by the hiring modes. Leave empty for the current month's thread. Otherwise write it exactly as it appears in the thread title, e.g. "July 2026".

## `hiringThreads` (type: `integer`):

Only used by the hiring modes. How many recent monthly threads to pull, newest first. Raise this to build a hiring-trend history.

## `workplace` (type: `string`):

Only used by the hiring modes. Keep only jobs classified as remote, hybrid or onsite.

## `tech` (type: `string`):

Only used by the hiring modes. Keep only jobs whose detected tech tags include this, e.g. rust, python, kubernetes.

## `withSalaryOnly` (type: `boolean`):

Only used by the hiring modes. Drop posts where no salary range could be read from the text.

## `withEmailOnly` (type: `boolean`):

Only used by the hiring modes. Drop posts with no contact address. Obfuscated forms like "jane (at) acme (dot) io" are decoded and count as present.

## `maxItems` (type: `integer`):

How many rows to return, and what you pay for. Search mode walks a date cursor, so values above 1,000 work here even though the underlying API refuses to page that deep.

## Actor input object example

```json
{
  "mode": "search",
  "searchQuery": "postgres",
  "allowTypos": false,
  "titleContains": "Ask HN",
  "rankedList": "top",
  "storyType": "story",
  "sortBy": "default",
  "author": "pg",
  "domain": "github.com",
  "minPoints": 0,
  "minComments": 0,
  "postedWithinDays": 0,
  "itemId": "https://news.ycombinator.com/item?id=8863",
  "username": "pg",
  "hiringMonth": "July 2026",
  "hiringThreads": 1,
  "workplace": "",
  "tech": "rust",
  "withSalaryOnly": false,
  "withEmailOnly": false,
  "maxItems": 100
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("pixflor/hacker-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("pixflor/hacker-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call pixflor/hacker-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pixflor/hacker-news-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/5d8RqA0Uo6NQMJol4/builds/6z0LPDz2antYFqVNa/openapi.json