# Hacker News Scraper: Search, Stories & Comments (`private_image_hg9/hacker-news-scraper`) Actor

Search every Hacker News story and comment by keyword, type, date, points and author, read the live top, new, best, Ask, Show and job lists, or fetch items by ID, with optional comment threads. Uses the official public APIs, so it is fast and needs no proxies.

- **URL**: https://apify.com/private_image_hg9/hacker-news-scraper.md
- **Developed by:** [Michael Shanker](https://apify.com/private_image_hg9) (community)
- **Categories:** News, Developer tools, Automation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$0.50 / 1,000 item saveds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Hacker News Scraper: Search, Stories & Comments

Search every story and comment ever posted on **Hacker News** by keyword, type, date, points and author. Read the live **top, new, best, Ask HN, Show HN and job** lists, or fetch any story or comment by its ID, with its whole **comment thread** if you want it. One clean row per item, ready for a spreadsheet, a database or an AI pipeline.

It reads Hacker News's two public APIs: the **Algolia HN Search API** (the search behind hn.algolia.com) and the **official HN API** on Firebase. No browser, no proxies, no logins, and nothing to break when the page layout changes.

**$0.50 per 1,000 items.** You pay only for stories, comments, jobs and polls saved. IDs that don't exist, deleted or dead items and unusable inputs still get a row, free.

### Who it's for

- **Founders and marketers**: find every discussion of your product, a competitor or a market, with points and comment counts.
- **Researchers and data teams**: build datasets of tech discussion by topic and date, going back to 2006.
- **AI and RAG pipelines**: comment threads as plain text, with `parentId`, `storyId` and `depth` to rebuild each conversation.
- **Newsletters, alerts and dashboards**: schedule a run on the front page or a search sorted by date, and send new items to Sheets, Slack, Make, Zapier or n8n.

### How it compares

| | This Actor | Hacker News scrapers that read the website |
|---|---|---|
| Source | Official public APIs (Algolia HN Search and HN Firebase) | news.ycombinator.com pages |
| Search all of HN history | Yes, stories and comments since 2006 | Usually only the pages you point it at |
| Filters | type (story, comment, Ask HN, Show HN, job, poll, front page), date range, minimum points and comments, author | varies by tool |
| More than 1,000 results per search | Yes, split into date windows for you | varies by tool |
| Comment threads | Flattened rows with `parentId`, `storyId`, `depth` | varies by tool |
| Text | HTML turned into plain text, shortened links expanded | varies by tool |
| You pay for | saved items only | varies by tool |

### Input

| Field | What it does |
|---|---|
| **Search queries** | Keywords, one search per line, e.g. `rust`, `"vector database"`. Searches titles, links, story text and comments. Repeated queries are run once. |
| **What to search** | Stories (default), comments, Ask HN, Show HN, jobs, polls, stories that made the front page, or everything. |
| **Sort by** | Relevance (Algolia's ranking, which favours points and comments) or newest first. |
| **From date / To date** | `2024-01-31` or `2024-01-31T18:00:00Z`, in UTC. A bare end date includes that whole day. |
| **Minimum points / Minimum comments** | Only stories above these numbers. Comments have no public points, so these are ignored for comment searches. |
| **Author** | A Hacker News username, e.g. `pg`. Case-sensitive. With no query, lists everything that user posted of the chosen type. |
| **Max results per search** | 1 to 50,000 (default 100). |
| **Live lists** | `top` (the front page), `new`, `best`, `ask`, `show`, `job`, in their current order. |
| **Max items per list** | From the top of each list (default 30). The top and new lists hold up to 500, the others up to 200. |
| **Item IDs or links** | `8863` or `https://news.ycombinator.com/item?id=8863`. Stories, comments, jobs, polls. Only news.ycombinator.com links are read. |
| **Include comments** | Also save each story's comments right after it. Each comment is a charged row. |
| **Max comments per story** | Stop each thread after this many comments (default 100), read in thread order. |

Leave **Search queries** empty and set a filter (author, dates, points) to list everything that matches the filters. Search filters apply to searches only; lists and item IDs are returned as they are.

Example input:

```json
{
  "searchQueries": ["rust", "\"vector database\""],
  "searchType": "story",
  "dateFrom": "2024-01-01",
  "minPoints": 100,
  "sortBy": "date",
  "maxItemsPerSearch": 500,
  "lists": ["top"],
  "itemIds": ["8863"],
  "includeComments": true,
  "maxCommentsPerStory": 50
}
```

### Output

One row per item. A story and one of its comments (values illustrative):

```json
{
  "id": 8863,
  "type": "story",
  "title": "My YC app: Dropbox - Throw away your USB drive",
  "url": "http://www.getdropbox.com/u/2/screencast.html",
  "domain": "getdropbox.com",
  "text": null,
  "author": "dhouston",
  "points": 111,
  "numComments": 71,
  "createdAt": "2007-04-04T19:16:40Z",
  "hnUrl": "https://news.ycombinator.com/item?id=8863",
  "parentId": null,
  "storyId": 8863,
  "storyTitle": null,
  "depth": 0,
  "tags": ["story"],
  "rank": null,
  "source": "item",
  "input": "8863",
  "status": "ok",
  "error": null,
  "scrapedAt": "2026-11-04T18:00:00+00:00"
}
```

```json
{
  "id": 9224,
  "type": "comment",
  "title": null,
  "url": null,
  "domain": null,
  "text": "Plain text of the comment.\n\nParagraphs are kept, links are written out in full.",
  "author": "someuser",
  "points": null,
  "numComments": null,
  "createdAt": "2007-04-05T02:11:40Z",
  "hnUrl": "https://news.ycombinator.com/item?id=9224",
  "parentId": 8863,
  "storyId": 8863,
  "storyTitle": "My YC app: Dropbox - Throw away your USB drive",
  "depth": 1,
  "tags": ["comment"],
  "rank": null,
  "source": "comments",
  "input": "8863",
  "status": "ok",
  "error": null,
  "scrapedAt": "2026-11-04T18:00:00+00:00"
}
```

- **type**: `story`, `comment`, `job`, `poll` or `pollopt`. Ask HN and Show HN posts are stories with an `ask_hn` or `show_hn` tag.
- **text**: the story text (Ask HN and similar) or the comment, as plain text. HTML tags are removed, entities decoded, paragraphs kept as blank lines, code blocks kept, and links Hacker News shortens with `...` written out in full.
- **parentId / storyId / depth**: for comments, the item replied to, the story the thread belongs to, and how deep the reply is (1 = a direct reply to the story). Comments found by a search have no `depth`, because search results don't include it.
- **rank**: the position in the search results or list.
- **source / input**: `search`, `list`, `item` or `comments`, and the query, list name or ID that produced the row.
- **status**: `ok` (charged), or free rows: `not_found` (no item has this ID), `deleted`, `dead` (flagged or killed, hidden on the site) and `invalid_input`. `error` says why.
- **points** and **numComments** are as of the moment the API was read. Comments have no public points.

Each item is saved once per run, even when two searches, a list and an ID all find it. The run also saves a `SUMMARY` record in the key-value store: rows by status and type, total matches per search, items charged, requests made, and whether the spending cap stopped the run.

### Pricing

Pay per event, no start fee: **$0.0005 per item saved ($0.50 per 1,000)**. Every story, comment, job or poll row with status `ok` is one item, so a story with 40 comments is 41 items. Rows for missing, deleted or dead items and unusable inputs are free.

Set a maximum charge per run in the run options and the Actor stops cleanly when it's reached, before fetching anything it couldn't charge for.

### Good to know

- **The 1,000-result cap.** Algolia returns at most 1,000 results for one query. When you ask for more and more exist, the Actor splits the search into date windows for you:
  - **Newest first:** after the newest 1,000, it asks again for items older than the oldest one seen, and so on, so results stay in exact date order with no repeats and no cap.
  - **Relevance:** the date range is split into windows of at most 1,000 matches each, newest window first, and results are in relevance order **within each window**, not one ranking across all of HN. For one global ranking, keep **Max results per search** at 1,000 or below.
    The log and `SUMMARY` say how many windows were used. Algolia's match counts are its own estimates.
- **Speed and politeness.** Algolia's documented limit is 10,000 requests per hour per IP. The Actor sends at most 2.5 Algolia requests per second (9,000 an hour, under that limit), and each returns up to 100 results, so 1,000 results take about 10 requests. The official API has no published limit; the Actor keeps at most 10 requests to it in flight. Comment threads come from Algolia in one request per story. Failed requests are retried with backoff (1, 2, 4, 8 seconds) and the API's own Retry-After is honoured.
- **Very new stories.** Algolia indexes new items within minutes. If a story's thread isn't there yet, or a thread is too large to read in one answer, comments are read from the official API instead, level by level, up to your comment limit.
- **Lists are live.** The top, new and best lists change by the minute; two runs a few minutes apart will differ.
- **Only the public APIs are contacted.** Story links are returned as data and never opened, and item links must be news.ycombinator.com links.
- **Usernames** are the public pseudonyms Hacker News shows. The Actor returns only what the APIs return publicly and doesn't look up user profiles.

### Source and terms

Data comes from Hacker News (news.ycombinator.com, run by Y Combinator) through its official public API, and from the Algolia HN Search API, a public search service Algolia runs for Hacker News. Posts and comments belong to their authors. Check Y Combinator's terms before republishing them, and credit Hacker News when you do.

This Actor isn't affiliated with or endorsed by Y Combinator, Hacker News or Algolia.

### More tools from the same developer

- [News Scraper: GDELT Articles, Full Text & 65 Languages](https://apify.com/private_image_hg9/news-search-gdelt)
- [RSS Feed Reader: Atom & RSS to JSON with Feed Finder](https://apify.com/private_image_hg9/rss-feed-reader-finder)
- [Google Trends Scraper: Interest Over Time, Regions & Queries](https://apify.com/private_image_hg9/google-trends-scraper)
- [OpenAlex Papers Search: Scholarly Works, Citations & Abstracts](https://apify.com/private_image_hg9/openalex-papers-search)
- [ClinicalTrials.gov Search: Studies, Phases, Sponsors & Status](https://apify.com/private_image_hg9/clinical-trials-search)
- [Federal Contracts Search: SAM.gov Opportunities & Award History](https://apify.com/private_image_hg9/federal-contracts-search)

# Actor input Schema

## `searchQueries` (type: `array`):

Keywords to search all of Hacker News for, one search per line (full-text search over titles, links, story text and comments). Leave empty to use only lists or item IDs, or to list everything that matches the filters below.

## `searchType` (type: `string`):

Which kind of item each search returns.

## `sortBy` (type: `string`):

Relevance ranks the best matches first (Algolia's ranking, which favours points and comments). Date gives the newest first.

## `dateFrom` (type: `string`):

Only items posted on or after this date (UTC), as YYYY-MM-DD or a full date-time like 2024-01-31T18:00:00Z.

## `dateTo` (type: `string`):

Only items posted on or before this date (UTC). A bare date includes that whole day.

## `minPoints` (type: `integer`):

Only stories with at least this many points. Comments have no public points, so this is ignored for comment searches.

## `minComments` (type: `integer`):

Only stories with at least this many comments. Ignored for comment searches.

## `author` (type: `string`):

Only items posted by this Hacker News username (case-sensitive, e.g. pg). With no search query, lists everything the user posted of the chosen type.

## `maxItemsPerSearch` (type: `integer`):

Stop each search after this many results. Algolia returns at most 1,000 per query; above 1,000 the search is split into date windows automatically (see the README).

## `lists` (type: `array`):

The live lists from the official Hacker News API, in their current order: top (front page), new, best, Ask HN, Show HN and jobs.

## `maxItemsPerList` (type: `integer`):

How many items to read from the top of each list. The top and new lists hold up to 500, the others up to 200.

## `itemIds` (type: `array`):

Specific stories, comments, jobs or polls, as numeric IDs (8863) or links (https://news.ycombinator.com/item?id=8863). Only news.ycombinator.com links are read.

## `includeComments` (type: `boolean`):

Also save each story's comments, one row per comment, right after the story, with parentId, storyId and depth so the thread can be rebuilt. Every comment is a charged row.

## `maxCommentsPerStory` (type: `integer`):

Stop each story's thread after this many comments, read in thread order (each reply right under its parent).

## Actor input object example

```json
{
  "searchQueries": [
    "rust"
  ],
  "searchType": "story",
  "sortBy": "relevance",
  "minPoints": 0,
  "minComments": 0,
  "maxItemsPerSearch": 100,
  "maxItemsPerList": 30,
  "includeComments": false,
  "maxCommentsPerStory": 100
}
```

# Actor output Schema

## `results` (type: `string`):

One row per story, comment, job or poll, plus free rows for IDs that were not found and inputs that could not be used.

## `summary` (type: `string`):

Counts for the run: rows by status and type, matches per search, items charged, and whether the spending cap was reached.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "rust"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("private_image_hg9/hacker-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "searchQueries": ["rust"] }

# Run the Actor and wait for it to finish
run = client.actor("private_image_hg9/hacker-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "rust"
  ]
}' |
apify call private_image_hg9/hacker-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,private_image_hg9/hacker-news-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/7YyAQmeVmx5AVNn4Z/builds/afpSa1uSd1D9bhHby/openapi.json