# Hacker News Scraper: Stories, Comments, Search, Users (API) (`charlesmarkette/hacker-news-scraper-stories-comments-search`) Actor

Get Hacker News data from the official APIs: top, new, best, Ask HN, Show HN and job stories, keyword and date search, full comment threads, and user profiles. Points, comments, authors and dates as clean rows, with an only-new mode for monitoring.

- **URL**: https://apify.com/charlesmarkette/hacker-news-scraper-stories-comments-search.md
- **Developed by:** [Charles Markette](https://apify.com/charlesmarkette) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.14 / 1,000 output items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

**Hacker News Scraper** gets stories, comments, search results and user profiles from Hacker News (news.ycombinator.com) through its **official public APIs**. No HTML scraping, no browser and no proxy. Pick a mode:

- **Lists:** the **top** (front page), **new**, **best**, **Ask HN**, **Show HN** and **job** lists, with each story's rank.
- **Search:** Algolia's HN search by **keyword**, **tags** (story, comment, show_hn, ask_hn, author\_…), **date range**, **minimum points** and **minimum comments**, newest first or by relevance. Date-sorted searches can go past Algolia's 1,000-result cap.
- **Items:** any story, comment, job or poll by ID or `news.ycombinator.com/item?id=…` URL.
- **Users:** profile (created date, karma, about) and, optionally, their latest stories or comments.

Turn on **Include comments** to get each story's whole comment thread, either **nested** inside the story row or **flat** (one row per comment with `storyId`, `parentId` and `depth`). Turn on **Only new items** and each scheduled run returns only stories you haven't had before.

### Why this Actor

- **Official APIs only.** Lists, items and users come from HN's [Firebase API](https://github.com/HackerNews/API); search and comment trees come from the [Algolia HN Search API](https://hn.algolia.com/api). Both are public and need no key. Nothing breaks when the HN website's layout changes.
- **Whole threads in one request.** A story's full comment tree comes from one Algolia request instead of one request per comment. Top-level comments are put in HN's own ranking order (from the story's Firebase `kids` list) when the story came from a list or an item ID.
- **Clean text.** HN returns story text, comments and profiles as HTML. You get **plain text** (links written out in full, code blocks kept), **Markdown**, or the original **HTML**.
- **Monitoring built in.** *Only new items* remembers the last 50,000 story IDs per task and skips them on later runs. Pair it with an email or Slack step to get new Show HN posts, or every new mention of your product.
- **Polite.** At most 10 parallel requests to Firebase (adjustable) and 2 to Algolia, with Algolia requests spaced 0.4 s apart to stay within its public limit of 10,000 requests per hour. Busy (429) and server errors (5xx) are retried with backoff, honouring `Retry-After`.
- **Honest output.** Deleted and dead (flagged) items are left out by default, or included with `deleted` / `dead` flags. An item that can't be fetched gets a free row with the reason in `error`, and the run carries on.

### Use cases

- **Front-page tracking:** snapshot the top 30 every hour with ranks and points, to chart how stories rise and fall.
- **Brand and competitor monitoring:** search for your product or competitors' names, sort by date, with *Only new items*, and send each new mention to Slack.
- **Show HN / Launch HN research:** every Show HN post with 100+ points in a date range, with its comments.
- **Discussion analysis and datasets:** full comment threads for sentiment analysis, summarising, or training data, in a flat table.
- **Hiring:** the latest job posts from the job list, or "Who is hiring?" comments with `tags: story_ID`.
- **User research:** karma, account age and latest submissions for a list of usernames.

### How to use it

1. Choose a **Mode** and fill in its section: lists, a search, item IDs/URLs, or usernames.
2. Optional: **Include comments** (nested or flat), choose a **Text format**, and turn on **Only new items** for scheduled runs.
3. Run it, or schedule it. Export as CSV, Excel or JSON, or chain it with an integration (see Related Actors).

**Input examples**

The front page, top 30 stories:

```json
{ "mode": "lists", "lists": ["top"], "maxItems": 30 }
```

Show HN posts from the last 7 days with at least 50 points, with up to 300 comments each, one row per comment:

```json
{
  "mode": "search",
  "tags": ["show_hn"],
  "dateFrom": "7 days",
  "minPoints": 50,
  "sort": "date",
  "maxItems": 500,
  "includeComments": true,
  "commentsFormat": "flat",
  "maxCommentsPerStory": 300
}
```

Every new comment that mentions your product, for an hourly schedule:

```json
{ "mode": "search", "query": "acme db", "tags": ["comment"], "sort": "date", "maxItems": 200, "onlyNew": true }
```

**Output example** (a story row; illustrative values)

```json
{
  "id": 41234567,
  "type": "story",
  "title": "Show HN: A tiny SQLite-backed job queue",
  "url": "https://github.com/example/tinyqueue",
  "domain": "github.com",
  "text": null,
  "author": "example_user",
  "points": 182,
  "commentCount": 64,
  "createdAt": "2026-10-20T14:03:11Z",
  "hnUrl": "https://news.ycombinator.com/item?id=41234567",
  "rank": 4,
  "list": "top",
  "storyId": null,
  "parentId": null,
  "depth": null,
  "storyTitle": null,
  "tags": ["story", "show_hn"],
  "karma": null,
  "submissionCount": null,
  "deleted": false,
  "dead": false,
  "error": null
}
```

With **Include comments**, story rows also get `commentsIncluded` (how many comments were output) and `commentsTruncated` (true when *Max comments per story* cut the thread). In **nested** mode they get a `comments` array:

```json
"comments": [
  {
    "id": 41234601, "author": "another_user", "text": "How does it handle crashes mid-job?",
    "createdAt": "2026-10-20T14:20:45Z", "hnUrl": "https://news.ycombinator.com/item?id=41234601",
    "parentId": 41234567, "depth": 0, "deleted": false,
    "replies": [ { "id": 41234688, "author": "example_user", "text": "Jobs are leased with a timeout…", "depth": 1, "parentId": 41234601, "replies": [] } ]
  }
]
```

In **flat** mode each comment is its own row after its story, with `type: "comment"`, `storyId`, `storyTitle`, `parentId` and `depth` (0 = a direct reply to the story).

Other columns: `listRanks` (when you pick several lists: the story's rank in each, e.g. `{"top": 3, "best": 12}`), `pollOptions` (polls: `[{id, text, points}]`), and for users `karma`, `submissionCount` and `text` (their About section). A run summary is saved as `OUTPUT` in the key-value store.

### How much does it cost?

Pay per event: **$0.0002 per output item**, plus **$0.00005 per run start**. Apify platform usage is included. An output item is a story, job or poll row, a user row, or a comment (a flat comment row, or each comment inside a nested story row: a story row with 40 nested comments counts as 41 items, the same as in flat mode).

These are free: error rows (items or users that couldn't be fetched or don't exist, invalid input), deleted-comment placeholders, and everything left out: deleted or dead items, duplicates, and items already seen by *Only new items*.

**Apify Store discounts:** the price above is for Apify's Free plan. Paid plans pay less per 1,000 items: $0.18 on Starter, $0.16 on Scale, $0.14 on Business. The "from" price at the top of the Store page is the Business price.

**Worked examples:**

*Examples use the Free-plan rate. The run-start charge applies to every run, counted per GB of Actor memory (minimum one; one at the default 256 MB).*

- The top 30 stories: 30 × $0.0002 + $0.00005 = **$0.00605**.
- 30 stories with their comments, 6,000 comments in total (with *Max comments per story* at 200 or more): 6,030 × $0.0002 + $0.00005 = **$1.20605**.
- A daily *Only new items* search (for example, all Show HN posts) that finds 150 new stories a day: 150 × $0.0002 + $0.00005 = $0.03005 per day, about **$0.90 per month**.
- An hourly front-page snapshot of 30 stories: 24 × $0.00605 = $0.1452 per day, about **$4.36 per month**.

If you set a maximum cost per run, the Actor stops cleanly at that limit. A nested story row that doesn't fully fit is output with as many of its comments as the budget allows (`commentsTruncated: true`), and the run stops there.

### FAQ and limits

#### Is this allowed? Do I need an API key?

No key is needed. The Firebase API is HN's official public API, and the Algolia HN Search API is the public search service HN links to from its own search box. This Actor reads only those two hosts. It isn't affiliated with Y Combinator or Algolia.

#### Why do comments lag behind for very new stories?

Comment trees come from Algolia's index, which usually trails the live site by a few minutes. A story with comments that aren't indexed yet is still output, with a note in `error`. Run again later for its comments. `commentCount` is HN's own count; `commentsIncluded` can be lower (deleted comments, index lag, or your *Max comments per story*).

#### In what order are comments?

Depth-first: each comment is followed by its replies. Top-level comments follow HN's ranking when the story came from a list or an item ID. For search results, and for replies below the top level, the order is the one Algolia returns.

#### Comments don't have points?

HN's APIs don't publish comment scores, so `points` is null for comments.

#### How do I get more than 1,000 search results?

Algolia returns at most 1,000 results per query. With **Sort: Date** the Actor continues automatically with older time windows until it has *Max stories* results or none are left, without duplicates. **Sort: Relevance** can't be split this way and stops at 1,000; narrow the query or the dates instead.

#### Can it monitor every new story?

The `new` list holds only the latest 500 stories, so a monitor that runs rarely can miss some. For a complete feed, use Search mode with `tags: ["story"]`, **Sort: Date**, **Posted after** (for example `2 days`) and **Only new items**.

#### Deleted and dead items

By default, deleted and dead stories are left out (free). A deleted comment is dropped unless it has replies; then it stays as a free placeholder (`deleted: true`, no author or text) so the thread stays connected. Turn on **Include deleted and dead items** to output them all. Search results come from Algolia's index, which has no dead or deleted flags.

#### Rate limits and speed

Firebase requests run in parallel (10 by default). Algolia requests run at most 2 at a time and at most 2.5 per second, so 500 stories with comments take about 3–4 minutes. Lists, items and users without comments are fast.

#### Usernames

HN usernames are case-sensitive. A user who doesn't exist gets a free row with an error. A user's latest submissions come from Algolia's search index.

### Related Actors

| Actor | Use it to |
|---|---|
| [Send Email Report & Slack, Discord, Teams, Telegram Alerts](https://apify.com/charlesmarkette/send-email-report-slack-discord-teams-telegram-alerts) | Get new Show HN posts, or new mentions of your product, in Slack or by email after each run. |
| [Article Extractor: Text, Markdown, Author, Date](https://apify.com/charlesmarkette/article-extractor-text-markdown-author-date) | Extract the linked articles: full text, author and date for each story's `url`. |
| [RSS & Atom Feed Reader](https://apify.com/charlesmarkette/rss-atom-feed-reader-only-new-items) | Watch blogs and news feeds alongside Hacker News, with only new items. |
| [Dataset Filter & Transform](https://apify.com/charlesmarkette/dataset-filter-transform-rename-flatten-fields) | Keep only stories above a points threshold, or rename and drop columns before export. |
| [Google Sheets Sync](https://apify.com/charlesmarkette/google-sheets-sync) | Append stories or comments to a Google Sheet. |
| [HTTP Request & API Calls](https://apify.com/charlesmarkette/http-request-api-calls-bulk-pagination) | Send the results on to your own API or webhook, or call other JSON APIs in bulk. |

Data: [Hacker News API](https://github.com/HackerNews/API) (Firebase) and [HN Search API](https://hn.algolia.com/api) by Algolia.

# Actor input Schema

## `mode` (type: `string`):

Lists: the stories on HN's front-page lists. Search: Algolia's HN search by keyword, tag, date and points. Items: specific stories or comments by ID or URL. Users: user profiles, optionally with their latest submissions.

## `lists` (type: `array`):

Lists mode. One or more of the official story lists. A story on several lists is output once (with its rank in each list in listRanks).

## `maxItems` (type: `integer`):

Lists mode: how many stories to read from the top of each list (the API gives up to 500 for top, new and best, 200 for Ask, Show and jobs). Search mode: how many search results to read in total (more than 1,000 needs Sort = Date; see the README).

## `query` (type: `string`):

Search mode. Words to look for in titles, URLs, story text and comments (Algolia full-text search). Leave empty to get everything that matches the tags and dates.

## `tags` (type: `array`):

Search mode. All must match: story, comment, poll, ask_hn, show_hn, front_page, author_USERNAME, story_ID (comments of one story). An OR group: (story,poll).

## `sort` (type: `string`):

Search mode. Date: newest first, and can read more than 1,000 results (it moves to older time windows automatically). Relevance: Algolia's ranking (text match, then points), at most 1,000 results.

## `dateFrom` (type: `string`):

Search mode. A date (2026-10-01), a time (2026-10-01T08:00:00Z) or a period back from now ('7 days', '24 hours').

## `dateTo` (type: `string`):

Search mode. Same formats. A date on its own includes that whole day (UTC).

## `minPoints` (type: `integer`):

Search mode. Only stories with at least this many points (comments have no points, so this leaves them out).

## `minComments` (type: `integer`):

Search mode. Only stories with at least this many comments.

## `items` (type: `array`):

Items mode. Item IDs (8863) or Hacker News URLs (https://news.ycombinator.com/item?id=8863): stories, comments, jobs or polls.

## `users` (type: `array`):

Users mode. Usernames (case-sensitive) or profile URLs (https://news.ycombinator.com/user?id=pg). Each gives one row: created date, karma, about text and number of submissions.

## `userSubmissions` (type: `integer`):

Users mode. Also output each user's latest N submissions, newest first (0 = profile only, up to 1,000). Each submission is an output row.

## `userSubmissionTypes` (type: `string`):

Users mode. Which submissions to list.

## `includeComments` (type: `boolean`):

Also get each story's comment thread (one Algolia request per story with comments). Each included comment is charged like a row; see the pricing section.

## `commentsFormat` (type: `string`):

Nested: a 'comments' array on the story row, replies inside replies. Flat: one row per comment after its story, with storyId, parentId and depth (best for spreadsheets and filtering).

## `maxCommentsPerStory` (type: `integer`):

Comments are taken in thread order (top-level comments in HN's ranking, each followed by its replies) up to this many.

## `textFormat` (type: `string`):

Story text, comments and user 'about' come from HN as HTML. Plain text (links written out), Markdown, or the original HTML.

## `includeDeleted` (type: `boolean`):

Off: deleted and dead (flagged or killed) stories are left out, and deleted comments are dropped unless they have replies (then they stay as a free placeholder row so the thread stays connected). On: all are output with deleted / dead set to true.

## `onlyNew` (type: `boolean`):

Output only stories (and search results) not output by an earlier run with the same settings. The first run outputs everything it finds. Memory: the last 50,000 IDs, in the key-value store 'hacker-news-scraper-state'.

## `stateKey` (type: `string`):

Optional. Runs with the same memory name share their 'only new' memory. By default each saved task has its own memory per set of settings. Letters, digits, '-', '\_' and '.'.

## `concurrency` (type: `integer`):

How many item requests to send to HN's Firebase API at the same time (1–20). Algolia requests are always limited to 2 at a time and spaced out to stay within its public rate limit.

## `maxResults` (type: `integer`):

Optional cap on output rows for the whole run (stories, users and, in flat mode, comment rows).

## Actor input object example

```json
{
  "mode": "lists",
  "lists": [
    "top"
  ],
  "maxItems": 10,
  "sort": "date",
  "userSubmissions": 0,
  "userSubmissionTypes": "stories",
  "includeComments": false,
  "commentsFormat": "nested",
  "maxCommentsPerStory": 200,
  "textFormat": "text",
  "includeDeleted": false,
  "onlyNew": false,
  "concurrency": 10
}
```

# Actor output Schema

## `items` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "lists",
    "lists": [
        "top"
    ],
    "maxItems": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("charlesmarkette/hacker-news-scraper-stories-comments-search").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "lists",
    "lists": ["top"],
    "maxItems": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("charlesmarkette/hacker-news-scraper-stories-comments-search").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "lists",
  "lists": [
    "top"
  ],
  "maxItems": 10
}' |
apify call charlesmarkette/hacker-news-scraper-stories-comments-search --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,charlesmarkette/hacker-news-scraper-stories-comments-search"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/aoCknisD5pQ3GCYKb/builds/3vgHoJsYy3G5IVRYT/openapi.json