# Hacker News Search Scraper (`tidyharvest/hacker-news-search-scraper`) Actor

Search Hacker News stories, comments, Ask HN and Show HN by keyword, author, thread, points and date, or grab the current front page. Full text, points, comments, links. No proxy. $0.19 per 1,000 items, down to $0.09 on higher Apify plans.

- **URL**: https://apify.com/tidyharvest/hacker-news-search-scraper.md
- **Developed by:** [TidyHarvest](https://apify.com/tidyharvest) (community)
- **Categories:** News, Developer tools
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.09 / 1,000 item scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Hacker News Search Scraper

Search all of **Hacker News**: **stories, comments, Ask HN, Show HN, polls and jobs**, by keyword, author, thread,
points, comment count and date range. Or pull the **current front page** (top, new, best, Ask, Show, jobs) in ranked order.
Every row has the title, link, author, points, comment count, date, the full text (clean plain text plus the original
HTML) and the news.ycombinator.com link. No login or proxy needed, and runs finish in seconds. **$0.19 per 1,000 items** (less on paid plans).

### What you can do with it

- **Brand and product monitoring:** find every story and comment that mentions your company, product or competitors, newest first, on a daily schedule.
- **Developer sentiment research:** collect comments on a technology (Rust, Postgres, LLM agents) over a date range for analysis or an LLM pipeline.
- **Launch tracking:** list every Show HN or Ask HN above a points threshold in a given month.
- **Thread export:** download every comment in one discussion (story ID + Comments), or turn on *Add each story's comments* to get the discussion under every search result or front-page story.
- **Who is talking:** add each author's karma, account age and profile text for influence and lead research.
- **Front page snapshots:** save the top 30 (or 500) stories with rank, points and comment count, every hour.
- **Author history:** everything a given HN user has posted.

Looking for job posts from the monthly "Ask HN: Who is hiring?" threads? Our separate **HN Who is Hiring** actor parses
those into one row per job. This actor is general search.

### How to use

1. Type one or more **search queries** (or leave the default to try it).
2. Pick an **item type** (stories, comments, Ask HN, Show HN...), a **sort** (relevance or newest first) and any filters: minimum points, minimum comments, date range, author, story ID.
3. Set **Max items**, run, and download the results as JSON, CSV or Excel, or read them via the API.

For the front page, set **Mode** to `list` and pick a **List type**.

### Input

| Field | What it does |
|---|---|
| `mode` | `search` (default) or `list` (current ranked list). |
| `queries` | Search terms, one per line. Default `web scraping`. An empty query with an author/story/date filter lists everything matching the filter. |
| `type` | `story` (default), `comment`, `ask_hn`, `show_hn`, `poll`, `job` or `all`. |
| `sort` | `relevance` (default) or `date` (newest first). |
| `minPoints`, `minComments` | Numeric thresholds. Comments have no points on HN, so keep `minPoints` at 0 for comments. |
| `dateFrom`, `dateTo` | `YYYY-MM-DD` (UTC), inclusive. |
| `author` | HN username. |
| `storyId` | One thread: an id or a news.ycombinator.com item URL. |
| `listType` | List mode: `top`, `new`, `best`, `ask`, `show`, `job`. |
| `maxItems` | Hard cap on results for the run (also caps cost). Default 50, maximum 50,000. |
| `includeComments`, `maxCommentsPerStory`, `commentDepth` | Also return each story's discussion as comment rows right after it, top comments in the page's ranking order (with `depth`, `threadOf`); each comment is one item. |
| `includeAuthorProfile` | Add the author's karma, account creation date and profile text to every row. |
| `maxItemsPerQuery` | Optional cap for each query. |

Example: every Show HN with 100+ points in September 2026, newest first:

```json
{ "queries": [""], "type": "show_hn", "minPoints": 100, "dateFrom": "2026-09-01", "dateTo": "2026-09-30", "sort": "date", "maxItems": 500 }
```

### Output

One row per story or comment:

```json
{
  "objectID": "22180559",
  "type": "story",
  "isAskHN": false,
  "isShowHN": false,
  "title": "Congrats! Web scraping is legal! (US precedent)",
  "url": "https://parsers.me/...",
  "author": "...",
  "points": 1057,
  "numComments": 336,
  "createdAt": "2020-01-30T13:49:57Z",
  "createdAtTimestamp": 1580392197,
  "text": null,
  "textHtml": null,
  "storyId": "22180559",
  "parentId": null,
  "storyTitle": "Congrats! Web scraping is legal! (US precedent)",
  "storyUrl": "https://parsers.me/...",
  "hnUrl": "https://news.ycombinator.com/item?id=22180559",
  "storyHnUrl": "https://news.ycombinator.com/item?id=22180559",
  "tags": ["story", "author_...", "story_22180559"],
  "query": "web scraping",
  "source": "Hacker News (news.ycombinator.com)"
}
```

Comments add `text` (plain text, links kept), `textHtml` (original HTML), `parentId`, and the parent story's
`storyTitle`/`storyUrl`. List mode adds `rank` and `list`.

| Field | Meaning |
|---|---|
| `objectID` | HN item id |
| `type` | story, comment, poll, pollopt or job |
| `isAskHN`, `isShowHN` | Ask HN / Show HN flags |
| `title`, `url` | Story title and the link it points to |
| `author` | HN username |
| `points`, `numComments` | Score and comment count (stories) |
| `createdAt`, `createdAtTimestamp` | Post time, ISO UTC and Unix seconds |
| `text`, `textHtml` | Body text: plain, and the original HTML |
| `storyId`, `parentId` | Thread and parent ids (comments) |
| `storyTitle`, `storyUrl`, `storyHnUrl` | The thread a comment belongs to |
| `hnUrl` | Link to the item on news.ycombinator.com |
| `tags` | Search tags (type, author, story) |
| `query` | The search that found it |
| `rank`, `list` | List mode only |

### Pricing

Pay per result: **$0.19 per 1,000 items** on the Free plan, down to $0.09 on higher plans (`item-scraped` event). Filters and caps apply before you are charged, and
duplicate items across your queries are only charged once. A run that finds nothing fails instead of silently
returning an empty dataset.

| Apify plan tier | Free | Bronze | Silver | Gold | Platinum | Diamond |
|---|---|---|---|---|---|---|
| per 1,000 items | $0.19 | $0.17 | $0.14 | $0.09 | $0.09 | $0.09 |

### Limits and notes

- **1,000 results per relevance search.** Hacker News search returns at most 1,000 hits for one query. With **sort = date** this actor gets past that, so you can pull tens of thousands of items for one query. Relevance searches stop at 1,000; split the date range or use several queries for more.
- Points and comment counts in search results can trail the live site by a few minutes.
- Comments have no points on Hacker News, so a points filter returns no comments.
- Deleted and flagged-dead items are not returned.
- Requests are spaced politely; a 50-item run takes a few seconds, large date-sorted runs a few hundred items per second.

Data comes from Hacker News (news.ycombinator.com). Every row links back to the item on news.ycombinator.com.
This actor is not affiliated with, endorsed by or sponsored by Y Combinator or Hacker News.

# Actor input Schema

## `mode` (type: `string`):

`search` runs your queries through the HN Search API. `list` returns the current ranked Hacker News list chosen in List type (queries are ignored).

## `queries` (type: `array`):

One search per line, e.g. `web scraping`, `rust`, `"large language model"`. Leave a single empty search plus an author, story or date filter to list everything matching the filters.

## `type` (type: `string`):

What kind of Hacker News items to return.

## `sort` (type: `string`):

Relevance search returns at most 1,000 items per query (an Algolia limit). Newest-first can go past 1,000 by walking back through time.

## `minPoints` (type: `integer`):

Only items with at least this many points. Comments carry no points, so leave at 0 for comments.

## `minComments` (type: `integer`):

Only stories with at least this many comments.

## `dateFrom` (type: `string`):

Earliest post date (YYYY-MM-DD, UTC).

## `dateTo` (type: `string`):

Latest post date (YYYY-MM-DD, UTC, inclusive).

## `author` (type: `string`):

Only items posted by this Hacker News user, e.g. `pg`.

## `storyId` (type: `string`):

Restrict to one thread: a story id like `22180559` or its news.ycombinator.com URL. Combine with Item type = Comments to get every comment in that thread.

## `listType` (type: `string`):

Which ranked list to return in `list` mode. Each item gets its `rank`.

## `maxItems` (type: `integer`):

Hard cap on results for the whole run (also caps your cost).

## `maxItemsPerQuery` (type: `integer`):

Cap for each query so one broad query does not use up the whole run. 0 or empty = no per-query cap.

## `includeComments` (type: `boolean`):

For every story (or list item) returned, also return its discussion as comment rows right after it, with `depth` and `threadOf`. Each comment counts as one item toward Max items and cost.

## `maxCommentsPerStory` (type: `integer`):

Cap on comment rows per story when comments are on (page order: top-level first, replies under them).

## `commentDepth` (type: `integer`):

Optional: 1 = top-level comments only, 2 = plus direct replies, ... Empty = whole tree.

## `includeAuthorProfile` (type: `boolean`):

Add the author's karma, account creation date and profile text (`authorKarma`, `authorCreatedAt`, `authorAbout`) to every row.

## Actor input object example

```json
{
  "mode": "search",
  "queries": [
    "web scraping"
  ],
  "type": "story",
  "sort": "relevance",
  "listType": "top",
  "maxItems": 50,
  "includeComments": false,
  "maxCommentsPerStory": 50,
  "includeAuthorProfile": false
}
```

# Actor output Schema

## `items` (type: `string`):

Hacker News stories and comments, one item per row.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "web scraping"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidyharvest/hacker-news-search-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "queries": ["web scraping"] }

# Run the Actor and wait for it to finish
run = client.actor("tidyharvest/hacker-news-search-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "web scraping"
  ]
}' |
apify call tidyharvest/hacker-news-search-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidyharvest/hacker-news-search-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/35y4Yp60MyLwB6W8Y/builds/vjEH8CQK61ehasWmV/openapi.json