# Hacker News Scraper - top, new and search stories, comments (`retrainmap/hackernews-stories`) Actor

Hacker News stories from the official public APIs: the top, new and best lists, or a full-text search by keyword, tag and date. One row per story with id, title, URL, domain, points, author, comment count, created time and HN link; optionally top comments. For trend trackers and researchers.

- **URL**: https://apify.com/retrainmap/hackernews-stories.md
- **Developed by:** [RetrainMap Team](https://apify.com/retrainmap) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 records

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Hacker News stories + comments — Algolia + Firebase APIs

One row per Hacker News story from the official public APIs: the **top / new / best** lists
(Firebase API) or a **full-text search** (Algolia API), with id, title, URL, domain, points,
author, comment count, created time, HN link and the text of Ask/Show posts. Optionally the
**top N comments per story** (HN's own ranked order) as extra rows. Nothing is scraped from
news.ycombinator.com — only the two APIs are read.

**The default input (top stories, 100 rows) succeeds within 5 minutes and returns rows** —
Apify auto-tests it daily.

### Data source and status

| | |
|---|---|
| Publisher | Y Combinator (Hacker News). Search index operated by Algolia |
| APIs | https://hacker-news.firebaseio.com/v0 — public, no key (documentation: https://github.com/HackerNews/API); https://hn.algolia.com/api/v1 — public, no key (documentation: https://hn.algolia.com/api) |
| What this Actor reads | Firebase: `/v0/topstories.json`, `/v0/newstories.json`, `/v0/beststories.json` and `/v0/item/{id}.json`. Algolia: `/search`, `/search_by_date` (query, tags, numericFilters on `created_at_i`, 100 hits per page) and `/items/{id}` for comment trees. At most one request per `request_interval_ms` (default 100 ms) |
| Refresh cadence | Both APIs are **live** — the Firebase lists change within seconds of the site; the Algolia index is updated continuously. The Actor reads live on every run; `fetched_at` on each row is the fetch time |
| Terms / attribution | The Firebase API is published by Y Combinator under the MIT licence (https://github.com/HackerNews/API/blob/master/LICENSE); the Algolia HN Search API is free for use with attribution ("Search by Algolia", per https://hn.algolia.com/api). User-submitted content belongs to its authors; please credit Hacker News and Algolia as the source and respect Algolia's limit of 10,000 requests per hour per IP |
| Known caveats | Firebase lists hold 500 (top, new) or 200 (best) ids; Algolia serves at most 1,000 hits per query — narrow the date range for more; `url`/`domain` are empty for Ask HN, jobs and polls; `points`/`num_comments` are null for comments; deleted or dead items are skipped and counted in the run summary; `text` is the item HTML converted to plain text and cut at 3,000 characters (`text_truncated` says so) |

Honesty note: the rows are the APIs' own fields renamed and flattened — nothing is inferred,
summarised or classified by this Actor, and a run stops with an error rather than guess when
an API's shape changes.

Identification: the requests carry a product-token User-Agent and the operator's contact
address in the standard `From:` header (RFC 9110 §10.1.2).

### Input

| Field | Type | Meaning |
|---|---|---|
| `mode` | `top` / `new` / `best` / `search` | Lists from Firebase (HN's order) or an Algolia search. Default `top` |
| `query` | string | Search mode: full-text query (title, URL, author, text). Empty = no term |
| `tags` | array | Search mode: `story`, `comment`, `ask_hn`, `show_hn`, `front_page`; several are ORed. Empty = `story` |
| `search_sort` | `relevance` / `date` | Search mode: `/search` or `/search_by_date` (newest first) |
| `date_from` / `date_to` | `YYYY-MM-DD` | Search mode, UTC, inclusive |
| `include_comments` | boolean | Add up to `comments_per_story` top-level comment rows after each story (ranked order). Default false |
| `comments_per_story` | integer | Default 10, at most 100 |
| `max_records` | integer | Total rows (stories + comments). Default 200 (prefilled 100), at most 5,000 |
| `request_interval_ms` | integer | Default 100 (floor 50) |
| `contact_email` | string | Sent in the `From:` header |

Example — stories about Python since August 2026 with their top 5 comments:

```json
{ "mode": "search", "query": "python", "tags": ["story"], "date_from": "2026-08-01", "include_comments": true, "comments_per_story": 5, "max_records": 30 }
```

### Output (dataset row)

`rank` (position in the list / search results, or among a story's comments), `type` (story /
comment / job / poll / pollopt), `id`, `title`, `url`, `domain`, `points`, `author`,
`num_comments`, `created_at` (UTC), `hn_url`, `text` (Ask/Show/comment text, plain, ≤ 3,000
chars), `text_truncated`, `story_id`, `parent_id`, `story_title` (comments only), `hn_tags`,
`list` (top / new / best / search / comments), `fetched_at`, `source`.

A run summary (filters, list size or Algolia hit count, counts of stories, comments and
skipped items, requests, whether the pay-per-event budget stopped the run) is stored as
`RUN_SUMMARY` in the run's key-value store.

### Pricing (pay per event)

| Event | Price |
|---|---|
| `run-start` — once per run | $0.10 |
| `record` — per row written (story or comment) | $0.005 |

The default top-100 pull costs $0.60; 1,000 rows cost $5.10. Rows stop when your run's
maximum charge is reached; the run summary says so.

### Operator

Steelyard Ventures LLC (RetrainMap) — info@steelyardclinical.com. Not affiliated with
Y Combinator, Hacker News or Algolia. The Actor writes only to its own dataset and
key-value store; it stores no credentials and sends nothing else.

# Actor input Schema

## `mode` (type: `string`):

top / new / best read the official Firebase lists (500, 500 and 200 ids) in HN's own order. search runs a full-text Algolia query using the fields below.

## `query` (type: `string`):

Full-text query for the Algolia API (matches title, URL, author and text). Empty = no term filter (then the tags and date range alone select the rows).

## `tags` (type: `array`):

Algolia item tags; several tags are ORed. story = any story (Ask/Show included), comment = comments, ask\_hn / show\_hn = those story kinds, front\_page = stories currently on the front page. Empty = story.

## `search_sort` (type: `string`):

relevance uses /search (Algolia relevance, then points and comments); date uses /search\_by\_date (newest first).

## `date_from` (type: `string`):

Optional, UTC, inclusive; applied as an Algolia numericFilter on created\_at\_i.

## `date_to` (type: `string`):

Optional, UTC, inclusive (through 23:59:59 of that day).

## `include_comments` (type: `boolean`):

When on, each story row is followed by up to comments\_per\_story rows of its top-level comments in HN's ranked order (one extra Algolia request per story, plus one Firebase request per story in search mode). Comment rows count towards max\_records and are charged as records.

## `comments_per_story` (type: `integer`):

Maximum top-level comments written per story when include\_comments is on.

## `max_records` (type: `integer`):

Stop after this many rows (stories plus comment rows) have been written. The lists hold at most 500 (top, new) or 200 (best) stories; Algolia returns at most 1,000 hits per query.

## `request_interval_ms` (type: `integer`):

Politeness delay towards the two APIs (one request per story in list modes). Floor 50 ms. Algolia's documented limit is 10,000 requests per hour per IP.

## `contact_email` (type: `string`):

Sent in the standard From: request header so the data owner can reach the operator (RFC 9110 s10.1.2).

## Actor input object example

```json
{
  "mode": "top",
  "tags": [
    "story"
  ],
  "search_sort": "relevance",
  "include_comments": false,
  "comments_per_story": 10,
  "max_records": 100,
  "request_interval_ms": 100,
  "contact_email": "info@steelyardclinical.com"
}
```

# Actor output Schema

## `results` (type: `string`):

One row per story (and optionally per top-level comment): id, type, title, URL, domain, points, author, comment count, created time, HN link, text.

## `results_csv` (type: `string`):

The same rows as a CSV file.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "tags": [
        "story"
    ],
    "max_records": 100
};

// Run the Actor and wait for it to finish
const run = await client.actor("retrainmap/hackernews-stories").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "tags": ["story"],
    "max_records": 100,
}

# Run the Actor and wait for it to finish
run = client.actor("retrainmap/hackernews-stories").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "tags": [
    "story"
  ],
  "max_records": 100
}' |
apify call retrainmap/hackernews-stories --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,retrainmap/hackernews-stories"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ouYKZQQuYS6ScGdoR/builds/CaxwziqkUXSp77zKT/openapi.json