# Hacker News Scraper (`scrapyx/hackernews-scraper`) Actor

Stories, comments, users and the live front page from the Hacker News APIs. Time-slices past the 1,000-result ceiling that caps every other HN scraper, and flags upstream's match count as an estimate when it is one — the two search endpoints disagree about it by up to 111x.

- **URL**: https://apify.com/scrapyx/hackernews-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** Developer tools, News, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Hacker News Scraper

Stories, comments, users and the live ranked lists from Hacker News, straight
from the two public APIs — `hn.algolia.com` (full-text search over every item
ever posted) and `hacker-news.firebaseio.com` (the canonical item store and
the live front page). No login, no API key, no browser.

### What it collects

| Mode | What you get |
| --- | --- |
| `search` | Full-text search over ~46M items. Filter by kind (story, comment, poll, job, Ask HN, Show HN, Launch HN, front page), by author, by date range, by minimum points or comments. |
| `items` | The **entire comment thread** for an item ID — pre-joined by upstream, so a 72-node thread costs one request, not 72. |
| `users` | Account profiles: karma, account age, submission count, and optionally the newest N submissions hydrated into full rows. |
| `lists` | The live ranked lists — front page, newest, best, Ask HN, Show HN and jobs — with list position preserved. |

### The thing that makes this different: it gets past 1,000 results

Hacker News caps **every** search at 1,000 results. `nbPages` is always
`ceil(1000 / hitsPerPage)`, never the real page count, and the moment you page
past the ceiling the API returns HTTP 200 with `hits: []` **and `nbHits: 0`** —
so a scraper that reads its total off the last page it fetched reports "0
results" for a query with 543,213 matches.

Sorted by date, this actor time-slices the query automatically on
`created_at_i`, because the ceiling is per *time window*, not per query.
Measured: **8,000 stories in 8 requests, zero duplicates**, where a naive
client stops at 1,000.

Sorted by relevance there is no workaround — slicing relevance would rank
within windows instead of overall — so the run stops at 1,000 and the summary
says `depthCapped: true` rather than pretending.

### Upstream's match count is a guess, and this actor says so

`nbHits` is an Algolia estimate unless `exhaustiveNbHits` is true, and the two
search endpoints disagree about it **by up to 111×** on the *same* filter:

| Filter | `/search` | `/search_by_date` | Ratio |
| --- | ---: | ---: | ---: |
| `tags=comment` | 316,444 | 35,112,651 | 0.01× |
| `tags=story` | 44,067,014 | 3,978,936 | 11.08× |
| `tags=ask_hn` | 2,247,258 | 180,703 | 12.44× |
| *(no tags)* | 45,837,157 | 45,837,157 | 1.00× ✅ exact |

So the total travels with `upstreamTotalIsExact`, and when it is *not* exact
the actor spends one extra request on the other endpoint and publishes that
number too, plus the ratio. When the total **is** exact it is trustworthy:
verified against a real walk over a fixed 24-hour window — upstream said 742,
the other endpoint said 742, and 742 unique stories came back.

`reachableFraction` is only computed against an exact total. A fraction of a
number that can be wrong by two orders of magnitude is not a measurement.

### Other things it gets right

- **Four ways to say "does not exist", only one of them a 404.** Algolia
  answers an unknown *item* with 404 but an unknown *username* with **HTTP
  500**; Firebase answers both with **HTTP 200 and a body of `null`**. A
  generic retry ladder burns four attempts and 14s of backoff on that 500. All
  four become a clean `ERROR` row naming the cause.
- **Deleted and dead items.** 8.3% of recent Firebase items are one or the
  other (deleted items are a bare tombstone with no author or text; dead ones
  read `[flagged]`). Algolia indexes neither, so the two hosts genuinely
  disagree about what exists — rows carry `isDeleted` / `isDead` and the
  summary counts them, so a gap is explained rather than mysterious.
- **Job posts have no real score.** Firebase returns `score: 1` on *every*
  job post (20/20 sampled) and Algolia usually omits `points` entirely. The
  number is passed through with `pointsAreMeaningful: false` beside it, so job
  ads don't sort as though they'd been downvoted to 1.
- **There is no field set common to all five item types.** A job post has no
  `points` and no `story_id`; a poll has no `url`; an Ask HN story has
  `url: null` with the body in `story_text`. Item text arrives under four
  different keys — normalised into one `text` field, with `textField` naming
  where it came from so a null reads as "this type has no body" rather than
  "the body went missing".
- **The time cursor is inclusive (`<=`), never `<`.** Items share whole-second
  timestamps, and a strict `<` drops every tie on the boundary second without
  saying so. The handful of known repeats this causes are deduped and counted
  in `duplicateItemsDropped`.

### Output

One dataset with `recordType` telling the rows apart:

- `SEARCH_SUMMARY` — one per query: upstream's total and whether it is exact,
  time windows fetched, ceiling and cap flags, the window actually covered,
  duplicates dropped, deleted/dead counts, and anything not found.
- `ITEM` — one per story, comment, poll, poll option or job.
- `USER` — one per account.
- `ERROR` — one per input that failed, so **every input maps to at least one row**.

Every row carries `_input`, `_source`, `_scrapedAt` and `recordType`, and
upstream's own fields are passed through verbatim alongside the normalised ones.

### Anti-bot and limits

**No WAF of any kind.** Seven TLS profiles (chrome124, chrome131, chrome136,
safari17\_0, edge101, firefox133, chrome99\_android) all returned 200 cold with
byte-identical bodies on both hosts, and so did a naked `urllib.request` with a
`python-urllib/3` User-Agent. 60 back-to-back requests returned 200 sixty
times, in 28.6s, with no rate-limit headers sent at all. A proxy is optional
and **off by default**.

Known limits:

- 1,000 results per time window (worked around for date sort, unavoidable for
  relevance sort).
- `hitsPerPage` above 1,000 is silently clamped, with no error.
- `front_page` is a live snapshot of roughly 30 items and the exact number
  moves (30 one hour, 29 the next); it reports its own count exactly.
- Firebase ranked lists are as long as HN has live items — measured 500, 500,
  200, 165, 31 and 28 — not a fixed size.

### Policy

`hn.algolia.com` serves no robots.txt (HTTP 404 — RFC 9309 §2.3.1.3 treats an
unavailable robots.txt as no restriction). `hacker-news.firebaseio.com` serves
`Disallow: /` with an explicit `Allow: /*.json$`; every path this actor touches
ends in `.json`. Neither names an AI crawler. `news.ycombinator.com` itself is
**not** scraped — it asks for `Crawl-delay: 30` and everything it renders is
available from the APIs without the wait.

Hacker News content is user-submitted; check HN's own terms before
redistributing it.

# Actor input Schema

## `mode` (type: `string`):

search = full-text search over every HN item ever posted. items = full comment threads for specific item IDs. users = account profiles and their submissions. lists = the live front page and the other ranked lists.

## `queries` (type: `array`):

Search terms for mode='search'. Each runs as its own query with its own summary row. Leave empty and set tags instead to list everything of one kind (e.g. every Show HN post in a date range).

## `tags` (type: `array`):

Restrict to certain kinds of item: story, comment, poll, pollopt, job, ask\_hn, show\_hn, launch\_hn, front\_page. Also accepts author\_<username> and story\_<id>. Several entries are ANDed. Note front\_page is a live snapshot of roughly 30 items — the exact count moves (30 one hour, 29 the next) and is reported in the summary, so it can never return much more than that however deep you page.

## `sortBy` (type: `string`):

date = newest first, and the ONLY order that can be walked past 1,000 results (the actor time-slices the query automatically; 8,000 stories took 8 requests). relevance = best match first, but Hacker News caps it at 1,000 results and no workaround exists, so the summary reports depthCapped.

## `dateFrom` (type: `string`):

Only return items created at or after this date. Also accepts a raw epoch second.

## `dateTo` (type: `string`):

Only return items created at or before this date. Also accepts a raw epoch second.

## `minPoints` (type: `integer`):

Only return items with at least this many points. Job posts carry no points at all and are excluded by any value here.

## `minComments` (type: `integer`):

Only return stories with at least this many comments. Applies to stories only — comments themselves have no comment count.

## `itemIds` (type: `array`):

For mode='items'. A number (8863) or a news.ycombinator.com/item?id=... URL. Each fetch returns the whole pre-joined comment tree in ONE request.

## `usernames` (type: `array`):

For mode='users'. A handle (pg) or a news.ycombinator.com/user?id=... URL. Read from the Firebase API, which carries the account age and the full submission list — the Algolia user endpoint returns only three fields and no submissions.

## `lists` (type: `array`):

For mode='lists'. Any of: topstories (the front page, in order), newstories, beststories, askstories, showstories, jobstories. List position is preserved as listRank. These are as long as HN has live items — measured 500, 500, 200, 28, 165 and 31 — not a fixed size.

## `includeComments` (type: `boolean`):

Emit every comment as its own row, with treeDepth and rootStoryId. In search mode this costs one extra request per story returned; in items mode the tree is what you asked for. A busy thread can be thousands of rows.

## `maxCommentDepth` (type: `integer`):

Stop descending a comment tree at this depth (0 = the story only, 1 = top-level replies). Leave empty for the whole tree.

## `includeTextHtml` (type: `boolean`):

HN item text is HTML with entity-escaped punctuation. A readable plain-text version is always emitted as `text`; turn this on to also keep the original markup as `textHtml`.

## `includeSubmittedIds` (type: `boolean`):

users mode only. Off by default because the list can be enormous — pg has 15,565 IDs. submissionCount is always reported either way.

## `maxSubmissionsPerUser` (type: `integer`):

users mode only. Hydrates the user's newest N submissions into full ITEM rows, one request each. 0 fetches none.

## `maxResultsPerQuery` (type: `integer`):

Set 0 for unlimited. With sortBy='date' the actor time-slices past Hacker News's 1,000-result ceiling, so this really is the only limit. With sortBy='relevance' nothing above 1,000 is reachable and the summary says so.

## `pageSize` (type: `integer`):

Between 1 and 1000. Values above 1000 are silently clamped by Hacker News, with no error. Leave at 1000 for the fewest requests.

## `maxConcurrency` (type: `integer`):

Requests in flight at once. Both APIs are fast and unmetered; 60 back-to-back requests returned 200 sixty times with no throttling.

## `minRequestInterval` (type: `integer`):

Politeness pacing shared across all workers. 0 uses the built-in default.

## `proxyConfiguration` (type: `object`):

Optional and OFF by default. Both API hosts are public and keyless and showed no WAF — all seven TLS profiles plus a plain urllib request returned identical responses.

## Actor input object example

```json
{
  "mode": "search",
  "queries": [
    "show hn",
    "postgres performance"
  ],
  "tags": [
    "show_hn"
  ],
  "sortBy": "date",
  "dateFrom": "2026-01-01",
  "dateTo": "2026-08-01",
  "itemIds": [
    "8863",
    "https://news.ycombinator.com/item?id=1"
  ],
  "usernames": [
    "pg",
    "dang"
  ],
  "lists": [
    "topstories",
    "showstories"
  ],
  "includeComments": false,
  "includeTextHtml": false,
  "includeSubmittedIds": false,
  "maxSubmissionsPerUser": 0,
  "maxResultsPerQuery": 200,
  "pageSize": 1000,
  "maxConcurrency": 4,
  "minRequestInterval": 0,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "rust"
    ],
    "tags": [
        "story"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/hackernews-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["rust"],
    "tags": ["story"],
}

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/hackernews-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "rust"
  ],
  "tags": [
    "story"
  ]
}' |
apify call scrapyx/hackernews-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/hackernews-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Yzfzs95qLLsYLQtE9/builds/vm8rELGkfEAb6YSBc/openapi.json