# Hacker News Scraper — Full Archive | $4/1K (`datawell/hackernews-scraper`) Actor

Search the whole Hacker News archive past the 1000-result cap that stops every other scraper. Stories and comments by keyword, points and date. Front page feeds, comment trees as clean plain text, user profiles. For market research and developer sentiment analysis.

- **URL**: https://apify.com/datawell/hackernews-scraper.md
- **Developed by:** [Finjard Neumeier](https://apify.com/datawell) (community)
- **Categories:** News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

$4.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Hacker News Scraper

Search the entire Hacker News archive, pull the current front page, expand whole
comment trees, and look up user profiles. No API key, no login.

***

### Why this scraper

**Search past the 1,000-result cap.** Hacker News search returns at most 50 pages
— 1,000 results — no matter how many matches exist. Ask for 10,000 posts about
Rust and every scraper built the obvious way stops at 1,000. This one walks a
moving time window instead and keeps going, verified duplicate-free.

**Comments as readable text.** Hacker News stores comments as HTML with escaped
entities. Exported raw, a CSV is full of `&#x27;` and `<p>` tags. Here it arrives
as plain text.

**Breadth-first comment trees.** When you limit how many comments to fetch, you
want the top of the discussion, not one deep thread. Comments come back level by
level, each row tagged with its depth.

***

### What you can extract

| Mode | What you get |
|---|---|
| **Search the whole archive** | Every story or comment matching a keyword, filterable by points, comment count and date range |
| **Current front page or list** | Top, newest, best, Ask HN, Show HN or job listings as they stand right now |
| **Comments on a story** | The full comment tree, flattened, with depth and parent for each reply |
| **User profiles** | Karma, join date, submission count and bio |

***

### Quick start

1. Pick **Search the whole archive**.
2. Type a keyword, or leave it empty and set **Minimum points** to 100.
3. Set **Maximum results**, click **Start**, download as CSV or JSON.

***

### Example inputs

**Everything ever posted about a topic**

```json
{
  "mode": "search",
  "searchQuery": "postgres",
  "sortBy": "date",
  "maxItems": 10000
}
```

**Only stories that actually landed**

```json
{
  "mode": "search",
  "searchQuery": "startup",
  "minPoints": 100,
  "maxItems": 2000
}
```

**Track a competitor being discussed**

```json
{
  "mode": "search",
  "searchQuery": "\"your product name\"",
  "contentType": "comment",
  "maxItems": 1000
}
```

**Show HN launches in a date range**

```json
{
  "mode": "search",
  "contentType": "show_hn",
  "since": "2026-01-01",
  "until": "2026-06-30",
  "maxItems": 3000
}
```

**Today's front page**

```json
{
  "mode": "frontPage",
  "listName": "topstories",
  "maxItems": 30
}
```

**A whole discussion**

```json
{
  "mode": "comments",
  "storyIds": ["https://news.ycombinator.com/item?id=49195231"],
  "maxDepth": 10,
  "maxItems": 2000
}
```

***

### Output fields

| Field | Description |
|---|---|
| `id`, `type` | Item ID and whether it is a story, comment, poll or job |
| `title` | Story headline; empty for comments |
| `url`, `domain` | The linked article and its domain |
| `hnUrl` | Link to the Hacker News discussion |
| `text` | Comment or self-post content, as plain text |
| `author` | Who posted it |
| `points`, `numComments` | Upvotes and comment count |
| `createdAt`, `createdAtTimestamp` | When, as ISO date and Unix timestamp |
| `storyId`, `parentId` | For comments: which story and which parent |
| `depth` | Comment mode: 1 is a top-level reply |
| `childCount` | Direct replies to this item |
| `tags` | Categories such as story or show\_hn |
| `username`, `karma`, `about`, `submissionCount` | User mode fields |

***

### Input notes

- **Stories** accept an item ID or a full `news.ycombinator.com` link.
- **Sort order** matters: *newest first* pages through the entire archive,
  *most relevant* stops at 1,000 results — that is a limit of the search backend,
  not of this Actor.
- **Several stories** split the `maxItems` budget evenly between them.
- **Quoted phrases** work in the search query: `"rust vs go"`.

***

### Limits

- **Points are snapshots.** Scores change; a story scraped today may differ tomorrow.
- **Relevance sort caps at 1,000.** Use *newest first* for full coverage.
- **Deleted and dead comments are skipped** rather than returned as empty rows.
- **Very deep trees** on huge threads take time — one request per comment.

***

### FAQ

**Do I need an API key?**
No. Hacker News runs both of its interfaces openly.

**Can I really get more than 1,000 search results?**
Yes, with *newest first*. That is the main thing this Actor does differently.

**How far back does the archive go?**
To 2007, when Hacker News launched.

**Can I search comments, not just stories?**
Yes — set **Stories or comments** to Comments.

**What formats can I export?**
CSV, Excel, JSON, XML, or straight from the Apify API.

***

### Legal and responsible use

This Actor reads publicly available data through Hacker News's own public
interfaces. It does not bypass authentication or access private content.

Usernames and comment text are personal data under GDPR. You remain responsible
for how you store, analyse and republish the output.

# Actor input Schema

## `mode` (type: `string`):

Pick what you want to collect. The fields below apply depending on this choice.

## `searchQuery` (type: `string`):

Keyword or phrase to search for across every Hacker News post since 2007. Leave empty to get everything matching your filters.

## `contentType` (type: `string`):

What kind of item to search for. 'Ask HN' and 'Show HN' are separate categories on Hacker News.

## `sortBy` (type: `string`):

'Newest first' can page through the entire archive. 'Most relevant' stops after 1000 results — a limit of the search backend.

## `minPoints` (type: `integer`):

Only items with at least this many upvotes. Set to 100 to get only stories that actually landed.

## `minComments` (type: `integer`):

Only items with at least this many comments — a good proxy for how contested a topic was.

## `since` (type: `string`):

Only items posted after this date. Format: 2026-01-31

## `until` (type: `string`):

Only items posted before this date. Format: 2026-01-31

## `listName` (type: `string`):

Which curated list to fetch. Used in front page mode.

## `storyIds` (type: `array`):

Item IDs (43217891) or Hacker News links (https://news.ycombinator.com/item?id=43217891). Used in comment mode.

## `maxDepth` (type: `integer`):

How many levels of replies to follow. Comments are collected breadth-first, so the top levels are complete before deeper ones.

## `usernames` (type: `array`):

Hacker News usernames to look up. Used in user mode.

## `maxItems` (type: `integer`):

Total number of records to collect. With several stories, the budget is split evenly between them.

## `requestDelaySeconds` (type: `integer`):

Seconds to wait between calls. Raise this for very large runs.

## Actor input object example

```json
{
  "mode": "search",
  "searchQuery": "rust",
  "contentType": "story",
  "sortBy": "date",
  "listName": "topstories",
  "maxDepth": 10,
  "maxItems": 100,
  "requestDelaySeconds": 0
}
```

# Actor output Schema

## `results` (type: `string`):

Scraped stories, comments or users. Story rows carry title, points, comment count and linked domain; comment rows carry plain-text content and tree depth.

## `resultsCsv` (type: `string`):

The same rows as a CSV download, ready for Excel or Google Sheets.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQuery": "rust"
};

// Run the Actor and wait for it to finish
const run = await client.actor("datawell/hackernews-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "searchQuery": "rust" }

# Run the Actor and wait for it to finish
run = client.actor("datawell/hackernews-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQuery": "rust"
}' |
apify call datawell/hackernews-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=datawell/hackernews-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/hPRrCLQ96oOOH2mnw/builds/lSehUPLzIVn74Uci1/openapi.json