# The Better Hacker News Scraper (`betterscrapers/the-better-hacker-news-scraper`) Actor

Affordable, fast Hacker News scraper. Export HN stories, comments, points, authors & links to JSON/CSV via the public Algolia HN Search API. HTTP-only, no proxy, low compute.

- **URL**: https://apify.com/betterscrapers/the-better-hacker-news-scraper.md
- **Developed by:** [Better Scrapers](https://apify.com/betterscrapers) (community)
- **Categories:** Developer tools, News, Other
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.89 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

## The Better Hacker News Scraper — Fast, Affordable HN Story & Comment Extractor

Scrape **Hacker News** at scale without the cost. The Better Hacker News Scraper is an affordable, efficient **Hacker News scraper / HN data extractor** that searches and exports **Hacker News stories, comments, Ask HN and Show HN posts, points, authors, and links** as structured JSON, CSV, or Excel. It runs entirely over plain HTTP against the **public Algolia Hacker News Search API** — no login, no API key, and **no proxy required** — so each run uses **fewer compute units** and stays cheap even when you pull thousands of records.

Whether you want to **monitor Hacker News mentions of your product**, build an **HN dataset for research**, track **trending tech news and Show HN launches**, or feed a **news aggregator or LLM pipeline**, this scraper gives you clean, normalized Hacker News data in seconds.

### What it extracts

For every matching Hacker News item, the scraper returns:

- **id** — the Hacker News item id (Algolia `objectID`)
- **title** — the story/post title (falls back to the parent story title for comments)
- **url** — the outbound link (falls back to the HN discussion URL for text posts and comments)
- **author** — the HN username of the submitter/commenter
- **points** — the story score / upvote count
- **numComments** — number of comments on the story
- **createdAt** — submission time as an ISO 8601 timestamp
- **text** — clean plain-text body of Ask HN / Show HN posts and comments (HTML tags stripped, entities decoded). Regular link stories have no body text on Hacker News, so this is `null` for them — that's expected, not missing data (the content lives at `url`).
- **hnUrl** — the canonical Hacker News discussion URL (`https://news.ycombinator.com/item?id=<id>`)

### Features

- **HTTP-only, no browser** — talks directly to the JSON API, so it is fast and light on compute.
- **No proxy required** — the Algolia HN Search API is public; runs work with zero proxy cost by default.
- **Full-text search** across all of Hacker News history via the `query` field.
- **Filter by content type** — stories, comments, polls, Show HN, Ask HN, or the current front page.
- **Newest-first or relevance** — flip `byDate` to switch between `search_by_date` and ranked search.
- **Automatic pagination** — fetches up to `maxPages` pages (100 items per page) and stops early when results run out.
- **Normalized, predictable output** — consistent field names ready for JSON, CSV, Excel, or an API dataset.
- **Robust parsing** — the core parser is a pure, unit-tested function with sensible fallbacks for missing fields.

### Input

| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `query` | string | `""` | Full-text search term. Leave empty to fetch the latest/front-page items for the selected tag. |
| `tags` | string | `"story"` | Content type to return: `story`, `comment`, `poll`, `show_hn`, `ask_hn`, `front_page`. You can also target a user with `author_<username>`. |
| `byDate` | boolean | `false` | When `true`, uses the `search_by_date` endpoint for newest-first results. When `false`, results are ranked by relevance/popularity. |
| `maxPages` | integer | `5` | How many result pages to fetch (100 items per page, 1–200). More pages return more results and use slightly more compute. |
| `proxyConfiguration` | object | `{ "useApifyProxy": false }` | Optional. The API is public and works with no proxy — leave off to keep runs cheapest. Enable a datacenter proxy only if you hit rate limits at very high volume. |

#### Example input

```json
{
    "query": "openai",
    "tags": "story",
    "byDate": false,
    "maxPages": 3
}
````

### Output

Each dataset item looks like this:

```json
{
    "id": "38912345",
    "title": "Show HN: A tiny, fast Hacker News scraper",
    "url": "https://example.com/show-hn-scraper",
    "author": "pg",
    "points": 412,
    "numComments": 87,
    "createdAt": "2024-01-15T09:30:00.000Z",
    "text": null,
    "hnUrl": "https://news.ycombinator.com/item?id=38912345"
}
```

For an Ask HN or comment with no outbound link, `url` falls back to the `hnUrl` discussion page and `text` contains the post/comment body.

### Example use cases

- **Brand & product monitoring** — track every Hacker News mention of your company, tool, or competitor.
- **Tech trend research** — build datasets of trending HN stories, points, and comment volume over time.
- **Show HN / Ask HN analysis** — study launch posts, engagement, and community feedback.
- **Newsletter & aggregator feeds** — pull the latest front-page or newest HN stories into your own product.
- **LLM & RAG pipelines** — collect Hacker News discussions as clean text for summarization or Q\&A.
- **Author tracking** — follow everything a specific HN user has submitted or commented.
- **Academic & market research** — export large, structured HN corpora for analysis in Excel, pandas, or BigQuery.

### How it works

1. The actor reads your input and builds a request to the public Algolia HN Search API — either `/search` (ranked) or `/search_by_date` (newest first), with your `query` and `tags`.
2. A lightweight `CheerioCrawler` fetches each page of JSON over HTTP (no browser is launched).
3. The pure `parseHits` parser normalizes every hit into the clean output shape above, filling sensible fallbacks for missing titles, URLs, and timestamps.
4. Results are pushed to the dataset, and the next page is enqueued until `maxPages` is reached or the results are exhausted.

### Why it's efficient and affordable

- **No browser, no rendering** — it never spins up Chromium, so runs finish quickly and consume **fewer compute units per run**.
- **No proxy cost by default** — the Algolia HN Search API is public and free, so there is no residential-proxy bill; the default configuration uses no proxy at all.
- **Compact JSON payloads** — the API returns structured JSON, so there is no HTML to download or parse and less bandwidth per record.
- **Early-exit pagination** — the crawler stops as soon as results run out, so short queries do not waste requests.

Together these choices keep the **cost per 1,000 Hacker News records low**, which makes this a practical, **low-cost Hacker News scraper** for both one-off exports and scheduled monitoring.

### FAQ

**Do I need a Hacker News or Algolia API key?**
No. The scraper uses the public Algolia HN Search API, which requires no key, no account, and no login.

**Do I need proxies to scrape Hacker News?**
No. The API is public and the actor works with no proxy, which is a big part of why it is so cheap to run. A datacenter proxy is available as an option only if you hit rate limits at very high volume.

**How many results can I get?**
Each page returns up to 100 items and you can fetch up to 200 pages per run (`maxPages`), so a single run can return tens of thousands of Hacker News stories or comments.

**Can I scrape comments as well as stories?**
Yes. Set `tags` to `comment` for comments, `ask_hn` or `show_hn` for those post types, or `front_page` for the current front page.

**What formats can I export?**
Any format Apify datasets support — JSON, CSV, Excel, HTML, RSS, or via the API — so the data drops straight into spreadsheets, databases, or downstream code.

### Legal & ethical note

This actor retrieves **publicly available** Hacker News data through the official public Algolia HN Search API. Please scrape responsibly: request only the data you need, respect Hacker News' and Algolia's Terms of Service and `robots.txt`, avoid excessive request rates, and comply with all applicable laws and data-protection regulations (including how you store and use any personal data such as usernames). You are responsible for how you use the data you collect.

# Actor input Schema

## `query` (type: `string`):

Full-text search term. Leave empty to fetch the latest/front-page items for the selected tag.

## `tags` (type: `string`):

Which HN content to return. Common values: story, comment, poll, show\_hn, ask\_hn, front\_page. You can also target a specific author with author\_<username>.

## `byDate` (type: `boolean`):

When enabled, uses the search\_by\_date endpoint to return the most recent matches first. When disabled, results are ranked by relevance/popularity.

## `maxPages` (type: `integer`):

How many result pages to fetch (100 items per page). Higher values return more results and use slightly more compute.

## `proxyConfiguration` (type: `object`):

Optional. The Algolia HN Search API is public and this actor works with no proxy at all — leave this off to keep runs cheapest. Enable a datacenter proxy only if you run at very high volume and hit rate limits.

## Actor input object example

```json
{
  "query": "rust programming",
  "tags": "story",
  "byDate": false,
  "maxPages": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "openai",
    "tags": "story",
    "byDate": false,
    "maxPages": 5,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("betterscrapers/the-better-hacker-news-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "openai",
    "tags": "story",
    "byDate": False,
    "maxPages": 5,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("betterscrapers/the-better-hacker-news-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "openai",
  "tags": "story",
  "byDate": false,
  "maxPages": 5,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call betterscrapers/the-better-hacker-news-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=betterscrapers/the-better-hacker-news-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "The Better Hacker News Scraper",
        "description": "Affordable, fast Hacker News scraper. Export HN stories, comments, points, authors & links to JSON/CSV via the public Algolia HN Search API. HTTP-only, no proxy, low compute.",
        "version": "0.1",
        "x-build-id": "M3Q9CCByNLYhVjJi1"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/betterscrapers~the-better-hacker-news-scraper/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-betterscrapers-the-better-hacker-news-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/betterscrapers~the-better-hacker-news-scraper/runs": {
            "post": {
                "operationId": "runs-sync-betterscrapers-the-better-hacker-news-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/betterscrapers~the-better-hacker-news-scraper/run-sync": {
            "post": {
                "operationId": "run-sync-betterscrapers-the-better-hacker-news-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "properties": {
                    "query": {
                        "title": "Search query",
                        "type": "string",
                        "description": "Full-text search term. Leave empty to fetch the latest/front-page items for the selected tag."
                    },
                    "tags": {
                        "title": "Content type (tag)",
                        "enum": [
                            "story",
                            "comment",
                            "poll",
                            "show_hn",
                            "ask_hn",
                            "front_page"
                        ],
                        "type": "string",
                        "description": "Which HN content to return. Common values: story, comment, poll, show_hn, ask_hn, front_page. You can also target a specific author with author_<username>.",
                        "default": "story"
                    },
                    "byDate": {
                        "title": "Sort by newest first",
                        "type": "boolean",
                        "description": "When enabled, uses the search_by_date endpoint to return the most recent matches first. When disabled, results are ranked by relevance/popularity.",
                        "default": false
                    },
                    "maxPages": {
                        "title": "Max pages",
                        "minimum": 1,
                        "maximum": 200,
                        "type": "integer",
                        "description": "How many result pages to fetch (100 items per page). Higher values return more results and use slightly more compute.",
                        "default": 5
                    },
                    "proxyConfiguration": {
                        "title": "Proxy configuration",
                        "type": "object",
                        "description": "Optional. The Algolia HN Search API is public and this actor works with no proxy at all — leave this off to keep runs cheapest. Enable a datacenter proxy only if you run at very high volume and hit rate limits.",
                        "default": {
                            "useApifyProxy": false
                        }
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```