# Hacker News Scraper: Stories, Comments, Who Is Hiring (`transparent_meteorite/hacker-news-search-scraper`) Actor

Scrape Hacker News stories, comments, users, front page, Ask HN, Show HN and monthly Who is hiring threads from the official Algolia and Firebase APIs. Points, comments, domains and dates. No login.

- **URL**: https://apify.com/transparent_meteorite/hacker-news-search-scraper.md
- **Developed by:** [Detroit Web Agent - Open Data](https://apify.com/transparent_meteorite) (community)
- **Categories:** Social media, News
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.09 / 1,000 record returneds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Hacker News Scraper: Stories, Comments, Who Is Hiring

Use this to search Hacker News by keyword, pull the front page and the Top, New, Best, Ask HN, Show HN and Jobs lists, fetch every comment of a story, look up a user and their submissions, and extract the monthly "Who is hiring?" threads, with points, comment counts, domains, authors and ISO timestamps. It reads the official Algolia Hacker News Search API and the official Hacker News Firebase API. No login, no cookies, no API key, no proxy.

Every row carries up to 34 fields: id, HN page URL, title, outbound URL, domain, plain-text body, author and author URL, points, comment count, ISO creation time and Unix timestamp, parent and story ids, story title for comments, nesting depth, list rank, child count, matched query and, for Who is hiring posts, a parsed headline, remote flag, apply emails and apply links.

### Run in 30 seconds

1. Open the **Input** tab. The prefilled values (mode **Search**, term `web scraping`, 25 results) are a ready-to-run test.
2. Click **Start**. The test run finishes in a few seconds.
3. Open the **Stories** table in the Output tab, or download the dataset as JSON, CSV, Excel or HTML.
4. Change **What to scrape** to switch between search, lists, story comments, users and Who is hiring.

### What does this Actor do?

| Mode | What you get | Typical use |
|---|---|---|
| Search | Stories, comments, Ask HN, Show HN, jobs or polls matching keywords, filtered by points, comments, author and date | Brand and competitor monitoring, research, trend tracking |
| List | Front page (with points), Top, New, Best, Ask HN, Show HN or Jobs, ranked | Daily digests, launch watching |
| Story comments | The story plus every comment, flattened with a depth number | Sentiment analysis, thread summaries, datasets |
| User | Profile (karma, about, join date, submission count) plus their stories and/or comments | Founder and author research |
| Who is hiring | The latest or any month's Who is hiring, Who wants to be hired or Freelancer thread, one row per post, with keyword and remote filters | Recruiting, job-market data, lead lists |

Search can go past Algolia's 1,000-hit cap: in date order the Actor keeps walking the time window backwards, so a broad query can return thousands of distinct rows.

### Why use it

- **One Actor for every Hacker News job.** Search, front page and lists, full comment trees, users and hiring threads share one input form, one dataset schema and one bill.
- **Who is hiring parsed for you.** Each post gets a headline, a remote flag, apply emails and apply links, and you can filter by keyword (any or all) and remote only.
- **Both official APIs.** Algolia gives ranked search with numeric filters; Firebase gives the live Top, New, Best, Ask, Show and Jobs lists. Neither needs a key.
- **Clean text.** HTML in posts and comments is converted to plain text with links kept.
- **Predictable cost.** One flat price per row returned. Rows skipped by your filters and duplicates are never charged, and the Actor respects your "Maximum cost per run" setting exactly.

### Input

- **What to scrape**: Search, List, Story comments, User or Who is hiring.
- **Search terms** and **Content type** (stories, comments, Ask HN, Show HN, front page, jobs, polls), **Sort by** relevance or newest, **Minimum points**, **Minimum comments**, **Posted since / until**, **Only from this user**.
- **List**: top, new, best, ask, show, job or front page.
- **Story IDs or URLs**, **Include the story row**, **Max comment depth**.
- **Usernames** and what to return besides the profile.
- **Thread type**, **Month** (`latest`, `2026-10` or `October 2026`), **Keywords**, **Keyword match**, **Remote only**.
- **Max results** and **Include child comment ids**.

### Example input

```json
{ "mode": "search", "searchTerms": ["web scraping"], "contentType": "story", "sortBy": "relevance", "minPoints": 50, "maxItems": 25 }
```

```json
{ "mode": "whoIsHiring", "hiringMonth": "latest", "hiringKeywords": ["rust"], "remoteOnly": true, "maxItems": 100 }
```

### Output sample

```json
{
  "recordType": "story",
  "id": 22180559,
  "hnUrl": "https://news.ycombinator.com/item?id=22180559",
  "title": "Congrats! Web scraping is legal! (US precedent)",
  "url": "https://parsers.me/us-court-fully-legalized-website-scraping-and-technically-prohibited-it/",
  "domain": "parsers.me",
  "author": "ehurynovich",
  "points": 1057,
  "commentCount": 388,
  "createdAt": "2020-01-29T14:06:37.000Z",
  "storyId": 22180559,
  "childCount": 70,
  "query": "web scraping",
  "source": "search"
}
```

Fields that do not apply to a row type are `null`, so every row has the same columns.

### Use cases

- Track mentions of your product, a competitor or a technology and alert on new high-point threads.
- Build a dataset of tech opinions from comment trees for research or model evaluation.
- Pull every Who is hiring post that mentions your stack and is remote, and export the apply emails.
- Rank the domains that reach the front page, or the authors who post the most successful Show HN launches.

### Tips for good results

- Use **Sort by newest first** with **Posted since** for incremental monitoring: store the highest `createdTimestamp` you saw and pass it next time.
- Use **Minimum points** to keep only stories that got traction.
- For comment datasets, set **Max comment depth** to 1 for top-level replies only.
- Who is hiring threads appear on the first working day of each month; use `latest` for the newest.

### Field reference

`recordType`, `id`, `hnUrl`, `title`, `url`, `domain`, `text`, `author`, `authorUrl`, `points`, `commentCount`, `createdAt`, `createdTimestamp`, `parentId`, `storyId`, `storyTitle`, `storyUrl`, `depth`, `rank`, `childCount`, `childIds`, `tags`, `query`, `source`, `userKarma`, `userAbout`, `userAccountCreated`, `userSubmissionCount`, `hiringHeadline`, `isRemote`, `applyEmails`, `applyUrls`, `threadTitle`, `threadMonth`.

### Pricing and cost control

Pay per result: **$0.000186 per row on the free plan, falling to $0.000093 on the Gold plan** (Bronze $0.000171, Silver $0.000145), plus a $0.00005 start fee per run. A 1,000-row run costs about $0.19 on the free plan. There are no subscriptions, proxy fees or hidden compute add-ons: the Actor needs no proxy and runs in 256 MB. To cap spend, set **Max results** and the run-level **Maximum cost per run**; the Actor reads the remaining budget before every row and stops the moment it would be exceeded.

**How this compares with the other Hacker News scrapers (live Store prices read through the Apify API, 2026-10-29, free-plan rate, 1,000-row run including start fees):**

| Actor | Users | Per row | Start fee | Cost for 1,000 rows |
|---|---|---|---|---|
| This Actor | new | $0.000186 | $0.00005 | $0.19 |
| gentle_cloud/hacker-news-scraper | 222 | $0.0002 | none | $0.20 |
| epctex/hackernews-scraper | 196 | $0.0003 | $0.00005 | $0.30 |
| ryanclinton/hackernews-search | 161 | $0.005 | $0.00005 | $5.00 |

At the free-plan rate this is the lowest per-row price in the table. The Actor needs no login or key, and the prefilled test run finishes in a few seconds.

### Limits

- Up to 100,000 rows per run (the **Max results** setting).
- Relevance-ordered search is capped by Algolia at about 1,000 hits per query; newest-first search with a date range goes deeper.
- A run stops starting new work after about 4.5 minutes and keeps what it has delivered.
- Only public Hacker News data is read: no login, no private data. A deleted or dead item that the APIs do not return is skipped.
- The Who is hiring parser extracts headline, remote flag, emails and links from free text with simple rules, so a post written in an unusual format can have empty parsed fields.

### Integrations and scheduling

Schedule the Actor from the Schedules tab (hourly, daily, weekly) to build a monitoring feed. Send results onward with the Apify integrations for Slack, email, Google Sheets, Make, Zapier or n8n, or call the Actor from your own code through the REST API.

### Fair use and data

The Actor reads Hacker News data through the two public, officially documented APIs (the Algolia HN Search API and the Hacker News Firebase API). It does not log in, does not scrape the website's HTML pages, does not read private data, and spaces its requests politely. Posts and comments are public content written by their authors; you remain responsible for using the data in line with the Hacker News terms and applicable privacy law, and for handling usernames and personal text with care. Do not use it for spam or harassment.

### Use with the API, n8n, Make, Zapier or AI agents

**API** (one call, returns the rows as JSON):

```bash
curl -X POST "https://api.apify.com/v2/acts/transparent_meteorite~hacker-news-search-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"search","searchTerms":["web scraping"],"maxItems":25}'
```

The call waits up to 5 minutes. For bigger inputs, start the run with `/runs` instead and read the dataset when it finishes.

**Python**

```python
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("transparent_meteorite/hacker-news-search-scraper").call(run_input={"mode": "search", "searchTerms": ["web scraping"], "maxItems": 25})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row)
```

- **n8n, Make, Zapier:** use the Apify app's "Run Actor" step with actor `transparent_meteorite/hacker-news-search-scraper`, paste the input JSON, then map the dataset items to your sheet or CRM.
- **Clay:** add an HTTP API column that calls the run-sync endpoint above.
- **Schedules:** in Apify Console, Schedules, run it daily or weekly with the same input for a steady feed.
- **AI agents (MCP):** connect the Apify MCP server at `https://mcp.apify.com?tools=transparent_meteorite/hacker-news-search-scraper` to Claude, Cursor or any MCP client, and your agent can run Algolia HN Scraper on request.

### FAQ

**Do I need a Hacker News account or API key?** No. Both APIs are open and keyless.

**How far back does search go?** All the way to 2006. Relevance order is capped by Algolia at about 1,000 hits per query; use newest-first with a date range to go deeper.

**Can it get every comment of a story?** Yes: Story comments returns the whole tree, flattened with a `depth` number and the parent id.

**Does it get the live front page?** Yes. The List mode with Front page uses the same ranking Algolia exposes; Top, New, Best, Ask, Show and Jobs come from the Firebase lists.

**Which month does Who is hiring use?** `latest` picks the newest thread; pass `2026-09` or `September 2026` for an older one.

**What happens when a user or story does not exist?** It is skipped, a warning is logged and nothing is charged for it.

**What happens if I hit my maximum cost per run?** The Actor stops cleanly, keeps what it already delivered, and shows a clear message in the run status.

**Why did I get fewer rows than Max results?** Your terms or filters matched fewer items. The final status message says so.

### Related actors

- [Bluesky Scraper: Posts, Profiles, Followers & Feeds](https://apify.com/transparent_meteorite/bluesky-posts-profiles-scraper) for Bluesky posts and accounts.
- [Show HN & npm PyPI Signals: Hacker News Tracker](https://apify.com/transparent_meteorite/developer-signals) to join Show HN and Launch HN posts with npm and PyPI package data.
- [Lemmy Scraper: Communities, Posts & Comments](https://apify.com/transparent_meteorite/lemmy-communities-posts-scraper) for link-aggregator communities on the fediverse.
- [Mastodon Hashtag Timeline Exporter](https://apify.com/transparent_meteorite/mastodon-posts-hashtags-scraper) for posts and trends from Mastodon instances.

### Support

Found a bug or need a field? Open an issue on the Issues tab with the run URL and you will get a reply there. Feature requests are welcome.

### Changelog

- **0.1**: First build: search, lists, story comments, users and Who is hiring threads.

# Actor input Schema

## `mode` (type: `string`):

Search uses Search terms. List needs a list type. Story comments needs Story IDs. User needs Usernames. Who is hiring needs nothing.

## `searchTerms` (type: `array`):

Keywords or phrases, one search per term. Leave empty to browse everything of the chosen type. Used by Search.

## `maxItems` (type: `integer`):

Total rows to return across all terms, ids or users. You are charged per row returned.

## `contentType` (type: `string`):

What to search. Used by Search.

## `sortBy` (type: `string`):

Relevance or newest first. Date order can page past 1,000 hits.

## `minPoints` (type: `integer`):

Skip stories below this score. Skipped rows are never charged.

## `minComments` (type: `integer`):

Skip stories with fewer comments.

## `since` (type: `string`):

Only items at or after this date, e.g. 2026-01-01 or 2026-01-01T12:00:00Z.

## `until` (type: `string`):

Only items before this date.

## `author` (type: `string`):

Restrict a search to one Hacker News username, e.g. pg.

## `listType` (type: `string`):

Used by List mode.

## `storyIds` (type: `array`):

Story ids or news.ycombinator.com/item?id=... URLs. Used by Story comments.

## `includeStory` (type: `boolean`):

Add the story itself as the first row.

## `maxDepth` (type: `integer`):

1 = top-level comments only.

## `usernames` (type: `array`):

Hacker News usernames, e.g. pg, dang. Used by User.

## `userContent` (type: `string`):

What to return besides the profile row.

## `hiringKind` (type: `string`):

Used by Who is hiring.

## `hiringMonth` (type: `string`):

latest, or a month such as 2026-10 or October 2026.

## `hiringKeywords` (type: `array`):

Keep only posts containing these words, e.g. rust, python.

## `keywordMatch` (type: `string`):

Any keyword or all keywords.

## `remoteOnly` (type: `boolean`):

Keep only posts that mention remote.

## `includeChildIds` (type: `boolean`):

Add the list of direct child ids to each row.

## Actor input object example

```json
{
  "mode": "search",
  "searchTerms": [
    "web scraping"
  ],
  "maxItems": 25,
  "contentType": "story",
  "sortBy": "relevance",
  "listType": "top",
  "includeStory": true,
  "maxDepth": 99,
  "userContent": "stories",
  "hiringKind": "hiring",
  "hiringMonth": "latest",
  "keywordMatch": "any",
  "remoteOnly": false,
  "includeChildIds": false
}
```

# Actor output Schema

## `stories` (type: `string`):

No description

## `comments` (type: `string`):

No description

## `hiring` (type: `string`):

No description

## `users` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "search",
    "searchTerms": [
        "web scraping"
    ],
    "maxItems": 25
};

// Run the Actor and wait for it to finish
const run = await client.actor("transparent_meteorite/hacker-news-search-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "search",
    "searchTerms": ["web scraping"],
    "maxItems": 25,
}

# Run the Actor and wait for it to finish
run = client.actor("transparent_meteorite/hacker-news-search-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "search",
  "searchTerms": [
    "web scraping"
  ],
  "maxItems": 25
}' |
apify call transparent_meteorite/hacker-news-search-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,transparent_meteorite/hacker-news-search-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Q0VrS1xaBEvEfdHIC/builds/Aefesft1UphJ8d9iO/openapi.json