# Hacker News Intelligence Scraper (`fascinating_lentil/hacker-news-intelligence-scraper`) Actor
Scrape Hacker News stories, comments, jobs, Ask HN, Show HN, and keyword search results. Export clean JSON or CSV with scores, authors, URLs, dates, filters, and nested discussions. No login or API key required.
- **URL**: https://apify.com/fascinating\_lentil/hacker-news-intelligence-scraper.md
- **Developed by:** [Md Jakaria Mirza](https://apify.com/fascinating_lentil) (community)
- **Categories:** Developer tools, News, Automation
- **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $0.75 / 1,000 hacker news item scrapeds
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event
## What's an Apify Actor?
Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.
In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):
```bash
npm install apify-client
```
In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):
```bash
pip install apify-client
```
In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):
````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash
In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).
If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).
For usage examples, see the [API](#api) section below.
For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).
# README
## Hacker News Scraper - Stories, Comments, Scores & Search
Scrape Hacker News stories, comments, scores, and search results into clean, structured data. This Hacker News scraper collects top, new, best, Ask HN, Show HN, and jobs feeds, runs full-text keyword search, fetches specific item IDs, and optionally nests comment threads. Export to JSON, CSV, Excel, or HTML, or pull via the Apify API. No login and no API key required.
Built with Node.js 20, TypeScript, and the Apify SDK. It uses the official Hacker News Firebase API for feeds and item details and the public HN Algolia Search API for keyword search, with retries and bounded concurrency so runs are reliable and repeatable. No browser and no proxy are required.
### What It Extracts
- `id` - numeric Hacker News item ID
- `type` - item type (story, comment, job, poll)
- `title` - story or item title
- `text` and `textHtml` - item body as plain text and HTML
- `url` - external link, when present
- `hnUrl` - the Hacker News discussion URL
- `domain` - hostname of the external link
- `author` - HN username
- `score` - points
- `commentCount` - number of comments
- `parentId` - parent item ID for comments
- `pollId` and `pollParts` - poll references
- `createdAt` and `createdAtUnix` - publication time (ISO and Unix)
- `rank` - position in the feed
- `feed` - source feed (top, new, best, ask, show, jobs)
- `query` - the search query that matched the record
- `dead` and `deleted` - moderation flags
- `comments` - optional nested comment threads (`id`, `author`, `text`, `textHtml`, `score`, `createdAt`, `createdAtUnix`, `parentId`, `depth`, `dead`, `deleted`, `hnUrl`)
- `collectedAt` - scrape timestamp
### Use Cases
1. Monitor developer, startup, and technology trends across the top, new, and best feeds.
2. Track product, brand, and competitor mentions with keyword and domain filters.
3. Discover Show HN launches and emerging tools as they appear.
4. Analyze Ask HN discussions and developer sentiment with nested comments.
5. Collect Hacker News jobs feed data for recruiting and hiring research.
6. Build alerts, dashboards, and datasets for AI and research pipelines.
### Pricing
This Actor uses Apify Pay Per Event pricing. You pay only for clean records delivered to the dataset. Empty, filtered, or failed records are not charged.
| Event name | Price per event | 1,000 results | 10,000 results |
| --- | ---: | ---: | ---: |
| `item-scraped` | $0.00075 | $0.75 | $7.50 |
### Input
| Field | Type | Required | Default | Description |
| --- | --- | --- | --- | --- |
| `mode` | string | yes | `feed` | `feed`, `search`, or `items` |
| `feed` | string | yes | `top` | Top, new, best, ask, show, or jobs |
| `query` | string | yes | `artificial intelligence` | Keyword query for search mode |
| `searchType` | string | yes | `story` | Search stories or comments |
| `itemIds` | string[] | no | - | HN item IDs for items mode |
| `maxResults` | integer | yes | `100` | Maximum dataset records (1-1000) |
| `minScore` | integer | no | `0` | Minimum points |
| `minComments` | integer | no | `0` | Minimum story comments |
| `includeKeywords` | string[] | no | `[]` | Require at least one title/text keyword |
| `excludeKeywords` | string[] | no | `[]` | Exclude matching title/text keywords |
| `authors` | string[] | no | `[]` | Exact HN usernames |
| `domain` | string | no | empty | Required substring in external hostname |
| `fromDate` | string | no | empty | Earliest publication date |
| `toDate` | string | no | empty | Latest publication date |
| `includeComments` | boolean | no | `false` | Nest comments in each record |
| `maxCommentsPerItem` | integer | no | `50` | Nested comment limit per record (1-500) |
| `commentDepth` | integer | no | `3` | Reply depth limit (1-10) |
| `includeDeadOrDeleted` | boolean | no | `false` | Include dead or deleted items |
#### Example input
```json
{
"mode": "feed",
"feed": "show",
"maxResults": 25,
"minScore": 10,
"includeKeywords": ["AI", "developer"],
"includeComments": true,
"maxCommentsPerItem": 20,
"commentDepth": 2
}
````
### How to Scrape Hacker News (Step by Step)
1. Click **Try for free** / **Run**.
2. Pick a `mode`: a `feed` (top, new, best, ask, show, jobs), keyword `search`, or specific `itemIds`.
3. Add filters such as `minScore`, `includeKeywords`, `authors`, `domain`, or a date range to narrow results.
4. Set `maxResults` (start small to test) and toggle `includeComments` if you want nested threads.
5. Run the Actor, then export results as JSON, CSV, Excel, or HTML, or pull them via the Apify API.
### Sample Output
```json
{
"id": 48487029,
"type": "story",
"title": "Show HN: An open-source AI agent for developers",
"text": null,
"textHtml": null,
"url": "https://github.com/example/ai-agent",
"hnUrl": "https://news.ycombinator.com/item?id=48487029",
"domain": "github.com",
"author": "devbuilder",
"score": 287,
"commentCount": 94,
"parentId": null,
"pollId": null,
"pollParts": [],
"createdAt": "2026-06-11T05:22:06.000Z",
"createdAtUnix": 1781155326,
"rank": 1,
"feed": "show",
"query": null,
"dead": false,
"deleted": false,
"comments": [
{
"id": 48487102,
"author": "curious_hacker",
"text": "This is great, how does it handle rate limits?",
"textHtml": "This is great, how does it handle rate limits?
",
"score": 0,
"createdAt": "2026-06-11T05:41:12.000Z",
"createdAtUnix": 1781156472,
"parentId": 48487029,
"depth": 1,
"dead": false,
"deleted": false,
"hnUrl": "https://news.ycombinator.com/item?id=48487102"
}
],
"collectedAt": "2026-06-11T06:00:00.000Z"
}
```
### How It Works
1. Validates the input and selects the collection mode (feed, search, or items).
2. Fetches feed and item details from the official Hacker News Firebase API, and keyword results from the public HN Algolia Search API.
3. Applies score, comment, keyword, author, domain, and date filters.
4. Optionally fetches and nests comment threads up to your depth and count limits.
5. Charges `item-scraped` only after a clean record is saved, then writes it to the Apify Dataset.
### Known Limits
- `text` and `textHtml` are only present for items that have body content; link stories return `null` for these fields.
- `comments` are only populated when `includeComments` is enabled, and are bounded by `maxCommentsPerItem` and `commentDepth`.
- Keyword search uses the HN Algolia API, so results and ranking follow that service's coverage and indexing.
- Dead or deleted items are excluded unless `includeDeadOrDeleted` is enabled.
- `maxResults` is capped at 1,000 records per run.
### Data Sources
This Actor uses the official Hacker News Firebase API and the public HN Algolia Search API. It does not rely on fragile page selectors.
### License
Apache-2.0.
# Actor input Schema
## `mode` (type: `string`):
Choose a Hacker News feed, keyword search, or explicit item IDs.
## `feed` (type: `string`):
The official Hacker News feed to collect when mode is Feed.
## `query` (type: `string`):
Full-text query used in Keyword search mode.
## `searchType` (type: `string`):
Search stories or comments through the public HN Algolia API.
## `itemIds` (type: `array`):
Numeric Hacker News item IDs used in Specific item IDs mode.
## `maxResults` (type: `integer`):
Maximum number of records written to the dataset.
## `minScore` (type: `integer`):
Keep items with at least this many points.
## `minComments` (type: `integer`):
Keep stories with at least this many comments.
## `includeKeywords` (type: `array`):
Keep records containing at least one keyword in the title or text. Leave empty to disable.
## `excludeKeywords` (type: `array`):
Remove records containing any of these keywords in the title or text.
## `authors` (type: `array`):
Only keep records by these exact Hacker News usernames.
## `domain` (type: `string`):
Only keep external links whose hostname contains this value.
## `fromDate` (type: `string`):
Optional ISO date or date-time, for example 2026-01-01.
## `toDate` (type: `string`):
Optional ISO date or date-time, for example 2026-12-31.
## `includeComments` (type: `boolean`):
Fetch comments and nest them inside each result record.
## `maxCommentsPerItem` (type: `integer`):
Maximum nested comments fetched for each result.
## `commentDepth` (type: `integer`):
Maximum nested reply depth to traverse.
## `includeDeadOrDeleted` (type: `boolean`):
Include records marked dead or deleted by Hacker News.
## Actor input object example
```json
{
"mode": "feed",
"feed": "top",
"query": "artificial intelligence",
"searchType": "story",
"maxResults": 100,
"minScore": 0,
"minComments": 0,
"includeKeywords": [],
"excludeKeywords": [],
"authors": [],
"domain": "",
"fromDate": "",
"toDate": "",
"includeComments": false,
"maxCommentsPerItem": 50,
"commentDepth": 3,
"includeDeadOrDeleted": false
}
```
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {};
// Run the Actor and wait for it to finish
const run = await client.actor("fascinating_lentil/hacker-news-intelligence-scraper").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {}
# Run the Actor and wait for it to finish
run = client.actor("fascinating_lentil/hacker-news-intelligence-scraper").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{}' |
apify call fascinating_lentil/hacker-news-intelligence-scraper --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"command": "npx",
"args": [
"mcp-remote",
"https://mcp.apify.com/?tools=fascinating_lentil/hacker-news-intelligence-scraper",
"--header",
"Authorization: Bearer "
]
}
}
}
```
## OpenAPI specification
```json
{
"openapi": "3.0.1",
"info": {
"title": "Hacker News Intelligence Scraper",
"description": "Scrape Hacker News stories, comments, jobs, Ask HN, Show HN, and keyword search results. Export clean JSON or CSV with scores, authors, URLs, dates, filters, and nested discussions. No login or API key required.",
"version": "1.0",
"x-build-id": "tie1gpjznkNrHj4xK"
},
"servers": [
{
"url": "https://api.apify.com/v2"
}
],
"paths": {
"/acts/fascinating_lentil~hacker-news-intelligence-scraper/run-sync-get-dataset-items": {
"post": {
"operationId": "run-sync-get-dataset-items-fascinating_lentil-hacker-news-intelligence-scraper",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
},
"/acts/fascinating_lentil~hacker-news-intelligence-scraper/runs": {
"post": {
"operationId": "runs-sync-fascinating_lentil-hacker-news-intelligence-scraper",
"x-openai-isConsequential": false,
"summary": "Executes an Actor and returns information about the initiated run in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/runsResponseSchema"
}
}
}
}
}
}
},
"/acts/fascinating_lentil~hacker-news-intelligence-scraper/run-sync": {
"post": {
"operationId": "run-sync-fascinating_lentil-hacker-news-intelligence-scraper",
"x-openai-isConsequential": false,
"summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
"tags": [
"Run Actor"
],
"requestBody": {
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/inputSchema"
}
}
}
},
"parameters": [
{
"name": "token",
"in": "query",
"required": true,
"schema": {
"type": "string"
},
"description": "Enter your Apify token here"
}
],
"responses": {
"200": {
"description": "OK"
}
}
}
}
},
"components": {
"schemas": {
"inputSchema": {
"type": "object",
"required": [
"mode",
"feed",
"query",
"searchType",
"maxResults"
],
"properties": {
"mode": {
"title": "Collection mode",
"enum": [
"feed",
"search",
"items"
],
"type": "string",
"description": "Choose a Hacker News feed, keyword search, or explicit item IDs.",
"default": "feed"
},
"feed": {
"title": "Feed",
"enum": [
"top",
"new",
"best",
"ask",
"show",
"jobs"
],
"type": "string",
"description": "The official Hacker News feed to collect when mode is Feed.",
"default": "top"
},
"query": {
"title": "Search query",
"type": "string",
"description": "Full-text query used in Keyword search mode.",
"default": "artificial intelligence"
},
"searchType": {
"title": "Search result type",
"enum": [
"story",
"comment"
],
"type": "string",
"description": "Search stories or comments through the public HN Algolia API.",
"default": "story"
},
"itemIds": {
"title": "Item IDs",
"type": "array",
"description": "Numeric Hacker News item IDs used in Specific item IDs mode.",
"items": {
"type": "string",
"pattern": "^[0-9]+$"
}
},
"maxResults": {
"title": "Maximum results",
"minimum": 1,
"maximum": 1000,
"type": "integer",
"description": "Maximum number of records written to the dataset.",
"default": 100
},
"minScore": {
"title": "Minimum score",
"minimum": 0,
"type": "integer",
"description": "Keep items with at least this many points.",
"default": 0
},
"minComments": {
"title": "Minimum comment count",
"minimum": 0,
"type": "integer",
"description": "Keep stories with at least this many comments.",
"default": 0
},
"includeKeywords": {
"title": "Include keywords",
"type": "array",
"description": "Keep records containing at least one keyword in the title or text. Leave empty to disable.",
"items": {
"type": "string"
},
"default": []
},
"excludeKeywords": {
"title": "Exclude keywords",
"type": "array",
"description": "Remove records containing any of these keywords in the title or text.",
"items": {
"type": "string"
},
"default": []
},
"authors": {
"title": "Authors",
"type": "array",
"description": "Only keep records by these exact Hacker News usernames.",
"items": {
"type": "string"
},
"default": []
},
"domain": {
"title": "Domain contains",
"type": "string",
"description": "Only keep external links whose hostname contains this value.",
"default": ""
},
"fromDate": {
"title": "Published after",
"type": "string",
"description": "Optional ISO date or date-time, for example 2026-01-01.",
"default": ""
},
"toDate": {
"title": "Published before",
"type": "string",
"description": "Optional ISO date or date-time, for example 2026-12-31.",
"default": ""
},
"includeComments": {
"title": "Include nested comments",
"type": "boolean",
"description": "Fetch comments and nest them inside each result record.",
"default": false
},
"maxCommentsPerItem": {
"title": "Maximum comments per item",
"minimum": 1,
"maximum": 500,
"type": "integer",
"description": "Maximum nested comments fetched for each result.",
"default": 50
},
"commentDepth": {
"title": "Comment depth",
"minimum": 1,
"maximum": 10,
"type": "integer",
"description": "Maximum nested reply depth to traverse.",
"default": 3
},
"includeDeadOrDeleted": {
"title": "Include dead or deleted items",
"type": "boolean",
"description": "Include records marked dead or deleted by Hacker News.",
"default": false
}
}
},
"runsResponseSchema": {
"type": "object",
"properties": {
"data": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"actId": {
"type": "string"
},
"userId": {
"type": "string"
},
"startedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"finishedAt": {
"type": "string",
"format": "date-time",
"example": "2025-01-08T00:00:00.000Z"
},
"status": {
"type": "string",
"example": "READY"
},
"meta": {
"type": "object",
"properties": {
"origin": {
"type": "string",
"example": "API"
},
"userAgent": {
"type": "string"
}
}
},
"stats": {
"type": "object",
"properties": {
"inputBodyLen": {
"type": "integer",
"example": 2000
},
"rebootCount": {
"type": "integer",
"example": 0
},
"restartCount": {
"type": "integer",
"example": 0
},
"resurrectCount": {
"type": "integer",
"example": 0
},
"computeUnits": {
"type": "integer",
"example": 0
}
}
},
"options": {
"type": "object",
"properties": {
"build": {
"type": "string",
"example": "latest"
},
"timeoutSecs": {
"type": "integer",
"example": 300
},
"memoryMbytes": {
"type": "integer",
"example": 1024
},
"diskMbytes": {
"type": "integer",
"example": 2048
}
}
},
"buildId": {
"type": "string"
},
"defaultKeyValueStoreId": {
"type": "string"
},
"defaultDatasetId": {
"type": "string"
},
"defaultRequestQueueId": {
"type": "string"
},
"buildNumber": {
"type": "string",
"example": "1.0.0"
},
"containerUrl": {
"type": "string"
},
"usage": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "integer",
"example": 1
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
},
"usageTotalUsd": {
"type": "number",
"example": 0.00005
},
"usageUsd": {
"type": "object",
"properties": {
"ACTOR_COMPUTE_UNITS": {
"type": "integer",
"example": 0
},
"DATASET_READS": {
"type": "integer",
"example": 0
},
"DATASET_WRITES": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_READS": {
"type": "integer",
"example": 0
},
"KEY_VALUE_STORE_WRITES": {
"type": "number",
"example": 0.00005
},
"KEY_VALUE_STORE_LISTS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_READS": {
"type": "integer",
"example": 0
},
"REQUEST_QUEUE_WRITES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_INTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"DATA_TRANSFER_EXTERNAL_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
"type": "integer",
"example": 0
},
"PROXY_SERPS": {
"type": "integer",
"example": 0
}
}
}
}
}
}
}
}
}
}
```