# Hacker News Scraper (`rl1987/hacker-news-scraper`) Actor Scrapes Hacker News by search, user, listing (front page/new/ask/show/jobs/best), or specific threads. - **URL**: https://apify.com/rl1987/hacker-news-scraper.md - **Developed by:** [R.L.](https://apify.com/rl1987) (community) - **Categories:** News, Social media - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 dataset rows This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Hacker News Scraper **Hacker News Scraper** extracts stories, comments, and user profiles from [Hacker News](https://news.ycombinator.com) in four modes: search, user, listing, and threads. Built with [Scrapy](https://scrapy.org/) and the [Apify SDK for Python](https://docs.apify.com/sdk/python/), it runs on the Apify platform with proxy rotation, scheduling, and API access to results. ### Why use this Hacker News Scraper? - Track how a topic or company is discussed on Hacker News over time (search mode). - Monitor a specific user's karma, bio, and submission history (user mode). - Pull the current front page, newest posts, Ask/Show HN, or job postings (listing mode). - Archive full comment trees for specific threads, with parent/child relationships preserved (threads mode). ### How to use Hacker News Scraper 1. Click **Try for free** (or **Start**) on the Actor page. 2. Choose a **Mode** in the Input tab: `search`, `user`, `listing`, or `threads`. 3. Fill in the field(s) relevant to that mode (search query, username, list type, or thread item IDs). 4. Optionally set **Max items** and **Max pages** to bound the run. 5. Click **Start** and download results from the **Dataset** tab once the run finishes. ### Input | Field | Applies to | Description | | --- | --- | --- | | `mode` | all | `search`, `user`, `listing`, or `threads` | | `searchQuery` / `searchSort` | search | Query text and `relevance`/`date` sort order (uses the HN Algolia Search API) | | `searchType` | search | Result type filter: `story` (default, matches hn.algolia.com), `comment`, `poll`, `job`, `ask_hn`, `show_hn`, `front_page`, or `all` | | `username` | user | HN username to scrape the profile and submissions of | | `listType` | listing | `top`, `new`, `best`, `ask`, `show`, or `jobs` | | `threadUrls` | threads | List of HN thread URLs, e.g. `https://news.ycombinator.com/item?id=1` | | `includeComments` | threads | Whether to scrape the full comment tree | | `maxItems` | all | Maximum items to output (0 = unlimited) | | `maxPages` | search, listing | Maximum result pages to fetch (0 = unlimited) | | `proxyConfiguration` | all | Apify proxy settings | See the Input tab for the full schema. ### Output Each dataset item is a story, comment, or user record. Example (listing mode): ```json { "id": "49033127", "type": "story", "title": "Flux 3 X Mimic: The Next Generation of Video-Action Models", "url": "https://bfl.ai/blog/flux-3-mimic", "domain": "bfl.ai", "points": 170, "author": "kensai", "created_at": "2026-07-24T09:31:48 1784885508", "num_comments": 21, "rank": 1, "source": "top" } ```` Example (threads mode, a comment): ```json { "id": "49034857", "type": "comment", "author": "forgotusername6", "text": "Is anyone feeding models touch data? ...", "created_at": "2026-07-24T12:53:03 1784897583", "parent_id": "49031796", "story_id": "49031796", "depth": 0, "source": "thread" } ``` You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. #### Data fields | Field | Description | | --- | --- | | `id` | HN item ID, or username in `user` mode | | `type` | `story`, `job`, `poll`, `comment`, or `user` | | `title` / `url` / `domain` | Story title, external URL, and URL domain | | `text` | Comment body or self-post text | | `points` | Story score | | `author` | Submitter/commenter username | | `created_at` | ISO timestamp (or HN's raw date string) | | `num_comments` | Total comment count on a story | | `rank` | Position on a listing page | | `parent_id` / `story_id` / `depth` | Comment tree position | | `karma` / `about` / `created` | User profile fields | ### Cost of scraping Hacker News Hacker News pages are lightweight static HTML, so runs are fast and cheap. A single listing page (30 items) or a small thread typically completes in a few seconds on the free Apify plan. Scraping large threads with thousands of comments, or many listing/search pages, will use proportionally more compute units. ### Tips - Use `maxItems` to cap runs when a thread or user has far more items than you need. - `listing` and `search` modes support `maxPages` for pagination; set it to `0` to fetch all available pages. - HN rate-limits aggressive crawling — enable the Apify proxy and avoid very high concurrency for large jobs. ### FAQ, disclaimers, and support This Actor is for scraping publicly available data from Hacker News. Use it in accordance with Hacker News' terms of service and applicable law; do not use it to collect personal data beyond what users have made public. HN's page structure can change, which may require selector updates. Found a bug or need a custom variant? Open an issue on the Actor's Issues tab. # Actor input Schema ## `mode` (type: `string`): What to scrape: search results, a user's profile and submissions, a listing page (front page / newest / etc.), or specific threads. ## `searchQuery` (type: `string`): Text to search for. Only used when `mode` is `search`. ## `searchSort` (type: `string`): Sort search results by relevance or by date. Only used when `mode` is `search`. ## `searchType` (type: `string`): Restrict results to this item type, matching the hn.algolia.com search UI's tabs (it defaults to Stories, not everything). Only used when `mode` is `search`. ## `username` (type: `string`): Hacker News username to scrape the profile and submissions of. Only used when `mode` is `user`. ## `listType` (type: `string`): Which listing to scrape. Only used when `mode` is `listing`. ## `threadUrls` (type: `array`): HN thread URLs to scrape, e.g. `https://news.ycombinator.com/item?id=1`. Only used when `mode` is `threads`. ## `includeComments` (type: `boolean`): Whether to scrape the full comment tree for each thread. Only used when `mode` is `threads`. ## `maxItems` (type: `integer`): Maximum number of items to output (0 = unlimited). ## `maxPages` (type: `integer`): Maximum number of listing/search result pages to fetch (0 = unlimited). Not used in `user` or `threads` mode. ## `proxyConfiguration` (type: `object`): Specifies proxy servers that will be used by the scraper in order to hide its origin. ## Actor input object example ```json { "mode": "listing", "searchSort": "relevance", "searchType": "story", "listType": "top", "threadUrls": [ { "url": "https://news.ycombinator.com/item?id=1" } ], "includeComments": true, "maxItems": 0, "maxPages": 1, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `results` (type: `string`): Scraped stories, comments, and/or user profiles, one per dataset item. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "threadUrls": [ { "url": "https://news.ycombinator.com/item?id=1" } ], "proxyConfiguration": { "useApifyProxy": true } }; // Run the Actor and wait for it to finish const run = await client.actor("rl1987/hacker-news-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "threadUrls": [{ "url": "https://news.ycombinator.com/item?id=1" }], "proxyConfiguration": { "useApifyProxy": True }, } # Run the Actor and wait for it to finish run = client.actor("rl1987/hacker-news-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "threadUrls": [ { "url": "https://news.ycombinator.com/item?id=1" } ], "proxyConfiguration": { "useApifyProxy": true } }' | apify call rl1987/hacker-news-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=rl1987/hacker-news-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Hacker News Scraper", "description": "Scrapes Hacker News by search, user, listing (front page/new/ask/show/jobs/best), or specific threads.", "version": "0.1", "x-build-id": "fUQMakLlhb0QyjdYN" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/rl1987~hacker-news-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-rl1987-hacker-news-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/rl1987~hacker-news-scraper/runs": { "post": { "operationId": "runs-sync-rl1987-hacker-news-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/rl1987~hacker-news-scraper/run-sync": { "post": { "operationId": "run-sync-rl1987-hacker-news-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "mode" ], "properties": { "mode": { "title": "Mode", "enum": [ "search", "user", "listing", "threads" ], "type": "string", "description": "What to scrape: search results, a user's profile and submissions, a listing page (front page / newest / etc.), or specific threads.", "default": "listing" }, "searchQuery": { "title": "Search query", "type": "string", "description": "Text to search for. Only used when `mode` is `search`." }, "searchSort": { "title": "Search sort order", "enum": [ "relevance", "date" ], "type": "string", "description": "Sort search results by relevance or by date. Only used when `mode` is `search`.", "default": "relevance" }, "searchType": { "title": "Search result type", "enum": [ "story", "comment", "poll", "job", "ask_hn", "show_hn", "front_page", "all" ], "type": "string", "description": "Restrict results to this item type, matching the hn.algolia.com search UI's tabs (it defaults to Stories, not everything). Only used when `mode` is `search`.", "default": "story" }, "username": { "title": "Username", "type": "string", "description": "Hacker News username to scrape the profile and submissions of. Only used when `mode` is `user`." }, "listType": { "title": "List type", "enum": [ "top", "new", "best", "ask", "show", "jobs" ], "type": "string", "description": "Which listing to scrape. Only used when `mode` is `listing`.", "default": "top" }, "threadUrls": { "title": "Thread URLs", "type": "array", "description": "HN thread URLs to scrape, e.g. `https://news.ycombinator.com/item?id=1`. Only used when `mode` is `threads`.", "default": [], "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "includeComments": { "title": "Include comments", "type": "boolean", "description": "Whether to scrape the full comment tree for each thread. Only used when `mode` is `threads`.", "default": true }, "maxItems": { "title": "Max items", "minimum": 0, "type": "integer", "description": "Maximum number of items to output (0 = unlimited).", "default": 0 }, "maxPages": { "title": "Max pages", "minimum": 0, "type": "integer", "description": "Maximum number of listing/search result pages to fetch (0 = unlimited). Not used in `user` or `threads` mode.", "default": 1 }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "Specifies proxy servers that will be used by the scraper in order to hide its origin.", "default": { "useApifyProxy": true } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```