# Hacker News Scraper — Stories, Comments & Profiles (`lone_cipher/hacker-news-scraper`) Actor Scrape Hacker News stories, comment trees, and submitter profiles via the public Firebase API. No login, no auth, no rate limits. Six feed modes (top, new, best, ask, show, job). - **URL**: https://apify.com/lone\_cipher/hacker-news-scraper.md - **Developed by:** [Ian Bartholomew](https://apify.com/lone_cipher) (community) - **Categories:** News, Social media - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Hacker News Scraper — Stories, Comments & Profiles ### What does Hacker News Scraper do? Hacker News Scraper enables you to extract data from [Hacker News](https://news.ycombinator.com/) using the public Firebase API. No login or authentication required. Hacker News Scraper can scrape: - Stories from any of the six HN feeds (top, new, best, ask, show, job) - Full comment trees for each story - Submitter profiles (karma, account age, bio) - Story metadata (score, comments count, timestamps, URLs) - Comment metadata (author, timestamp, parent/child relationships) ### Why scrape Hacker News? Hacker News is one of the most influential tech communities online. It's an excellent source of data for AI training, market research, and competitive intelligence. Here are just some of the ways you could use Hacker News data: - Build training datasets for tech-aware language models - Track which products, libraries, and frameworks are gaining traction - Research founders and their posting patterns before investing - Monitor trends in specific HN categories (Ask HN, Show HN, jobs) - Build RSS-like feeds of curated HN content for newsletters - Analyze comment sentiment and engagement on tech discussions If you would like more inspiration on how scraping Hacker News could help your business or organization, check out our [industry pages](https://apify.com/industries). ### How to scrape Hacker News It's easy to scrape Hacker News with Hacker News Scraper. Just follow these few steps and you'll get your data in minutes. 1. Click on **Try for free**. 2. Select a feed mode (Top Stories is the default). 3. Configure the max number of stories and whether to include comments and submitter profiles. 4. Click on **Run**. 5. When Hacker News Scraper has finished, preview or download your data from the Dataset tab. ### How much will it cost to scrape Hacker News? Apify gives you $5 free usage credits every month on the [Apify Free plan](https://apify.com/pricing). A typical run of 100 stories costs $0.50 — well within the free tier. But if you need to scrape Hacker News regularly or at scale, we recommend our [$49/month Starter plan](https://apify.com/pricing) for consistent access to fresh data. For large-scale scraping operations, the [Scale plan](https://apify.com/pricing) offers even more value with higher monthly allowances. ### Results The actor returns data in JSON format, one record per story: ```json { "type": "story", "id": 42345678, "title": "Show HN: I built a thing", "url": "https://example.com", "score": 234, "by": "username", "time": 1718371200, "timeIso": "2024-06-14T10:00:00Z", "descendants": 87, "commentCount": 87, "type_hn": "story", "submitter": { "id": "username", "created": 1234567890, "createdIso": "2009-02-13T23:31:30Z", "karma": 5678, "about": null }, "comments": [ { "id": 42345679, "by": "commenter1", "text": "Great work!", "time": 1718371300, "timeIso": "2024-06-14T10:01:40Z", "deleted": false, "dead": false, "parent": 42345678 } ] } ```` ### Tips for scraping Hacker News - The Firebase API is free and has no rate limits — runs are fast (50+ concurrent fetches) - For large comment threads, lower `maxCommentsPerStory` to keep runs bounded - Disable `fetchProfiles` for cheaper runs that only need story metadata - The `job_stories` and `ask_stories` modes are great for niche research - Combine with the Story URL to link directly back to HN ### Is it legal to scrape Hacker News? Note that personal data is protected by GDPR in the European Union and by other regulations around the world. Hacker News data is public, but you should not scrape personal data unless you have a legitimate reason to do so. If you're unsure whether your reason is legitimate, consult your lawyers. We also recommend that you read our blog post: [is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/). # Actor input Schema ## `mode` (type: `string`): Which HN feed to scrape. ## `maxItems` (type: `integer`): Maximum number of stories to fetch (capped at 500). ## `includeComments` (type: `boolean`): Fetch comment trees for each story. Disable for cheaper runs. ## `maxCommentsPerStory` (type: `integer`): Cap on comments fetched per story (capped at 200). ## `fetchProfiles` (type: `boolean`): Fetch user profile (karma, created date) for each story submitter. ## Actor input object example ```json { "mode": "top_stories", "maxItems": 100, "includeComments": true, "maxCommentsPerStory": 50, "fetchProfiles": true } ``` # Actor output Schema ## `results` (type: `string`): Hacker News stories fetched from the public Firebase API. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("lone_cipher/hacker-news-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("lone_cipher/hacker-news-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call lone_cipher/hacker-news-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=lone_cipher/hacker-news-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Hacker News Scraper — Stories, Comments & Profiles", "description": "Scrape Hacker News stories, comment trees, and submitter profiles via the public Firebase API. No login, no auth, no rate limits. Six feed modes (top, new, best, ask, show, job).", "version": "0.1", "x-build-id": "nUWcFZltRc1DfgpKa" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/lone_cipher~hacker-news-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-lone_cipher-hacker-news-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/lone_cipher~hacker-news-scraper/runs": { "post": { "operationId": "runs-sync-lone_cipher-hacker-news-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/lone_cipher~hacker-news-scraper/run-sync": { "post": { "operationId": "run-sync-lone_cipher-hacker-news-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "mode": { "title": "Story Mode", "enum": [ "top_stories", "new_stories", "best_stories", "ask_stories", "show_stories", "job_stories" ], "type": "string", "description": "Which HN feed to scrape.", "default": "top_stories" }, "maxItems": { "title": "Max Stories", "minimum": 1, "maximum": 500, "type": "integer", "description": "Maximum number of stories to fetch (capped at 500).", "default": 100 }, "includeComments": { "title": "Include Comments", "type": "boolean", "description": "Fetch comment trees for each story. Disable for cheaper runs.", "default": true }, "maxCommentsPerStory": { "title": "Max Comments per Story", "minimum": 0, "maximum": 200, "type": "integer", "description": "Cap on comments fetched per story (capped at 200).", "default": 50 }, "fetchProfiles": { "title": "Fetch Submitter Profiles", "type": "boolean", "description": "Fetch user profile (karma, created date) for each story submitter.", "default": true } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```