# Hacker News Scraper — Stories, Comments & Search for RAG (`ahampton83/hackernews-scraper`) Actor Fetch Hacker News stories, comment threads, and search results as clean structured data with markdown comments. Search by keyword, grab front page stories, or get full comment trees. MCP-enabled for Claude, Cursor, and AI agents. - **URL**: https://apify.com/ahampton83/hackernews-scraper.md - **Developed by:** [Aaron Hampton](https://apify.com/ahampton83) (community) - **Categories:** AI, Developer tools, MCP servers - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Hacker News Scraper — Stories, Comments & Search for RAG Fetch Hacker News stories, comment threads, and search results as clean structured data with markdown-formatted comments. Search by keyword, grab front page stories, or get full comment trees with depth tracking. Perfect for RAG pipelines, trend monitoring, startup research, and AI agents. ### Features - **Keyword search** — search all HN stories and comments via the Algolia API - **Front page** — get current top stories with points, authors, and metadata - **Full comment trees** — recursive comment fetching with depth tracking and parent IDs - **User profiles** — fetch user karma, about, and submission history - **Clean markdown** — all HTML stripped, comments converted to readable markdown - **Dual-mode** — run as a normal Apify Actor OR connect as an MCP server for Claude, Cursor, and other AI agents - **No browser required** — uses free public APIs (Algolia + Firebase), no proxies needed ### Use Cases - **RAG pipelines** — feed HN discussions into vector databases for Q&A over tech community knowledge - **Trend monitoring** — track trending topics, startups, and technologies on HN - **Startup research** — search for mentions of companies, products, or technologies - **AI agents** — let Claude/Cursor search HN via MCP tools - **Sentiment analysis** — analyze community reactions to product launches and news ### Input (Normal Actor Mode) | Field | Type | Description | |-------|------|-------------| | `mode` | enum | `search`, `front_page`, `story`, or `user` | | `query` | string | Search keyword (mode=search) | | `storyId` | integer | HN story ID (mode=story) | | `username` | string | HN username (mode=user) | | `maxStories` | integer | Max stories to return (default 20) | | `includeComments` | boolean | Fetch comment threads (default true) | | `maxCommentsPerStory` | integer | Max comments per story (default 30) | ### Output ```json { "id": 38309611, "title": "OpenAI's board has fired Sam Altman", "url": "https://openai.com/blog/...", "hnUrl": "https://news.ycombinator.com/item?id=38309611", "author": "davidbarker", "points": 5710, "numComments": 2530, "createdAt": "2023-11-17T20:28:50Z", "comments": [ { "id": 15, "author": "sama", "textMarkdown": "Clean markdown comment text...", "parent": 38309611, "depth": 0 } ] } ```` ### MCP Tools When connected as an MCP server, the following tools are available: | Tool | Description | |------|-------------| | `search_hn` | Search HN stories by keyword. Returns title, points, author, URL. | | `get_story` | Fetch a full story with optional comment thread. | | `get_front_page` | Get current top stories from the HN front page. | ### Pricing Pay-per-event. First results are free; subsequent events billed at: | Event | Price | |-------|-------| | Story fetched | $0.002 | | Comment thread fetched | $0.001 | | MCP tool call | $0.005 | | Actor start | $0.00005 | Volume discounts available at higher tiers. # Actor input Schema ## `mode` (type: `string`): search = keyword search via Algolia, front\_page = top stories from Firebase API, story = full story with comment thread, user = user profile + submissions ## `query` (type: `string`): Keyword search query (required for mode=search). Searches HN story titles and text via Algolia. ## `storyId` (type: `integer`): Hacker News story/item ID (required for mode=story). E.g. 1 for the first HN post. ## `username` (type: `string`): HN username (required for mode=user). E.g. 'pg'. ## `maxStories` (type: `integer`): Maximum number of stories to return (search & front\_page modes). ## `includeComments` (type: `boolean`): Fetch full comment threads for each story. ## `maxCommentsPerStory` (type: `integer`): Maximum comments to fetch per story (to limit run time on big threads). ## `outputFormat` (type: `string`): markdown = include markdown-formatted text, json = raw structured data only. ## Actor input object example ```json { "mode": "front_page", "query": "", "storyId": 1, "username": "", "maxStories": 20, "includeComments": true, "maxCommentsPerStory": 50, "outputFormat": "markdown" } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("ahampton83/hackernews-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("ahampton83/hackernews-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call ahampton83/hackernews-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=ahampton83/hackernews-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Hacker News Scraper — Stories, Comments & Search for RAG", "description": "Fetch Hacker News stories, comment threads, and search results as clean structured data with markdown comments. Search by keyword, grab front page stories, or get full comment trees. MCP-enabled for Claude, Cursor, and AI agents.", "version": "0.1", "x-build-id": "Iz8N9iouYqCmOpHXc" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/ahampton83~hackernews-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-ahampton83-hackernews-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/ahampton83~hackernews-scraper/runs": { "post": { "operationId": "runs-sync-ahampton83-hackernews-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/ahampton83~hackernews-scraper/run-sync": { "post": { "operationId": "run-sync-ahampton83-hackernews-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "mode" ], "properties": { "mode": { "title": "Mode", "enum": [ "search", "front_page", "story", "user" ], "type": "string", "description": "search = keyword search via Algolia, front_page = top stories from Firebase API, story = full story with comment thread, user = user profile + submissions", "default": "front_page" }, "query": { "title": "Search query", "type": "string", "description": "Keyword search query (required for mode=search). Searches HN story titles and text via Algolia.", "default": "" }, "storyId": { "title": "Story ID", "type": "integer", "description": "Hacker News story/item ID (required for mode=story). E.g. 1 for the first HN post.", "default": 1 }, "username": { "title": "Username", "type": "string", "description": "HN username (required for mode=user). E.g. 'pg'.", "default": "" }, "maxStories": { "title": "Max stories", "minimum": 1, "maximum": 100, "type": "integer", "description": "Maximum number of stories to return (search & front_page modes).", "default": 20 }, "includeComments": { "title": "Include comments", "type": "boolean", "description": "Fetch full comment threads for each story.", "default": true }, "maxCommentsPerStory": { "title": "Max comments per story", "minimum": 0, "maximum": 500, "type": "integer", "description": "Maximum comments to fetch per story (to limit run time on big threads).", "default": 50 }, "outputFormat": { "title": "Output format", "enum": [ "markdown", "json" ], "type": "string", "description": "markdown = include markdown-formatted text, json = raw structured data only.", "default": "markdown" } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```