# Hacker News Scraper – Stories, Search & Jobs (`your_scraper_guy/hacker-news-scraper-stories-search-jobs`) Actor Scrape Hacker News stories, comments, and the monthly Who is hiring? thread. Full-text keyword search with filters. No proxy needed, uses HN's official APIs. - **URL**: https://apify.com/your\_scraper\_guy/hacker-news-scraper-stories-search-jobs.md - **Developed by:** [Code With Aqib](https://apify.com/your_scraper_guy) (community) - **Categories:** Developer tools, News - **Stats:** 2 total users, 0 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Hacker News Scraper — Stories, Keyword Search & Who is Hiring? Jobs Scrape Hacker News the reliable way: through its own official public APIs, not by parsing fragile HTML. No login, no API key, no proxy, no CAPTCHAs, no IP blocks. If you've been burned by scrapers that break every time a site changes its markup, this one won't, because it never touches the markup at all. ### 🔍 What you can do with it - **Browse any feed** — Top, New, Best, Ask HN, Show HN, or Jobs, with optional minimum-score and domain filters - **Full-text keyword search** — search titles, self-text, and comments across all of Hacker News, filtered by author, date range, minimum points, and minimum comment count, sorted by relevance or newest-first - **Who is hiring? extraction** — automatically finds the latest monthly hiring thread and returns every job listing as a clean, structured row, optionally filtered by keyword (e.g. "remote", "React", "Series A") - **Comment trees** — attach each story's comments, capped to however many you need ### 🧑‍💻 Who uses this - **Recruiters & talent teams** — pull the current "Who is hiring?" thread as structured rows instead of scrolling a giant comment thread by hand - **Market & competitive intelligence** — track when your company, product, or competitor gets mentioned, sorted by newest first for daily monitoring - **Tech trend research** — pull top stories by score over any time window to see what the developer community is actually paying attention to - **AI / LLM data pipelines** — clean, structured JSON output ready to feed into research or training pipelines - **Newsletter & content curators** — grab the day's top stories on a schedule and auto-populate a digest ### ⚡ Why this one, not another HN scraper Most scrapers on the Store either parse Hacker News' HTML directly (which breaks whenever the site's markup changes and often needs a paid proxy to avoid blocks) or only cover a single feed. This one: - Uses HN's two **official, public, unlimited APIs** (Firebase + Algolia), the same ones HN's own front-end uses - Needs **zero proxy configuration** — nothing to break, nothing to pay for - Covers **three real modes** in one actor: browsing, searching, and hiring-thread extraction, instead of forcing you to combine several tools - Returns a **consistent, unified schema** across every mode, so downstream processing stays simple ### 📥 Input | Field | Applies to | Description | |---|---|---| | `mode` | all | `list`, `search`, or `hiring` | | `storyType` | list | `top`, `new`, `best`, `ask`, `show`, `job` | | `minScore` | list | Only include stories at or above this score | | `domainFilter` | list | Only include stories linking to a specific domain | | `searchQuery` | search | Full-text keyword query | | `searchTags` | search | Content types to include: story, comment, poll, show_hn, ask_hn, job | | `sortByDate` | search | Newest-first instead of most-relevant-first | | `author` | search | Restrict to one username | | `minPoints` / `minComments` | search | Engagement thresholds | | `dateFrom` / `dateTo` | search | ISO date range | | `hiringKeyword` | hiring | Only return job listings containing this keyword | | `includeComments` | all | Attach each item's comment tree | | `maxCommentsPerItem` | all | Cap comments per item | | `includeText` | all | Include full self-text for Ask HN / Show HN posts | | `maxItems` | all | Max rows returned | ### 🚀 Quick start examples Browse today's top stories: ```json { "mode": "list", "storyType": "top", "maxItems": 50 } ```` Monitor mentions of your product, newest first: ```json { "mode": "search", "searchQuery": "your-product-name", "sortByDate": true, "maxItems": 50 } ``` Pull remote job listings from the latest hiring thread: ```json { "mode": "hiring", "hiringKeyword": "remote", "maxItems": 200 } ``` ### 📤 Example output ```json { "type": "story", "id": 44012345, "title": "Why we rewrote our scraper in Rust", "author": "some_user", "score": 312, "numComments": 87, "createdAt": "2026-07-10T09:15:00.000Z", "url": "https://example.com/blog-post", "domain": "example.com", "hnDiscussionUrl": "https://news.ycombinator.com/item?id=44012345", "text": null } ``` ### ❓ FAQ **Does this need a proxy or API key?** No. Both Firebase and Algolia's HN APIs are public and unauthenticated. **How current is the data?** Real-time. Both APIs reflect HN live. **Can I schedule this?** Yes, use Apify's built-in Schedules to run it daily or hourly for ongoing monitoring. **Does it cover comments?** Yes, set `includeComments: true` on any mode. **Is this affiliated with Y Combinator or Hacker News?** No. This is an independent tool built on Hacker News' publicly documented APIs. # Actor input Schema ## `mode` (type: `string`): List: browse a feed (top/new/best/ask/show/job). Search: keyword search with filters via Algolia. Hiring: pull the latest 'Who is hiring?' thread as structured job rows. ## `storyType` (type: `string`): Which Hacker News feed to pull from. Only used when Mode = list. ## `minScore` (type: `integer`): Only include stories with at least this many points. Only used when Mode = list. Set 0 to disable. ## `domainFilter` (type: `string`): Only include stories linking to this domain, e.g. "github.com". Leave empty to include all domains. Only used when Mode = list. ## `searchQuery` (type: `string`): Full-text search across titles, self-text, and comments. Only used when Mode = search. ## `searchTags` (type: `array`): Which content types to include in search results. Only used when Mode = search. ## `sortByDate` (type: `boolean`): On: newest matches first (best for monitoring/alerts). Off: most relevant first. Only used when Mode = search. ## `author` (type: `string`): Only include items posted by this exact username. Only used when Mode = search. ## `minPoints` (type: `integer`): Only used when Mode = search. Set 0 to disable. ## `minComments` (type: `integer`): Only used when Mode = search. Set 0 to disable. ## `dateFrom` (type: `string`): ISO date, e.g. 2026-01-01. Only used when Mode = search. ## `dateTo` (type: `string`): ISO date, e.g. 2026-07-01. Only used when Mode = search. ## `hiringKeyword` (type: `string`): Only include job listings whose text contains this keyword, e.g. "remote" or "React". Case-insensitive. Only used when Mode = hiring. ## `includeComments` (type: `boolean`): Fetch each story's comment tree via the Algolia API and attach it to the row. Increases run time. ## `maxCommentsPerItem` (type: `integer`): Caps how many comments are attached per story when Include comments is on. ## `includeText` (type: `boolean`): Include the full text body for Ask HN / Show HN posts when present ## `maxItems` (type: `integer`): Maximum number of stories/results to return ## Actor input object example ```json { "mode": "list", "storyType": "top", "minScore": 0, "domainFilter": "", "searchQuery": "", "searchTags": [ "story" ], "sortByDate": false, "author": "", "minPoints": 0, "minComments": 0, "dateFrom": "", "dateTo": "", "hiringKeyword": "", "includeComments": false, "maxCommentsPerItem": 20, "includeText": true, "maxItems": 50 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("your_scraper_guy/hacker-news-scraper-stories-search-jobs").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("your_scraper_guy/hacker-news-scraper-stories-search-jobs").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call your_scraper_guy/hacker-news-scraper-stories-search-jobs --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=your_scraper_guy/hacker-news-scraper-stories-search-jobs", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Hacker News Scraper – Stories, Search & Jobs", "description": "Scrape Hacker News stories, comments, and the monthly Who is hiring? thread. Full-text keyword search with filters. No proxy needed, uses HN's official APIs.", "version": "0.1", "x-build-id": "ICzpDFdDZ2BeYfk8t" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/your_scraper_guy~hacker-news-scraper-stories-search-jobs/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-your_scraper_guy-hacker-news-scraper-stories-search-jobs", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/your_scraper_guy~hacker-news-scraper-stories-search-jobs/runs": { "post": { "operationId": "runs-sync-your_scraper_guy-hacker-news-scraper-stories-search-jobs", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/your_scraper_guy~hacker-news-scraper-stories-search-jobs/run-sync": { "post": { "operationId": "run-sync-your_scraper_guy-hacker-news-scraper-stories-search-jobs", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "mode" ], "properties": { "mode": { "title": "Mode", "enum": [ "list", "search", "hiring" ], "type": "string", "description": "List: browse a feed (top/new/best/ask/show/job). Search: keyword search with filters via Algolia. Hiring: pull the latest 'Who is hiring?' thread as structured job rows.", "default": "list" }, "storyType": { "title": "[List mode] Story list", "enum": [ "top", "new", "best", "ask", "show", "job" ], "type": "string", "description": "Which Hacker News feed to pull from. Only used when Mode = list.", "default": "top" }, "minScore": { "title": "[List mode] Minimum score", "minimum": 0, "type": "integer", "description": "Only include stories with at least this many points. Only used when Mode = list. Set 0 to disable.", "default": 0 }, "domainFilter": { "title": "[List mode] Domain filter (optional)", "type": "string", "description": "Only include stories linking to this domain, e.g. \"github.com\". Leave empty to include all domains. Only used when Mode = list.", "default": "" }, "searchQuery": { "title": "[Search mode] Keyword query", "type": "string", "description": "Full-text search across titles, self-text, and comments. Only used when Mode = search.", "default": "" }, "searchTags": { "title": "[Search mode] Content types", "type": "array", "description": "Which content types to include in search results. Only used when Mode = search.", "items": { "type": "string", "enum": [ "story", "comment", "poll", "show_hn", "ask_hn", "job" ] }, "default": [ "story" ] }, "sortByDate": { "title": "[Search mode] Sort by newest", "type": "boolean", "description": "On: newest matches first (best for monitoring/alerts). Off: most relevant first. Only used when Mode = search.", "default": false }, "author": { "title": "[Search mode] Filter by author (optional)", "type": "string", "description": "Only include items posted by this exact username. Only used when Mode = search.", "default": "" }, "minPoints": { "title": "[Search mode] Minimum points", "minimum": 0, "type": "integer", "description": "Only used when Mode = search. Set 0 to disable.", "default": 0 }, "minComments": { "title": "[Search mode] Minimum comment count", "minimum": 0, "type": "integer", "description": "Only used when Mode = search. Set 0 to disable.", "default": 0 }, "dateFrom": { "title": "[Search mode] Date from (optional)", "type": "string", "description": "ISO date, e.g. 2026-01-01. Only used when Mode = search.", "default": "" }, "dateTo": { "title": "[Search mode] Date to (optional)", "type": "string", "description": "ISO date, e.g. 2026-07-01. Only used when Mode = search.", "default": "" }, "hiringKeyword": { "title": "[Hiring mode] Filter by keyword (optional)", "type": "string", "description": "Only include job listings whose text contains this keyword, e.g. \"remote\" or \"React\". Case-insensitive. Only used when Mode = hiring.", "default": "" }, "includeComments": { "title": "Include comments", "type": "boolean", "description": "Fetch each story's comment tree via the Algolia API and attach it to the row. Increases run time.", "default": false }, "maxCommentsPerItem": { "title": "Max comments per story", "minimum": 1, "maximum": 200, "type": "integer", "description": "Caps how many comments are attached per story when Include comments is on.", "default": 20 }, "includeText": { "title": "Include self-text", "type": "boolean", "description": "Include the full text body for Ask HN / Show HN posts when present", "default": true }, "maxItems": { "title": "Max items", "minimum": 1, "maximum": 1000, "type": "integer", "description": "Maximum number of stories/results to return", "default": 50 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```