# Hacker News Threads & Who is Hiring Exporter (`vellumlabs/hn-threads-and-hiring`) Actor Hacker News exporter using the official Firebase and Algolia APIs: front/new/best/ask/show/jobs lists, search, full story threads as Markdown, and Who-is-Hiring threads parsed into structured jobs (company, role, location, remote, salary hints). Pay per story or job. - **URL**: https://apify.com/vellumlabs/hn-threads-and-hiring.md - **Developed by:** [Vellum Kasane](https://apify.com/vellumlabs) (community) - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 story exporteds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Hacker News Threads & Who is Hiring Exporter Export **Hacker News stories with their full comment threads as clean Markdown**, search HN by keyword and date, and turn the monthly **"Ask HN: Who is hiring?"** thread into **structured job posts** (company, role, location, remote, salary, visa, links). Everything comes from the official [Firebase HN API](https://github.com/HackerNews/API) and the [Algolia HN Search API](https://hn.algolia.com/api), so there is no HTML scraping, no proxies and no blocking. What you get that plain HN scrapers do not: - **Whole threads, ready for an LLM.** Story text and comments are converted from HN's markup to Markdown, comments are flattened with `depth` and `parent_id`, and the entire thread is also rendered as one Markdown document (replies nested as blockquotes) in `markdown`. - **One request per thread.** Comment trees come from Algolia's `items/` endpoint, which returns the full nested tree in a single call, so a front page with comments is about 40 API requests, not thousands. - **Who is hiring, parsed.** Each top-level post of the hiring thread becomes a record with `company`, `role`, `location`, `remote` / `remote_detail` / `onsite` / `hybrid`, `salary`, `visa`, `employment_type`, `yc_batch`, `urls`, the original `headline` and the full post as Markdown. Pick the newest thread automatically or pass thread IDs. - **Watch mode: only what is new since your last run.** Set a `stateKey` and schedule the Actor: each run returns (and bills) only the job posts or stories that no earlier run with that key exported. A daily `who-is-hiring` schedule becomes a feed of new HN job posts, typically a few dozen a day early in the month and a handful later, instead of re-buying the whole 300-500 post thread. - **Nine modes.** Front page, newest, best, Ask HN, Show HN, Jobs, keyword search (newest or relevance, with a date window), specific story IDs, Who is hiring. - **Rate-limit aware.** Concurrency capped (default 5), automatic exponential back-off on HTTP 429 / 5xx, `Retry-After` honoured. ### Who is it for - **AI agents and MCP clients** that need HN discussion around a topic as context (call it from the [Apify MCP server](https://mcp.apify.com), then read the dataset). - **Research and monitoring**: track what HN says about your product, library or competitor, on a schedule, as Markdown you can diff. - **Recruiting and job search**: a structured monthly feed of HN hiring posts filtered by remote, location or salary. - **Knowledge bases**: drop thread Markdown into Obsidian, Notion or a Claude Code wiki. ### Input | Field | Type | Default | Notes | | --------------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------- | | `mode` | string | `front` | `front`, `new`, `best`, `ask`, `show`, `jobs`, `search`, `who-is-hiring`, `story`. | | `query` | string | `""` | Search terms for `search` mode (Algolia syntax). | | `searchSort` | string | `date` | `date` (newest first) or `relevance`. | | `storyIds` | string\[] | `[]` | HN item IDs or item URLs for `story` and `who-is-hiring` modes. | | `maxItems` | integer | 100 | Cap on stories (or job posts in `who-is-hiring`). Caps cost. | | `sinceDays` | integer | 0 | Only items newer than N days (`0` = no filter). In `who-is-hiring` mode, all hiring threads in the window. | | `includeComments` | boolean | `true` | Fetch comment trees and render thread Markdown. | | `maxCommentDepth` | integer | 3 | `0` = top-level comments only. | | `maxCommentsPerStory` | integer | 200 | Depth-first cap per story. | | `maxConcurrency` | integer | 5 | Parallel API requests (max 10). | | `stateKey` | string | `""` | Watch mode. Name of a key-value store that remembers exported IDs; runs return only unseen items. | Example inputs: ```json { "mode": "front", "maxItems": 20, "includeComments": true, "maxCommentDepth": 2, "maxCommentsPerStory": 50 } ``` ```json { "mode": "search", "query": "local-first sync", "sinceDays": 90, "maxItems": 50 } ``` ```json { "mode": "who-is-hiring", "maxItems": 500 } ``` Daily feed of new HN job posts (schedule it once a day): ```json { "mode": "who-is-hiring", "maxItems": 500, "stateKey": "hn-hiring-watch" } ``` ### Output Story modes produce one item per story: ```json { "type": "story", "id": 49797323, "hn_url": "https://news.ycombinator.com/item?id=49797323", "title": "Can gzip be a language model?", "url": "https://nathan.rs/posts/gzip-lm/", "domain": "nathan.rs", "story_type": "story", "points": 280, "author": "networked", "created_at": "2026-09-22T06:40:12.000Z", "comment_count": 132, "text_markdown": "", "comments": [ { "id": 49797901, "parent_id": null, "author": "someone", "created_at": "2026-09-22T07:12:44.000Z", "depth": 0, "text_markdown": "This is the classic compression-as-classification trick..." }, { "id": 49798010, "parent_id": 49797901, "author": "other", "created_at": "...", "depth": 1, "text_markdown": "> classic trick\n\nRight, and ..." } ], "comments_stored": 50, "markdown": "# Can gzip be a language model?\n\n[nathan.rs](https://nathan.rs/posts/gzip-lm/) · 280 points · by networked · 2026-09-22T06:40:12.000Z · 132 comments · [HN](https://news.ycombinator.com/item?id=49797323)\n\n---\n\n## Comments (50 stored)\n\n**someone** · 2026-09-22T07:12:44.000Z · [#](https://news.ycombinator.com/item?id=49797901)\n\nThis is the classic ...\n\n> **other** · ... \n>\n> Right, and ...", "fetched_at": "2026-09-22T14:42:51.358Z" } ``` `who-is-hiring` mode produces one item per job post: ```json { "type": "job", "id": 49522903, "hn_url": "https://news.ycombinator.com/item?id=49522903", "story_id": 49522897, "story_title": "Ask HN: Who is hiring? (September 2026)", "thread_month": "2026-09", "author": "thelastofthelas", "created_at": "2026-09-01T15:01:54.000Z", "headline": "Modash.io | Senior Product Engineer | Remote (Europe) | Full-time | €75k–110k | https://modash.io", "company": "Modash.io", "role": "Senior Product Engineer", "location": "Remote (Europe)", "remote": true, "remote_detail": "Europe", "onsite": false, "hybrid": false, "salary": "€75k–110k", "visa": null, "employment_type": "full-time", "yc_batch": null, "tags": [], "urls": ["https://modash.io", "https://modash.io/engineering", "https://apply.workable.com/modash/j/C1507B65C3"], "markdown": "Modash.io | Senior Product Engineer | Remote (Europe) | Full-time | €75k–110k | [https://modash.io](https://modash.io)\n\nModash helps brands find, manage, and pay creators...", "fetched_at": "2026-09-22T14:43:18.526Z" } ``` The structured job fields are parsed from HN's `Company | Role | Location | REMOTE | Full-time | $salary | url` convention with heuristics; `headline`, `tags` (unclassified headline segments such as `ONSITE`, `Visa sponsorship available`) and `markdown` always keep the original text so nothing is lost. `visa` is `sponsored`, `not-sponsored` or `null`. `SUMMARY.json` in the key-value store holds run statistics (items stored, skipped, API requests, retries, duration). ### Pricing This Actor uses **pay-per-event** pricing: | Event | Price | When | | ---------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | `story-exported` | **$0.001 per story** ($1 per 1,000) | Once per story written to the dataset, comments included. Deleted, dead and filtered-out stories are never charged. | | `job-parsed` | **$0.001 per job post** ($1 per 1,000) | Once per Who-is-Hiring post written to the dataset. | | Actor start | $0.005 per run | Apify's standard start event; covers the first seconds of compute. | The front page with comments costs about **$0.03**; a full monthly hiring thread (about 250-450 posts) about **$0.45**. Set **Max total charge per run** to hard-cap spend; the Actor stops gracefully before exceeding it. Platform usage (compute, storage) is included in the event prices. ### Tips - **Big threads**: `maxCommentsPerStory` is depth-first, so raising `maxCommentDepth` without raising the cap keeps you on the first few top-level branches. For "top comments only" use `maxCommentDepth: 0`. - **Monitoring a topic**: schedule `search` mode daily with `searchSort: date` and a `stateKey` (for example `hn-watch-postgres`). Each run stops as soon as it reaches stories an earlier run exported, so you pay only for new ones and no downstream deduplication is needed. - **New job posts every day**: schedule `who-is-hiring` with a `stateKey`. The first run exports the current thread (up to `maxItems`); every later run exports only posts added since, and also re-checks last month's thread so late posts are not lost when the new month's thread appears. - **All hiring threads of a year**: `who-is-hiring` with `sinceDays: 365` processes every monthly thread in the window (`maxItems` still caps the total). - **Specific threads**: paste HN item URLs into `storyIds` with `mode: story`. ### Limits - Firebase lists return at most 500 (`front`, `new`, `best`) or 200 (`ask`, `show`, `jobs`) IDs, so those modes cap at that many stories per run. Use `search` with `sinceDays` for older or larger sets (Algolia paginates up to 1,000 hits per query). - Algolia allows 10,000 requests per hour per IP; at default concurrency this Actor stays far below it and backs off on 429. - Comment trees are read from Algolia and can lag the live site by a few minutes; `points` and `comment_count` come from the same source as the listing (Firebase for list modes, Algolia for search). - Watch mode remembers the newest 50,000 exported IDs per `stateKey`. It tracks IDs, not edits: a job post edited after it was exported is not exported again. Use a separate `stateKey` per schedule or per filter set, otherwise one schedule "uses up" items for the other. Deleting the key-value store resets the watch. - In watch mode `front`/`best` return stories that are new *to you*, not stories that are new on HN; for "newly submitted" use `new` or `search` sorted by date. - Text is what HN returns: `text_markdown` is empty for link posts, and deleted comments are skipped (their replies are kept one level up). ### Verified Local runs on 2026-09-22 with `apify run` (Apify CLI 1.10.0, Apify SDK 3.7.2, Node 24): - **`front` (prefill input: 20 stories, comments depth 2, 50 per story)**: 41 API requests (1 list + 20 Firebase items + 20 Algolia trees), 0 retries, 0 failed, **20 stories stored**, runtime 4.3 s. - **`who-is-hiring` (`maxItems: 50`)**: 2 API requests (thread lookup + one 497 KB tree with 258 top-level posts), **50 job posts stored**, runtime 2.0 s. Parse coverage on those 50 live posts: company 50, location 45, URLs 42, employment type 33, role 26, remote 25, salary 19, visa 4. - **`search` (`rust async`, `sinceDays: 365`, `maxItems: 5`)**: 5 requests, **5 stories stored** with comments, runtime 1.0 s. - Pay-per-event path tested with `ACTOR_TEST_PAY_PER_EVENT=1 ACTOR_MAX_TOTAL_CHARGE_USD=5` (local test price $1/event): 5 `story-exported` events charged, then "Run budget will be exhausted by the next item - stopping gracefully". - Unit tests: `npm test` (20 tests with `node --test`: HN HTML to Markdown, Firebase/Algolia normalisation, comment flattening with depth and caps, thread rendering, Who-is-Hiring parser on six real posts plus synthetic edge cases, URL extraction, thread month). `eslint` and `apify validate-schema` pass. ### Changelog - **0.2.0** (2026-09-24): watch mode (`stateKey`): scheduled runs return and bill only items not exported before; `who-is-hiring` re-checks the previous month's thread; date-sorted search stops at the first fully-seen page. State is saved even if a run fails, so billed items are never billed twice. - **0.1.0** (2026-09-22): initial release. Nine modes, Markdown threads, Who-is-Hiring parser, pay-per-event billing with two events. ### Support Open an issue on the Actor's **Issues** tab. Issues are answered within one business day; bug fixes ship as new builds without changing the input schema. Made by Vellum Labs. # Actor input Schema ## `mode` (type: `string`): What to export. `front`/`new`/`best`/`ask`/`show`/`jobs` read the official Firebase lists; `search` uses Algolia HN Search with **Query**; `story` exports the stories in **Story IDs**; `who-is-hiring` finds the latest 'Ask HN: Who is hiring?' thread (or the ones in **Story IDs**) and parses every job post. ## `query` (type: `string`): Keyword query for `search` mode (Algolia syntax, e.g. `rust async`, `"local-first"`). Ignored in other modes. ## `searchSort` (type: `string`): `date` returns the newest matching stories first (Algolia `search_by_date`); `relevance` uses Algolia ranking (points, comments, recency). ## `storyIds` (type: `array`): HN item IDs or item URLs (e.g. `49522897` or `https://news.ycombinator.com/item?id=49522897`) for `story` and `who-is-hiring` modes. Leave empty in `who-is-hiring` mode to use the newest thread automatically. ## `maxItems` (type: `integer`): Hard cap on dataset items per run: stories in story modes, parsed job posts in `who-is-hiring`. Each stored item is one billing event (`story-exported` or `job-parsed`), so this also caps your cost. ## `sinceDays` (type: `integer`): `0` = no date filter. In list modes stories older than this are skipped; in `search` mode it becomes an Algolia `created_at_i` filter; in `who-is-hiring` mode it selects all hiring threads posted in the window (default: newest thread only). ## `includeComments` (type: `boolean`): Fetch each story's comment tree (one Algolia request per story) and store the comments flattened with their depth, plus the whole thread rendered as Markdown. ## `maxCommentDepth` (type: `integer`): Deepest reply level kept. `0` = top-level comments only, `1` = replies to them, and so on. ## `maxCommentsPerStory` (type: `integer`): Cap on comments stored per story (depth-first order, so parents come before their replies). ## `maxConcurrency` (type: `integer`): Parallel requests against the HN APIs. Algolia allows 10,000 requests per hour per IP; the Actor also backs off automatically on HTTP 429. ## `stateKey` (type: `string`): Name of a key-value store that remembers every story / job post already exported, e.g. `hn-hiring-watch`. With a state key, each run returns and bills **only items that no previous run with the same key exported**, so a daily schedule on `who-is-hiring` gives you just the new job posts, and on `front` or `search` just the new stories. After the first run, `who-is-hiring` also re-checks last month's thread. Leave empty for a normal one-off export. Use a different key per schedule / per filter set. ## Actor input object example ```json { "mode": "front", "query": "", "searchSort": "date", "storyIds": [], "maxItems": 20, "sinceDays": 0, "includeComments": true, "maxCommentDepth": 2, "maxCommentsPerStory": 50, "maxConcurrency": 5, "stateKey": "" } ``` # Actor output Schema ## `items` (type: `string`): No description ## `summary` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "mode": "front", "maxItems": 20, "maxCommentDepth": 2, "maxCommentsPerStory": 50 }; // Run the Actor and wait for it to finish const run = await client.actor("vellumlabs/hn-threads-and-hiring").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "mode": "front", "maxItems": 20, "maxCommentDepth": 2, "maxCommentsPerStory": 50, } # Run the Actor and wait for it to finish run = client.actor("vellumlabs/hn-threads-and-hiring").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "mode": "front", "maxItems": 20, "maxCommentDepth": 2, "maxCommentsPerStory": 50 }' | apify call vellumlabs/hn-threads-and-hiring --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,vellumlabs/hn-threads-and-hiring" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/9RYBXvxGIJKqbhUkL/builds/aRQSdVlT4wmF6hdF5/openapi.json