Hacker News Threads & Who is Hiring Exporter

Export Hacker News stories with their full comment threads as clean Markdown, search HN by keyword and date, and turn the monthly "Ask HN: Who is hiring?" thread into structured job posts (company, role, location, remote, salary, visa, links). Everything comes from the official Firebase HN API and the Algolia HN Search API, so there is no HTML scraping, no proxies and no blocking.

What you get that plain HN scrapers do not:

• Whole threads, ready for an LLM. Story text and comments are converted from HN's markup to Markdown, comments are flattened with depth and parent_id, and the entire thread is also rendered as one Markdown document (replies nested as blockquotes) in markdown.
• One request per thread. Comment trees come from Algolia's items/<id> endpoint, which returns the full nested tree in a single call, so a front page with comments is about 40 API requests, not thousands.
• Who is hiring, parsed. Each top-level post of the hiring thread becomes a record with company, role, location, remote / remote_detail / onsite / hybrid, salary, visa, employment_type, yc_batch, urls, the original headline and the full post as Markdown. Pick the newest thread automatically or pass thread IDs.
• Watch mode: only what is new since your last run. Set a stateKey and schedule the Actor: each run returns (and bills) only the job posts or stories that no earlier run with that key exported. A daily who-is-hiring schedule becomes a feed of new HN job posts, typically a few dozen a day early in the month and a handful later, instead of re-buying the whole 300-500 post thread.
• Nine modes. Front page, newest, best, Ask HN, Show HN, Jobs, keyword search (newest or relevance, with a date window), specific story IDs, Who is hiring.
• Rate-limit aware. Concurrency capped (default 5), automatic exponential back-off on HTTP 429 / 5xx, Retry-After honoured.

Who is it for

• AI agents and MCP clients that need HN discussion around a topic as context (call it from the Apify MCP server, then read the dataset).
• Research and monitoring: track what HN says about your product, library or competitor, on a schedule, as Markdown you can diff.
• Recruiting and job search: a structured monthly feed of HN hiring posts filtered by remote, location or salary.
• Knowledge bases: drop thread Markdown into Obsidian, Notion or a Claude Code wiki.

Input

Field	Type	Default	Notes
mode	string	front	front, new, best, ask, show, jobs, search, who-is-hiring, story.
query	string	""	Search terms for search mode (Algolia syntax).
searchSort	string	date	date (newest first) or relevance.
storyIds	string[]	[]	HN item IDs or item URLs for story and who-is-hiring modes.
maxItems	integer	100	Cap on stories (or job posts in who-is-hiring). Caps cost.
sinceDays	integer	0	Only items newer than N days (0 = no filter). In who-is-hiring mode, all hiring threads in the window.
includeComments	boolean	true	Fetch comment trees and render thread Markdown.
maxCommentDepth	integer	3	0 = top-level comments only.
maxCommentsPerStory	integer	200	Depth-first cap per story.
maxConcurrency	integer	5	Parallel API requests (max 10).
stateKey	string	""	Watch mode. Name of a key-value store that remembers exported IDs; runs return only unseen items.

Example inputs:

{ "mode": "front", "maxItems": 20, "includeComments": true, "maxCommentDepth": 2, "maxCommentsPerStory": 50 }

{ "mode": "search", "query": "local-first sync", "sinceDays": 90, "maxItems": 50 }

{ "mode": "who-is-hiring", "maxItems": 500 }

Daily feed of new HN job posts (schedule it once a day):

{ "mode": "who-is-hiring", "maxItems": 500, "stateKey": "hn-hiring-watch" }

Output

Story modes produce one item per story:

{
    "type": "story",
    "id": 49797323,
    "hn_url": "https://news.ycombinator.com/item?id=49797323",
    "title": "Can gzip be a language model?",
    "url": "https://nathan.rs/posts/gzip-lm/",
    "domain": "nathan.rs",
    "story_type": "story",
    "points": 280,
    "author": "networked",
    "created_at": "2026-09-22T06:40:12.000Z",
    "comment_count": 132,
    "text_markdown": "",
    "comments": [
        {
            "id": 49797901,
            "parent_id": null,
            "author": "someone",
            "created_at": "2026-09-22T07:12:44.000Z",
            "depth": 0,
            "text_markdown": "This is the classic compression-as-classification trick..."
        },
        {
            "id": 49798010,
            "parent_id": 49797901,
            "author": "other",
            "created_at": "...",
            "depth": 1,
            "text_markdown": "> classic trick\n\nRight, and ..."
        }
    ],
    "comments_stored": 50,
    "markdown": "# Can gzip be a language model?\n\n[nathan.rs](https://nathan.rs/posts/gzip-lm/) · 280 points · by networked · 2026-09-22T06:40:12.000Z · 132 comments · [HN](https://news.ycombinator.com/item?id=49797323)\n\n---\n\n## Comments (50 stored)\n\n**someone** · 2026-09-22T07:12:44.000Z · [#](https://news.ycombinator.com/item?id=49797901)\n\nThis is the classic ...\n\n> **other** · ... \n>\n> Right, and ...",
    "fetched_at": "2026-09-22T14:42:51.358Z"
}

who-is-hiring mode produces one item per job post:

{
    "type": "job",
    "id": 49522903,
    "hn_url": "https://news.ycombinator.com/item?id=49522903",
    "story_id": 49522897,
    "story_title": "Ask HN: Who is hiring? (September 2026)",
    "thread_month": "2026-09",
    "author": "thelastofthelas",
    "created_at": "2026-09-01T15:01:54.000Z",
    "headline": "Modash.io | Senior Product Engineer | Remote (Europe) | Full-time | €75k–110k | https://modash.io",
    "company": "Modash.io",
    "role": "Senior Product Engineer",
    "location": "Remote (Europe)",
    "remote": true,
    "remote_detail": "Europe",
    "onsite": false,
    "hybrid": false,
    "salary": "€75k–110k",
    "visa": null,
    "employment_type": "full-time",
    "yc_batch": null,
    "tags": [],
    "urls": ["https://modash.io", "https://modash.io/engineering", "https://apply.workable.com/modash/j/C1507B65C3"],
    "markdown": "Modash.io | Senior Product Engineer | Remote (Europe) | Full-time | €75k–110k | [https://modash.io](https://modash.io)\n\nModash helps brands find, manage, and pay creators...",
    "fetched_at": "2026-09-22T14:43:18.526Z"
}

The structured job fields are parsed from HN's Company | Role | Location | REMOTE | Full-time | $salary | url convention with heuristics; headline, tags (unclassified headline segments such as ONSITE, Visa sponsorship available) and markdown always keep the original text so nothing is lost. visa is sponsored, not-sponsored or null.

SUMMARY.json in the key-value store holds run statistics (items stored, skipped, API requests, retries, duration).

Pricing

This Actor uses pay-per-event pricing:

Event	Price	When
story-exported	$0.001 per story ($1 per 1,000)	Once per story written to the dataset, comments included. Deleted, dead and filtered-out stories are never charged.
job-parsed	$0.001 per job post ($1 per 1,000)	Once per Who-is-Hiring post written to the dataset.
Actor start	$0.005 per run	Apify's standard start event; covers the first seconds of compute.

The front page with comments costs about $0.03; a full monthly hiring thread (about 250-450 posts) about $0.45. Set Max total charge per run to hard-cap spend; the Actor stops gracefully before exceeding it. Platform usage (compute, storage) is included in the event prices.

Tips

• Big threads: maxCommentsPerStory is depth-first, so raising maxCommentDepth without raising the cap keeps you on the first few top-level branches. For "top comments only" use maxCommentDepth: 0.
• Monitoring a topic: schedule search mode daily with searchSort: date and a stateKey (for example hn-watch-postgres). Each run stops as soon as it reaches stories an earlier run exported, so you pay only for new ones and no downstream deduplication is needed.
• New job posts every day: schedule who-is-hiring with a stateKey. The first run exports the current thread (up to maxItems); every later run exports only posts added since, and also re-checks last month's thread so late posts are not lost when the new month's thread appears.
• All hiring threads of a year: who-is-hiring with sinceDays: 365 processes every monthly thread in the window (maxItems still caps the total).
• Specific threads: paste HN item URLs into storyIds with mode: story.

Limits

• Firebase lists return at most 500 (front, new, best) or 200 (ask, show, jobs) IDs, so those modes cap at that many stories per run. Use search with sinceDays for older or larger sets (Algolia paginates up to 1,000 hits per query).
• Algolia allows 10,000 requests per hour per IP; at default concurrency this Actor stays far below it and backs off on 429.
• Comment trees are read from Algolia and can lag the live site by a few minutes; points and comment_count come from the same source as the listing (Firebase for list modes, Algolia for search).
• Watch mode remembers the newest 50,000 exported IDs per stateKey. It tracks IDs, not edits: a job post edited after it was exported is not exported again. Use a separate stateKey per schedule or per filter set, otherwise one schedule "uses up" items for the other. Deleting the key-value store resets the watch.
• In watch mode front/best return stories that are new to you, not stories that are new on HN; for "newly submitted" use new or search sorted by date.
• Text is what HN returns: text_markdown is empty for link posts, and deleted comments are skipped (their replies are kept one level up).

Verified

Local runs on 2026-09-22 with apify run (Apify CLI 1.10.0, Apify SDK 3.7.2, Node 24):

• front (prefill input: 20 stories, comments depth 2, 50 per story): 41 API requests (1 list + 20 Firebase items + 20 Algolia trees), 0 retries, 0 failed, 20 stories stored, runtime 4.3 s.
• who-is-hiring (maxItems: 50): 2 API requests (thread lookup + one 497 KB tree with 258 top-level posts), 50 job posts stored, runtime 2.0 s. Parse coverage on those 50 live posts: company 50, location 45, URLs 42, employment type 33, role 26, remote 25, salary 19, visa 4.
• search (rust async, sinceDays: 365, maxItems: 5): 5 requests, 5 stories stored with comments, runtime 1.0 s.
• Pay-per-event path tested with ACTOR_TEST_PAY_PER_EVENT=1 ACTOR_MAX_TOTAL_CHARGE_USD=5 (local test price $1/event): 5 story-exported events charged, then "Run budget will be exhausted by the next item - stopping gracefully".
• Unit tests: npm test (20 tests with node --test: HN HTML to Markdown, Firebase/Algolia normalisation, comment flattening with depth and caps, thread rendering, Who-is-Hiring parser on six real posts plus synthetic edge cases, URL extraction, thread month). eslint and apify validate-schema pass.

Changelog

• 0.2.0 (2026-09-24): watch mode (stateKey): scheduled runs return and bill only items not exported before; who-is-hiring re-checks the previous month's thread; date-sorted search stops at the first fully-seen page. State is saved even if a run fails, so billed items are never billed twice.
• 0.1.0 (2026-09-22): initial release. Nine modes, Markdown threads, Who-is-Hiring parser, pay-per-event billing with two events.

Support

Open an issue on the Actor's Issues tab. Issues are answered within one business day; bug fixes ship as new builds without changing the input schema.

Made by Vellum Labs.