HN Who Wants to Be Hired Scraper

Hacker News Who Wants to Be Hired

Apify Actor that turns one monthly Ask HN: Who wants to be hired? thread into structured candidate rows (location, remote, skills, contact when published, etc.).

Zero setup: just click Start. Leave every field empty and the Actor automatically picks the latest monthly thread. Pass a storyId only if you want an older month.

What it does

1. Uses the latest monthly thread by default, or the storyId / storyUrl you pass.
2. Loads the story and top-level comment ids from the official Hacker News Firebase API.
3. Fetches each top-level comment (nested replies are skipped).
4. Parses common free-text fields with simple regex/keywords (Location, Remote, Willing to relocate, Technologies / Tech, Email).
5. Pushes one dataset item per candidate comment.

Uncertain fields stay null. No LLM / AI API calls. No login. No private messages.

Data source

• Official API only: Hacker News Firebase API — docs: github.com/HackerNews/API
• Endpoints used: item/{id}.json for the story and each top-level kid
• Optional discovery (not required at runtime): HN Algolia Search API to find monthly thread ids

This Actor does not scrape G2, Etsy, Meta, Indeed, or other third-party boards.

Input

Field	Type	Required	Default	Description
storyId	integer	one of	—	HN item id of the monthly thread
storyUrl	string	one of	—	e.g. https://news.ycombinator.com/item?id=49922568
maxComments	integer	no	150	Cap on top-level kids fetched
requestsPerSecond	number	no	4	Polite rate limit (stay ≤ ~5)

Example (also in .actor/INPUT_EXAMPLE.json):

{
  "storyId": 49922568,
  "storyUrl": "https://news.ycombinator.com/item?id=49922568",
  "maxComments": 150,
  "requestsPerSecond": 4
}

Either storyId or storyUrl is enough; if both are set, storyId wins.

Output

Each dataset row:

Field	Type	Notes
username	string | null	HN by
commentId	integer
commentUrl	string	Link to the comment
storyId	integer	Parent thread
storyUrl	string
rawText	string	HTML stripped, entities decoded
location	string | null	From Location:
remote	boolean | null	From Remote:
willingRelocate	boolean | null	From Willing to relocate / Relocate
skills	string[]	From Technologies / Tech / Skills
emailOrContact	string | null	Only if explicitly written in the comment
postedAt	string | null	ISO-8601 from HN time

See SAMPLE_OUTPUT.json for real rows from a local run.

Sample run (local)

• Thread: Ask HN: Who wants to be hired? (October 2026) (storyId 49922568)
• Cap: maxComments=150 of 799 top-level kids
• Result: 140 structured rows (commentsFetched=150, skipped=10, errors=0)
• Artifacts: SAMPLE_OUTPUT.json, RUN_LOG.txt

Pricing (pay-per-event)

Intended Store pricing: $2 per 100 structured candidate records → $0.02 per dataset item.

PPE event prices are configured in Apify Console (not in actor.json). See PRICING.md for the console TODO:

• Prefer synthetic event apify-default-dataset-item at $0.02 (charges automatically on each Actor.pushData() to the default dataset)
• Optionally keep apify-actor-start at the Apify default

Limitations

• Top-level comments only (no nested replies)
• Heuristic parsing; free-form posts may leave fields null
• Space-separated skill lists without commas may stay as one token
• maxComments caps volume for rate/cost safety
• Does not verify emails or contact candidates

Legal / ToS note

Uses public Hacker News data through the official Firebase API (HackerNews/API). Respect polite rate limits. Contact details appear only when authors publish them in their own comments; handle under applicable privacy and anti-spam rules. Do not use this Actor to spam or harass posters.

Local development

npm install
# edit storage/key_value_stores/default/INPUT.json or copy .actor/INPUT_EXAMPLE.json
npm start
# dataset → storage/datasets/default/

With Apify CLI (after login, later): apify run then apify push.

Memory limits for platform runs: 256–1024 MB (see .actor/actor.json).

License

MIT