HN Jobs Scraper

Scrapes the monthly "Ask HN: Who is Hiring?" threads (current month plus up to 24 months back, or specific threads by URL) and the persistent news.ycombinator.com/jobs board, plus full-text search across all HN comments. Each run returns clean, structured records over HN's official API — no headless browser, no API keys, no setup. Runs with Apify limited permissions, and the source is public for inspection (link below).

Verified 2026-09-03 on the live thread: 454 posts across 2 monthly threads in 5 seconds; 19/19 parser tests green. Thread and search rows always include the full post text; runs that fail the quality gate are not billed (see Pricing). No credentials required — the actor reads public HN endpoints and writes only to its own dataset; it never touches your systems.

What you get per listing

• title — listing title as shown on HN
• company — company name with YC-batch suffixes stripped
• role — role without the company prefix
• remote_status — remote, hybrid, onsite, or null when unstated
• url — external job posting URL (falls back to the HN item URL)
• apply_url — direct application link when the post contains one
• hnItemId / hnItemUrl — canonical HN reference
• site — host of the external URL
• postedAt — ISO-8601 UTC timestamp, plus human postedRelative
• author — HN username that posted the listing
• thread_id / thread_month / thread_title — which monthly thread a posting came from (thread rows; null on jobs-board rows)
• description — full job-post text (optional, see below)
• salary_text — pay snippet as written in the post (e.g. "$150k - $190k")
• salary_min_usd / salary_max_usd / salary_basis — parsed numbers (annual, or hourly annualized x2080 for minSalary filtering)
• visa_sponsorship — true/false when the post states it, null when unstated
• location — place as written in the post (e.g. "Berlin", "Remote EMEA")
• emails — contact emails found in the post
• tech_stack — detected stack keywords (80+ terms: python, django, fastapi, postgres, kubernetes, aws, llm, …). Word-boundary matched — "going" never tags "go"; golang posts still match
• is_new / first_seen — cross-run monitoring: true on first sight, with first-seen timestamp (a fresh clone has no such history)
• source / scrapedAt

Sample record

{
  "title": "Acme Robotics | Senior Backend Engineer (Go, Postgres) | Remote",
  "company": "Acme Robotics",
  "role": "Senior Backend Engineer (Go, Postgres)",
  "remote_status": "remote",
  "url": "https://acmerobotics.com/careers/senior-backend",
  "apply_url": "https://acmerobotics.com/careers/senior-backend",
  "site": "acmerobotics.com",
  "hnItemId": "41234567",
  "hnItemUrl": "https://news.ycombinator.com/item?id=41234567",
  "postedAt": "2026-08-28T14:02:11Z",
  "postedRelative": "6 days ago",
  "description": "We're a 12-person team building autonomous...",
  "salary_text": "$150k - $190k",
  "emails": ["jobs@acmerobotics.com"],
  "tech_stack": ["go", "postgres", "kubernetes"],
  "author": "whoishiring",
  "thread_id": "49522897",
  "thread_month": "2026-09",
  "thread_title": "Ask HN: Who is hiring? (September 2026)",
  "is_new": true,
  "first_seen": "2026-09-03T09:15:44Z",
  "source": "who-is-hiring",
  "scrapedAt": "2026-09-03T09:15:44Z"
}

Input options

• source — jobs-board (the persistent /jobs index), who-is-hiring (monthly threads, ~200+ posts each via HN's official API), search (full-text query over all HN comments via searchQuery), or both. Thread discovery is automatic — no hardcoded IDs, works every month
• maxPages — listing pages to follow, 1–50 (about 30 listings per page). The /jobs board is small — typically a few hundred active listings total — so most runs exhaust it long before page 50
• searchQuery — full-text query for source=search (e.g. "Django remote"). maxPages controls result pages (100 per page)
• maxItems — hard cap on listings returned, 0–20000 (default 30, 0 = unlimited). This is your spend control: cost is bounded by maxItems, never by maxPages
• fetchDescription — /jobs board only: follow each item page for the full post text. Thread and search rows always include full text. Set false for titles + metadata only on the board
• descriptionMaxChars — truncate long descriptions (default 8000, 0 = no truncation)
• keyword — keep only listings matching this keyword across title, company and description. Empty = no filtering; targeted runs spend less
• new_only — push only listings never seen by previous runs. A run with nothing new ends cleanly with no bill. Ideal for scheduled monitoring
• month — who-is-hiring month as YYYY-MM (e.g. 2023-04). Empty = current month. Unlocks the full archive back years
• monthsBack — recent monthly threads to include (1-24). Each extra month is one more API call
• threadUrls — specific thread links/IDs, one per line. Overrides discovery
• threadType — hiring (companies posting roles), seeking (candidates looking for work), or freelancer (contract gigs). Monthly thread family for who-is-hiring sources
• remoteOnly — keep only remote roles. Ideal for remote job seekers
• visaOnly — keep only posts explicitly offering visa sponsorship. Ideal for international job seekers
• excludeKeywords — comma-separated terms to drop (title/company/description)
• minSalary — annual USD floor (hourly rates annualized). Rows without a parseable salary are dropped when set

Pricing

Pay per event — you are billed for results, not for run time.

• Job listing — $0.0015 per listing returned
• Full job description — $0.001 per description, billed only when fetchDescription is on and the retrieved text is at least 50 characters. Empty and near-empty fetches are not billed
• A default 30-listing run with descriptions costs at most $0.075 ($0.045 + $0.03), and less when some descriptions come back short or empty. There are no separate platform-usage charges — compute is included
• Quality guard: if fewer than 90% of a run's rows carry a title, URL and HN item id, the run is not billed at all

Author-run comparison vs logiover (Sept 2026)

Head-to-head on the September 2026 thread (217 alive posts), run by me, the actor's author — not an independent test: this repo has the full field-coverage diff, run logs, and both datasets, plus compare.py so you can reproduce it from both public actors. Summary: 217/217 posts returned vs 205; role 100% vs 70%; location 79% vs 87% — the rest ties. Updated 2026-09-03.

Exporting your results

Results land in an Apify dataset: export CSV, JSON or Excel, or forward them to Google Sheets, Airtable, Slack or Zapier using Apify's built-in integrations and webhooks — one record per row. Every record carries a stable hnItemId, so you can dedupe re-runs on that column.

Support

Something look wrong, or a run you think you shouldn't have been billed for? Open a ticket on this actor's Issues tab — I read every one and reply.

Source

Public: https://github.com/hermes-earn-bot/hn-jobs-scraper — inspect the parser, the billing logic, and the tests. This is also what makes the actor eligible for Apify's Open Source Fair Share program.

Reliability notes

• Polite by default: retries 429/5xx with exponential backoff, 1.5s between listing pages
• If Hacker News rate-limits a burst of description fetches, the affected listing still ships with title, company, URL and date — only its description stays empty (and you are not charged for it)
• If a run returns no listings at all, it ends with an error rather than handing you an empty dataset — and a run whose rows fall below the 90% well-formed threshold is not billed. A transient HN outage is the usual cause; re-running a few minutes later normally succeeds

Changelog

• 2026-09-03 — threadType: hiring/seeking/freelancer monthly families (candidate + contract threads, same engine, same schema; freelancer discovery accepts community authors — those threads are human-posted, not whoishiring)

• 2026-09-03 — Provenance fields (author, thread_id/month/title); removed guessy employment_type plus unused score/rank

• 2026-09-03 — Full-text search mode (source=search), first-class location, full post text on every thread row at base price, maxItems=0 unlimited, tech dictionary to 80+ terms

• 2026-09-03 — Multi-month pulls (monthsBack to 24, month archive back years, threadUrls for arbitrary threads); numeric minSalary with hourly annualization; excludeKeywords

• 2026-09-03 — visaOnly filter; newOnly monitoring mode (empty runs end unbilled); open-sourced parser on GitHub