Hacker News Scraper: Stories, Comments, Who Is Hiring

Use this to search Hacker News by keyword, pull the front page and the Top, New, Best, Ask HN, Show HN and Jobs lists, fetch every comment of a story, look up a user and their submissions, and extract the monthly "Who is hiring?" threads, with points, comment counts, domains, authors and ISO timestamps. It reads the official Algolia Hacker News Search API and the official Hacker News Firebase API. No login, no cookies, no API key, no proxy.

Every row carries up to 34 fields: id, HN page URL, title, outbound URL, domain, plain-text body, author and author URL, points, comment count, ISO creation time and Unix timestamp, parent and story ids, story title for comments, nesting depth, list rank, child count, matched query and, for Who is hiring posts, a parsed headline, remote flag, apply emails and apply links.

Run in 30 seconds

1. Open the Input tab. The prefilled values (mode Search, term web scraping, 25 results) are a ready-to-run test.
2. Click Start. The test run finishes in a few seconds.
3. Open the Stories table in the Output tab, or download the dataset as JSON, CSV, Excel or HTML.
4. Change What to scrape to switch between search, lists, story comments, users and Who is hiring.

What does this Actor do?

Mode	What you get	Typical use
Search	Stories, comments, Ask HN, Show HN, jobs or polls matching keywords, filtered by points, comments, author and date	Brand and competitor monitoring, research, trend tracking
List	Front page (with points), Top, New, Best, Ask HN, Show HN or Jobs, ranked	Daily digests, launch watching
Story comments	The story plus every comment, flattened with a depth number	Sentiment analysis, thread summaries, datasets
User	Profile (karma, about, join date, submission count) plus their stories and/or comments	Founder and author research
Who is hiring	The latest or any month's Who is hiring, Who wants to be hired or Freelancer thread, one row per post, with keyword and remote filters	Recruiting, job-market data, lead lists

Search can go past Algolia's 1,000-hit cap: in date order the Actor keeps walking the time window backwards, so a broad query can return thousands of distinct rows.

Why use it

• One Actor for every Hacker News job. Search, front page and lists, full comment trees, users and hiring threads share one input form, one dataset schema and one bill.
• Who is hiring parsed for you. Each post gets a headline, a remote flag, apply emails and apply links, and you can filter by keyword (any or all) and remote only.
• Both official APIs. Algolia gives ranked search with numeric filters; Firebase gives the live Top, New, Best, Ask, Show and Jobs lists. Neither needs a key.
• Clean text. HTML in posts and comments is converted to plain text with links kept.
• Predictable cost. One flat price per row returned. Rows skipped by your filters and duplicates are never charged, and the Actor respects your "Maximum cost per run" setting exactly.

Input

• What to scrape: Search, List, Story comments, User or Who is hiring.
• Search terms and Content type (stories, comments, Ask HN, Show HN, front page, jobs, polls), Sort by relevance or newest, Minimum points, Minimum comments, Posted since / until, Only from this user.
• List: top, new, best, ask, show, job or front page.
• Story IDs or URLs, Include the story row, Max comment depth.
• Usernames and what to return besides the profile.
• Thread type, Month (latest, 2026-10 or October 2026), Keywords, Keyword match, Remote only.
• Max results and Include child comment ids.

Example input

{ "mode": "search", "searchTerms": ["web scraping"], "contentType": "story", "sortBy": "relevance", "minPoints": 50, "maxItems": 25 }

{ "mode": "whoIsHiring", "hiringMonth": "latest", "hiringKeywords": ["rust"], "remoteOnly": true, "maxItems": 100 }

Output sample

{
  "recordType": "story",
  "id": 22180559,
  "hnUrl": "https://news.ycombinator.com/item?id=22180559",
  "title": "Congrats! Web scraping is legal! (US precedent)",
  "url": "https://parsers.me/us-court-fully-legalized-website-scraping-and-technically-prohibited-it/",
  "domain": "parsers.me",
  "author": "ehurynovich",
  "points": 1057,
  "commentCount": 388,
  "createdAt": "2020-01-29T14:06:37.000Z",
  "storyId": 22180559,
  "childCount": 70,
  "query": "web scraping",
  "source": "search"
}

Fields that do not apply to a row type are null, so every row has the same columns.

Use cases

• Track mentions of your product, a competitor or a technology and alert on new high-point threads.
• Build a dataset of tech opinions from comment trees for research or model evaluation.
• Pull every Who is hiring post that mentions your stack and is remote, and export the apply emails.
• Rank the domains that reach the front page, or the authors who post the most successful Show HN launches.

Tips for good results

• Use Sort by newest first with Posted since for incremental monitoring: store the highest createdTimestamp you saw and pass it next time.
• Use Minimum points to keep only stories that got traction.
• For comment datasets, set Max comment depth to 1 for top-level replies only.
• Who is hiring threads appear on the first working day of each month; use latest for the newest.

Field reference

recordType, id, hnUrl, title, url, domain, text, author, authorUrl, points, commentCount, createdAt, createdTimestamp, parentId, storyId, storyTitle, storyUrl, depth, rank, childCount, childIds, tags, query, source, userKarma, userAbout, userAccountCreated, userSubmissionCount, hiringHeadline, isRemote, applyEmails, applyUrls, threadTitle, threadMonth.

Pricing and cost control

Pay per result: $0.000186 per row on the free plan, falling to $0.000093 on the Gold plan (Bronze $0.000171, Silver $0.000145), plus a $0.00005 start fee per run. A 1,000-row run costs about $0.19 on the free plan. There are no subscriptions, proxy fees or hidden compute add-ons: the Actor needs no proxy and runs in 256 MB. To cap spend, set Max results and the run-level Maximum cost per run; the Actor reads the remaining budget before every row and stops the moment it would be exceeded.

How this compares with the other Hacker News scrapers (live Store prices read through the Apify API, 2026-10-29, free-plan rate, 1,000-row run including start fees):

Actor	Users	Per row	Start fee	Cost for 1,000 rows
This Actor	new	$0.000186	$0.00005	$0.19
gentle_cloud/hacker-news-scraper	222	$0.0002	none	$0.20
epctex/hackernews-scraper	196	$0.0003	$0.00005	$0.30
ryanclinton/hackernews-search	161	$0.005	$0.00005	$5.00

At the free-plan rate this is the lowest per-row price in the table. The Actor needs no login or key, and the prefilled test run finishes in a few seconds.

Limits

• Up to 100,000 rows per run (the Max results setting).
• Relevance-ordered search is capped by Algolia at about 1,000 hits per query; newest-first search with a date range goes deeper.
• A run stops starting new work after about 4.5 minutes and keeps what it has delivered.
• Only public Hacker News data is read: no login, no private data. A deleted or dead item that the APIs do not return is skipped.
• The Who is hiring parser extracts headline, remote flag, emails and links from free text with simple rules, so a post written in an unusual format can have empty parsed fields.

Integrations and scheduling

Schedule the Actor from the Schedules tab (hourly, daily, weekly) to build a monitoring feed. Send results onward with the Apify integrations for Slack, email, Google Sheets, Make, Zapier or n8n, or call the Actor from your own code through the REST API.

Fair use and data

The Actor reads Hacker News data through the two public, officially documented APIs (the Algolia HN Search API and the Hacker News Firebase API). It does not log in, does not scrape the website's HTML pages, does not read private data, and spaces its requests politely. Posts and comments are public content written by their authors; you remain responsible for using the data in line with the Hacker News terms and applicable privacy law, and for handling usernames and personal text with care. Do not use it for spam or harassment.

Use with the API, n8n, Make, Zapier or AI agents

API (one call, returns the rows as JSON):

curl -X POST "https://api.apify.com/v2/acts/transparent_meteorite~hacker-news-search-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"search","searchTerms":["web scraping"],"maxItems":25}'

The call waits up to 5 minutes. For bigger inputs, start the run with /runs instead and read the dataset when it finishes.

Python

from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("transparent_meteorite/hacker-news-search-scraper").call(run_input={"mode": "search", "searchTerms": ["web scraping"], "maxItems": 25})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row)

• n8n, Make, Zapier: use the Apify app's "Run Actor" step with actor transparent_meteorite/hacker-news-search-scraper, paste the input JSON, then map the dataset items to your sheet or CRM.
• Clay: add an HTTP API column that calls the run-sync endpoint above.
• Schedules: in Apify Console, Schedules, run it daily or weekly with the same input for a steady feed.
• AI agents (MCP): connect the Apify MCP server at https://mcp.apify.com?tools=transparent_meteorite/hacker-news-search-scraper to Claude, Cursor or any MCP client, and your agent can run Algolia HN Scraper on request.

FAQ

Do I need a Hacker News account or API key? No. Both APIs are open and keyless.

How far back does search go? All the way to 2006. Relevance order is capped by Algolia at about 1,000 hits per query; use newest-first with a date range to go deeper.

Can it get every comment of a story? Yes: Story comments returns the whole tree, flattened with a depth number and the parent id.

Does it get the live front page? Yes. The List mode with Front page uses the same ranking Algolia exposes; Top, New, Best, Ask, Show and Jobs come from the Firebase lists.

Which month does Who is hiring use? latest picks the newest thread; pass 2026-09 or September 2026 for an older one.

What happens when a user or story does not exist? It is skipped, a warning is logged and nothing is charged for it.

What happens if I hit my maximum cost per run? The Actor stops cleanly, keeps what it already delivered, and shows a clear message in the run status.

Why did I get fewer rows than Max results? Your terms or filters matched fewer items. The final status message says so.

Related actors

• Bluesky Scraper: Posts, Profiles, Followers & Feeds for Bluesky posts and accounts.
• Show HN & npm PyPI Signals: Hacker News Tracker to join Show HN and Launch HN posts with npm and PyPI package data.
• Lemmy Scraper: Communities, Posts & Comments for link-aggregator communities on the fediverse.
• Mastodon Hashtag Timeline Exporter for posts and trends from Mastodon instances.

Support

Found a bug or need a field? Open an issue on the Issues tab with the run URL and you will get a reply there. Feature requests are welcome.

Changelog

• 0.1: First build: search, lists, story comments, users and Who is hiring threads.