Hacker News Scraper

Export public Hacker News stories, comments, jobs, and polls as structured rows for research and monitoring.

• Search a keyword or phrase, choose live feeds, or enter known item IDs.
• Keep titles, text, authors, scores, timestamps, and thread relationships.
• Add comment and poll-option rows when you need discussion details.
• Read attributable public contact clues, with reasons for missing values.
• Pay $1.50 per 1,000 delivered rows on the Free plan, plus a start fee.

What does Hacker News Scraper do?

Hacker News Scraper collects public content from news.ycombinator.com. Each dataset row represents one item, identified by id and rowType. Ask HN and Show HN posts are stories, not separate row types.

• Search: use searchQuery for matching stories, comments, jobs, polls, or poll options.
• Feeds: use feeds for top, new, best, ask, show, or job.
• Known items: use itemIds for specific posts or comments.
• Descendants: includeDescendants is off by default. In feed and item-ID modes, enabling it adds separate comment and poll-option rows, billed at the same price as other rows.
• Public profile details: enrichDetails is on by default and supplements website and social links in existing rows, not extra profile rows.

What data can you extract from Hacker News?

All five row types share these fields. Values depend on what the public item supplies.

Field	Description	Example
id	Stable string identifier.	"8863"
itemId	Same Hacker News identifier.	"8863"
rowType	Item category; also available as type.	story
by	Public author username.	jane_doe
createdAt	Publication time in UTC.	2026-10-01T00:00:00+00:00
hnUrl	Direct discussion link.	https://news.ycombinator.com/item?id=8863
rootItemId	Selected root or associated story ID.	"8863"
depth	Distance below the root; sometimes unknown.	1
sourceSearchQuery	Search input that produced the row.	AI agents
searchRank	Position in search results.	1
emails	Attributable public email records.	[{"value":"name@example.com","type":"public"}]
phones	Currently empty; phone extraction is not verified.	[]
address	Explicit public location text and recognized components.	{"full":"Example City"}
geo	Coordinates are not inferred.	null
person	Username and an explicit role, when recognizable.	{"name":"jane_doe","role":null}
org	Organization named in a job title.	{"name":"Example Company","id":null}
web	Attributable website and social links.	{"website":"https://example.com","socials":[]}
leadSource	Where contact clues appeared.	detail
leadReveal	Whether a public identity or contact value was visible.	visible
leadAbsences	Reasons for missing contact values.	Keys such as phones

Story rows (story)

Field	Description	Example
title	Story, Ask HN, or Show HN headline.	Show HN: Example project
url	Linked article or project, if present.	https://example.com/project
score	Published score.	42
descendants	Published total comment count.	12

Comment rows (comment)

Field	Description	Example
text	Comment body; may contain HTML.	<p>Example reply</p>
parentId	Immediate parent item.	"8863"
rootTitle	Associated story title, when available.	Show HN: Example project

Job rows (job)

Field	Description	Example
title	Hiring post headline.	Example Company is hiring
url	Public job or careers link.	https://example.com/careers
org	Organization name parsed from the job title.	{"name":"Example Company","id":null}

Poll rows (poll)

Field	Description	Example
title	Poll question.	Which example do you prefer?
parts	Up to 50 option IDs; unknown in search rows.	["9001","9002"]
partsCount	Total options before truncation; unknown in search rows.	2

Poll-option rows (pollopt)

Field	Description	Example
text	Option label.	Option A
score	Published option score.	8

kids also holds at most 50 child IDs; kidsCount keeps the total. Search rows use null for these relationship lists and counts, not an invented zero. Search comment and poll-option depth is also null.

How to scrape Hacker News

1. Open Hacker News Scraper in Apify Console.
2. Fill one target field: searchQuery, itemIds, or feeds.
3. Set maxResults to 5 for a small first run.
4. Click Start and wait for the run to finish.
5. Download the dataset or read it through the API.

An empty input, including {}, starts the form's example search for AI, limited to 3 rows. Normal start and row prices apply; this is not a free preview. Filling one target field selects its mode automatically. If several are filled, searchQuery wins over itemIds, which wins over feeds; ignored targets add no rows or row charges. Clear the prefilled search keyword before selecting another mode.

Keyword search

Enter one phrase in searchQuery. Use recordTypes to select the types you want. Search returns matching rows directly; includeDescendants and maxDepth do not expand search hits into full threads.

Live feeds

Select one or more feeds. Use ask and show for discussions and projects, or job for hiring posts. To include replies, enable includeDescendants and retain comment in recordTypes.

Known item IDs

Enter positive IDs from the number after item?id= in a Hacker News link. With searchQuery empty, only those roots and any enabled descendants are considered, not additional feed roots.

What to fill in

Field	What to enter	Default
searchQuery	One keyword or phrase, up to 200 characters.	Empty; form prefills AI
itemIds	Up to 100 positive item IDs.	Empty
feeds	Any of top, new, best, ask, show, job.	Form prefills all six
recordTypes	Any of story, comment, job, poll, pollopt.	All five
includeDescendants	Add separate descendant rows in feed/item-ID mode.	false
enrichDetails	Supplement public website and social links.	true
skipNoLeads	Keep only rows with contact, location, organization, website, or social clues.	false
maxDepth	Descendant levels, 0–10; roots are level 0.	2
maxResults	Maximum rows per input, 1–2000.	70; form prefills 3
maxFetches	Candidate-reading cap per collection part, 1–5000; at least maxResults.	500

Search:

{
  "searchQuery": "AI agents",
  "recordTypes": ["story", "comment"],
  "maxResults": 5
}

Feed:

{
  "feeds": ["ask"],
  "includeDescendants": true,
  "recordTypes": ["story", "comment"],
  "maxDepth": 1,
  "maxResults": 5
}

Known item:

{
  "itemIds": [8863],
  "includeDescendants": true,
  "maxResults": 5
}

maxItems and limit are aliases for maxResults. query, keyword, search, and q are aliases for searchQuery. Close field-name misspellings can be corrected; other unknown fields are ignored with a note.

Each normal run writes INPUT_REPORT to its default key-value store before collection. It records mode selection, corrections, ignored fields, and example input use. INPUT_ECHO contains normalized input; RUN_SUMMARY records results; ERROR_SUMMARY explains failures.

Invalid targets stop before any row is charged, with a message shaped like INVALID_INPUT: <field> — <how to fix>. Other validation errors, such as maxFetches below maxResults, name the constraint without necessarily using that prefix. A start fee can still apply.

Output example

Illustrative story row with placeholder content, not a captured live result:

{
  "itemId": "8863",
  "id": "8863",
  "rowType": "story",
  "type": "story",
  "by": "jane_doe",
  "createdAt": "2026-10-01T00:00:00+00:00",
  "title": "Show HN: Example project",
  "text": null,
  "url": "https://example.com/project",
  "hnUrl": "https://news.ycombinator.com/item?id=8863",
  "score": 42,
  "descendants": 0,
  "parentId": null,
  "kids": [],
  "kidsCount": 0,
  "parts": [],
  "partsCount": 0,
  "deleted": false,
  "dead": false,
  "sourceFeeds": ["show"],
  "rootItemId": "8863",
  "rootTitle": "Show HN: Example project",
  "depth": 0,
  "source": "hacker-news-firebase",
  "scrapedAt": "2026-10-01T00:01:00+00:00",
  "phones": [],
  "emails": [],
  "address": {
    "street": null, "unit": null, "locality": null,
    "region": null, "postalCode": null, "country": null, "full": null
  },
  "geo": null,
  "person": {"name": "jane_doe", "role": null},
  "org": null,
  "web": {"website": null, "socials": []},
  "leadSource": "detail",
  "leadReveal": "visible"
}

You can export the dataset as JSON, CSV, Excel, XML, RSS, or an HTML table.

Why use this Hacker News scraper?

• Pay for delivered rows. Candidate reads do not create separate row charges.
• Keep delivered data. Rows already written remain available after a stop or timeout; rows still being collected are not guaranteed to arrive.
• Understand failures. Inspect the status message and ERROR_SUMMARY rather than treating every empty dataset as no matches.
• Preserve missing values. Unavailable fields remain empty, with contact gaps explained in leadAbsences.
• Combine inputs. Several feeds or item IDs can share a run; keyword input is one phrase.
• No proxy setup. You do not need to set up proxies.

How much does it cost to scrape Hacker News?

The Actor uses pay-per-event pricing. Platform usage is included in these prices.

Apify plan	Price per start event	Price per row	Price per 1,000 rows
Free	$0.005	$0.0015	$1.50
Starter	$0.005	$0.0015	$1.50
Scale	$0.005	$0.0013	$1.30
Business	$0.005	$0.0011	$1.10

Platinum and Diamond have the same row price as Business. Start events are charged per GB of run memory, with a minimum of one event. At the default 512 MB, the start charge is $0.005.

• maxResults applies to each search input, feed, or item ID. It defaults to 70; larger requests are collected in parts of up to 70 rows.
• Root and descendant rows share that input's allowance; comments are not free extras.
• Free Apify plan runs are capped at 200 rows per run and 10 total runs of this Actor, as set by the developer. Paid plans remove that run-count restriction.
• The whole run stops at your Maximum cost per run (maxTotalChargeUsd in the API). Availability and filters can reduce the delivered count.

Example: one search returning 5 rows on the Free plan at 512 MB costs 5 × $0.0015 + $0.005 = $0.0125.

What happens when the run timeout is too short

Some Hacker News collections, such as several feeds with includeDescendants: true and a high maxDepth, or a large maxResults, can take longer than the run timeout you chose. In that case the run does not fail. It ends early with the status message "Queued on the collection backend; this run will resume automatically when results are ready (about N s)." and shows as ABORTED for a while.

When the results are ready, the same run starts again by itself, reports "Resumed: delivering N rows.", writes the rows to the same dataset and finishes as SUCCEEDED. Rows that were already delivered are skipped, so no row is delivered or charged twice. You do not need to start another run.

• This can happen at most 3 times per run. To avoid it, keep the default timeout of 3600 seconds.
• The synchronous API endpoints (run-sync, run-sync-get-dataset-items) wait at most 300 seconds and can return an empty dataset while the run is paused. For larger jobs, start the run asynchronously and read the dataset after the run reaches SUCCEEDED, or use a webhook on the "succeeded" event.

Use with AI agents and the API

Apify MCP server

Claude, ChatGPT, Cursor, and VS Code clients with remote MCP support can use:

https://mcp.apify.com?tools=lentic_clockss/hacker-news-scraper

Sign in with Apify through OAuth, or send Authorization: Bearer <YOUR_APIFY_TOKEN>. The tool starts a run and returns run and dataset IDs; the agent reads rows with get-dataset-items. For clients without remote support, run npx @apify/actors-mcp-server --tools lentic_clockss/hacker-news-scraper with APIFY_TOKEN configured.

Example prompt: “Search Hacker News for AI agents and return five stories with titles, scores, and discussion links.”

Apify API

from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("lentic_clockss/hacker-news-scraper").call(
    run_input={"searchQuery": "AI agents", "maxResults": 5}
)
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(row)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('lentic_clockss/hacker-news-scraper')
  .call({ searchQuery: 'AI agents', maxResults: 5 });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

For HTTP, send the input as JSON to POST https://api.apify.com/v2/acts/lentic_clockss~hacker-news-scraper/runs with an Authorization: Bearer <YOUR_APIFY_TOKEN> header. After completion, read GET https://api.apify.com/v2/datasets/<datasetId>/items?format=json.

Standby HTTP endpoint

Base URL: https://lentic-clockss--hacker-news-scraper.apify.actor.

Route	Returns
GET /	Service information.
GET /health	Service readiness.
GET /input-example	A small example input.
GET /openapi.json	The OpenAPI document.

These are read-only information routes. They do not start collection or return scraped rows. Use a normal Actor run for data.

Integrations

Use the Integrations tab to connect results to Google Sheets, Make, Zapier, n8n, Slack, Airtable, or your webhook. Use Apify Schedules for recurring runs.

FAQ

Is it legal to scrape Hacker News?

This Actor collects public data. You are responsible for checking applicable terms and data protection law; processing personal data requires a lawful basis. Public visibility alone is not permission for every use.

Why did my run end as ABORTED?

Check the run's status message and log to identify why it stopped, including whether it was manually aborted. This Actor does not promise automatic recovery. Inspect the dataset for rows already delivered before starting another run.

How many results can I get?

Request 1–2000 rows per input with maxResults. Actual counts depend on available matches, filters, descendant depth, reading limits, and your cost cap. Free-plan runs stop at 200 rows.

Why are some fields empty?

Not every item has a title, external URL, contact detail, or score. Search rows omit relationship lists that the search results do not supply. Empty values are not fabricated or replaced with guessed data.

Can I collect comments without collecting whole threads?

Yes. Search with recordTypes: ["comment"]. For replies beneath known posts, use item IDs or feeds with includeDescendants: true and include comment in recordTypes.

Why can a selected type have zero results?

A selected feed or keyword may have no matching items of that type. Polls and jobs are not guaranteed in every search. Filters and skipNoLeads can further reduce results.

Does a username count as a contact clue?

No. With skipNoLeads: true, a username alone does not qualify. The row needs a contact, location, organization, website, or social clue. An article URL is not automatically the author's website.

Can the HTTP endpoint start a collection?

No. Its routes provide service information, an input example, and the API description. Start collection through Apify Console, MCP, or the Apify API.

Limitations

• Live feeds and rankings change; a run is a snapshot, not a complete historical archive.
• Search accepts one phrase, with no date-window or automatic translation controls.
• Descendant depth and row limits can prevent a complete thread export.
• Public profile enrichment is limited to 25 distinct authors and about 20 seconds per collection part. Failures or limits leave gaps in leadAbsences.
• Phones, street numbers, postal codes, and coordinates are not verified outputs. Organization IDs are not supplied; roles require explicit recognizable wording.
• Individual malformed items can be skipped; collection failures may produce partial results or an error, not necessarily a complete dataset.

Related scrapers

More social media and video scrapers from the same publisher:

• TikTok Scraper
• Reddit Scraper
• YouTube Shorts Scraper
• YouTube Research Scraper

Browse all of our scrapers at apify.com/lentic_clockss.

Support

Report a problem or request a field through the Issues tab of this Actor's page. Include the run ID and the input you used.