Hacker News Data Scraper

🟠 Hacker News Data Scraper – Extract Posts, Comments, Points & User Data

The Hacker News Data Scraper pulls structured post and comment data out of news.ycombinator.com so you can analyse what the technology community is actually reading, arguing about and hiring for. Point it at the front page, Show HN, the jobs board, or any individual item thread, and it returns clean JSON records with titles, outbound links, points, comment counts, submitter usernames and the full comment tree.

Hacker News is one of the highest-signal sources of early technology discourse on the web — new product launches surface there before they hit tech press, engineering hiring trends show up on the jobs page months before they appear in market reports, and comment threads contain candid practitioner opinion that no survey will ever capture. This Hacker News scraper turns that firehose into a dataset you can query, chart or feed into a model, without writing a line of HTML parsing yourself.

---

📊 What Data Can You Extract with This Hacker News Scraper?

Every run produces dataset items grouped around the following categories of Hacker News data.

Category	Fields	What you get
🆔 Item identity	id, url, scrapedType	The Hacker News item ID, its canonical HN permalink, and whether the record is a post or a comment
📰 Story content	title, link	The submitted headline and the outbound destination URL the story points to
📈 Engagement metrics	points, numberOfComments	Upvote score and the discussion volume attached to the item
👤 Submitter data	postedUserName, postedUserLink	The HN username that submitted the item and a direct link to their profile
💬 Discussion thread	comments	The array of comments attached to the item, optionally nested by indentation depth
🕒 Recency	age	The relative age string shown on the site, such as "1 day ago"

The field that does the most work in analysis is points paired with age. Score alone tells you what performed well; score relative to how long an item has been live tells you what is currently accelerating — which is the difference between a retrospective ranking and an early-warning signal.

---

🌟 Key Features of the Hacker News Scraper

Feature	Description
🎯 Multi-section coverage	Scrape the front page, /front, /show, /jobs, or any item?id= thread by adding URLs to startUrls
💬 Full comment extraction	Comments are collected into the comments array alongside the parent post, not as disconnected records
🌲 Optional thread hierarchy	Turn on enableCommentHierarchy to nest replies according to their indentation, preserving conversation structure
🔢 Item cap	maxItems puts a hard ceiling on output, keeping test runs cheap and predictable
📄 Pagination control	endPage sets how deep into a listing's pagination the Hacker News scraper will walk
🧩 Custom output extension	extendOutputFunction accepts a JavaScript function so you can attach your own derived fields to every record
⚡ HTML-based, no browser	Built on plain HTTP requests and HTML parsing rather than a headless browser, so runs stay fast and light
👤 Submitter attribution	Every post carries postedUserName and postedUserLink, making user-level analysis straightforward
🔄 Automatic proxy rotation	Requests are routed through rotating proxies managed by the actor, with no proxy configuration needed from you

---

🚀 Why Choose This Hacker News Scraper?

Posts and their discussions arrive together. Instead of scraping a listing and then chasing every thread separately, each post record carries its comments array inline. One dataset, one join-free structure, ready for text analysis.

Section-aware by design. /show behaves differently from /jobs, which behaves differently from an item page. The scraper accepts all of them in the same startUrls list and tags each record with scrapedType so you always know what shape you are looking at.

Structure is preserved when you want it. Flat comment lists lose the argument. Enabling enableCommentHierarchy reconstructs the reply tree from the site's indentation, which matters enormously if you are studying how discussions branch or trying to attribute a reply to its parent.

Extensible without forking. The extendOutputFunction input lets you compute extra fields — a domain extracted from link, a normalised timestamp, a keyword flag — inside the run itself, so your dataset arrives in the shape your downstream job expects.

---

📥 Input

{
  "startUrls": [
    { "url": "https://news.ycombinator.com/front" },
    { "url": "https://news.ycombinator.com/show" },
    { "url": "https://news.ycombinator.com/jobs" },
    { "url": "https://news.ycombinator.com" },
    { "url": "https://news.ycombinator.com/item?id=26566373" }
  ],
  "maxItems": 50,
  "endPage": 1,
  "enableCommentHierarchy": false,
  "extendOutputFunction": "($) => { return {} }"
}

🔧 Hacker News Scraper Input Fields

Field	Type	Required	Default	Description
startUrls	array	Yes	Front page, /front, /show, /jobs and a sample item URL	List of Hacker News URLs to scrape.
maxItems	integer	No	50	Maximum number of items to scrape.
endPage	integer	No	1	Maximum page number to scrape.
enableCommentHierarchy	boolean	No	false	If true, comments will be nested based on indentation.
extendOutputFunction	string	No	($) => { return {} }	A custom JavaScript function to extend the output.

💡 Input Examples

Scrape the current front page only, capped at 30 stories:

{
  "startUrls": [{ "url": "https://news.ycombinator.com" }],
  "maxItems": 30,
  "endPage": 1
}

Pull a single discussion thread with a properly nested comment tree:

{
  "startUrls": [{ "url": "https://news.ycombinator.com/item?id=26566373" }],
  "enableCommentHierarchy": true,
  "maxItems": 200
}

Track startup hiring by walking several pages of the jobs and Show HN sections:

{
  "startUrls": [
    { "url": "https://news.ycombinator.com/jobs" },
    { "url": "https://news.ycombinator.com/show" }
  ],
  "endPage": 5,
  "maxItems": 300
}

---

📤 Output

{
  "id": "49129990",
  "title": "How to Exist",
  "url": "https://news.ycombinator.com/item?id=49129990",
  "link": "https://www.raptitude.com/2026/07/how-to-exist/",
  "points": 385,
  "postedUserName": "walterbell",
  "postedUserLink": "https://news.ycombinator.com/user?id=walterbell",
  "numberOfComments": 234,
  "comments": [
    {
      "scrapedType": "comment",
      "userName": "augment_me",
      "userLink": "https://news.ycombinator.com/user?id=augment_me"
    }
  ],
  "scrapedType": "post",
  "age": "1 day ago"
}

🧾 Hacker News Output Fields

Field	Type	Description
id	string | null	Unique identifier of the item.
title	string | null	Title of the item.
url	string | null	Canonical URL of the scraped item on Hacker News.
link	string | null	Outbound link the submission points to.
points	integer | null	Points (upvote score) of the item.
postedUserName	string | null	Username of the submitter.
postedUserLink	string | null	Profile URL of the submitter.
numberOfComments	integer | null	Number of comments on the item.
comments	array | null	Comments attached to the item.
scrapedType	string | null	Record type, e.g. post or comment.
age	string | null	Relative age of the item as displayed on the site.

Records inside the comments array follow the same field vocabulary, carrying their own scrapedType of comment along with the commenting user's name and profile link.

---

💻 How to Use the Hacker News Scraper (Step by Step)

Step 1: Choose which Hacker News sections to scrape

Open the Input tab and edit startUrls. The default list covers the main page, /front (top stories by date), /show (Show HN launches), /jobs (startup hiring posts) and a sample item thread. Remove what you do not need — each extra section adds requests and dilutes your maxItems budget across sources you may not care about.

Step 2: Add specific threads by item ID

Any individual discussion can be scraped directly by adding its permalink, in the form https://news.ycombinator.com/item?id=NNNNNNNN. This is the right approach when you are monitoring one launch, one outage postmortem or one long-running debate rather than sampling the site broadly.

Step 3: Set the item cap and pagination depth

maxItems limits total output across all sources, and endPage controls how far into a listing's pagination the Hacker News scraper walks. Start conservative — the default of 50 items and one page — verify the output shape is what you expect, then raise both for a full collection run.

Step 4: Decide whether you need nested comments

Leave enableCommentHierarchy off if you plan to run text analysis over comments as a flat corpus; the records are simpler and easier to feed into most NLP pipelines. Turn it on when reply structure matters — for example, when you need to know which comment a rebuttal was aimed at, or to measure how deep threads run.

Step 5: Extend the output if you need derived fields

extendOutputFunction takes a JavaScript function and merges whatever object it returns into each record. The default is a no-op that returns an empty object. Use it to add fields your pipeline expects — a normalised domain from link, a boolean flag for a keyword match in title, or a bucketed score band derived from points.

Step 6: Run the scraper and monitor the log

Start the run and watch the log as it works through the URL list. It reports the pages it fetches and the items it emits, so you can catch a bad URL or an unexpectedly empty section early. If a listing returns nothing, check that the URL still exists — Hacker News section paths are stable, but item IDs disappear when posts are removed.

Step 7: Export and analyse the Hacker News data

Once the run finishes, open the Dataset tab. Export to JSON if you want the nested comments arrays intact, or CSV if you are only interested in the post-level columns like title, points, numberOfComments and age. From there the data is ready for a spreadsheet, a notebook or a scheduled load into your warehouse.

---

🔌 API Access & Integrations

Run the Hacker News scraper and receive dataset items in a single synchronous call:

curl -X POST "https://api.apify.com/v2/acts/scrapers-hub~hacker-news-data-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{ "url": "https://news.ycombinator.com" }],
    "maxItems": 30,
    "endPage": 1
  }'

The same job in Python with the official Apify client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

run = client.actor("scrapers-hub/hacker-news-data-scraper").call(
    run_input={
        "startUrls": [{"url": "https://news.ycombinator.com/show"}],
        "maxItems": 100,
        "endPage": 3,
        "enableCommentHierarchy": True,
    }
)

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["points"], item["title"], item["link"])

Results live in a standard Apify dataset, so you can route them into Zapier, Make, Google Sheets or Slack, or trigger a webhook when a run finishes to push fresh Hacker News data into your own system.

---

💡 Best Use Cases for Hacker News Data

📈 Technology trend monitoring

Collect the front page daily and track how often specific tools, languages or companies appear in title, weighted by points. Because age gives you recency, you can separate a story that is genuinely climbing from one that peaked yesterday and is coasting on accumulated score.

🚀 Product launch intelligence

Show HN is where technical founders put early products in front of a brutally honest audience. Scraping /show and reading title, link and the comments array gives you a running feed of new launches plus the practitioner critique that follows — often the most useful competitive research available for free.

🧑‍💻 Startup hiring and talent signals

The /jobs section is a live indicator of which startups are expanding and for which roles. Pair title with postedUserName and age over several weeks and you get a rough hiring-velocity signal per company, long before it shows up in formal job-market data.

💬 Developer sentiment analysis

Set enableCommentHierarchy to true and run sentiment or topic modelling over the comments array to see how engineers actually feel about a framework, a licensing change or an acquisition. The numberOfComments field tells you where the heat is; the comment text tells you why.

🔗 Content and link discovery

The link field points at the original article, repository or paper. Harvesting those outbound URLs across the front page over time builds a high-quality reading list, and it is a strong seed set for anyone building a technical content recommender.

🏆 Community influence mapping

postedUserName and postedUserLink let you aggregate submissions by user. Ranking submitters by total points or by median score per post surfaces the accounts that consistently find things the community cares about — useful for sourcing, PR outreach and understanding how attention flows on the site.

📚 Dataset building for research and models

Hacker News threads are a large, well-structured corpus of technical discussion. Combining post metadata with nested comments gives researchers and model builders a labelled-by-score dataset that is far cleaner than a generic web crawl.

---

⚙️ Tips for Better Hacker News Scraping Results

• Trim startUrls to the sections you actually analyse. A shorter list means your maxItems budget goes further on the pages that matter instead of being spread thinly.
• Use endPage deliberately. Listing pages get progressively older as you paginate; for trend work, one or two pages of the front page each run is usually more useful than ten pages scraped once.
• Scrape on a schedule rather than in one huge run. Hacker News rankings change hourly. Frequent small runs capture the trajectory of a story; a single large run only captures a snapshot.
• Turn off comment hierarchy for bulk text analysis. Flat comment records are simpler to tokenise and load, and hierarchy adds nothing if you are aggregating sentiment across a whole thread.
• Keep id as your deduplication key. The same story can appear on both the main page and /front, so deduplicate on id when merging results from multiple sections.
• Test extendOutputFunction with a tiny maxItems first. A syntax error in a custom function is much cheaper to discover on a five-item run than on a five-hundred-item one.

---

🛠️ Troubleshooting

The run finished but the dataset is empty. Check your startUrls first. An item?id= URL for a deleted or flagged post will return nothing, and a mistyped section path will too. Try the plain front page URL as a control — if that produces items, the problem is with the specific URLs, not the scraper.

Comments are missing from my post records. Listing pages show story metadata but not full discussions. To get the comment body content for a thread, add that thread's item?id= URL to startUrls directly, and make sure maxItems is high enough to accommodate a busy discussion.

Why is points null on some items? Job postings on /jobs do not carry a score, and some record types simply have no points value. A null here is accurate output, not a scraping failure — filter on scrapedType if you only want scored posts.

My extendOutputFunction is not adding anything. The function must return an object; the default ($) => { return {} } returns an empty one, which merges nothing. Confirm your function returns keys and values, and check the run log for evaluation errors.

The scraper is returning fewer items than maxItems. maxItems is a ceiling, not a target. If endPage is 1 and the section only lists thirty entries, you get thirty. Raise endPage to walk deeper into pagination when you need more volume.

---

❓ Frequently Asked Questions About Hacker News Scraping

What is a Hacker News scraper used for? It converts the posts, scores and discussions on news.ycombinator.com into structured records you can analyse — tracking technology trends, monitoring product launches, studying developer sentiment, or building a research dataset.

Can I scrape Hacker News comments as well as posts? Yes. Comments come back in the comments array attached to each post, and each comment record carries its own scrapedType, username and profile link.

How do I scrape a specific Hacker News thread? Add its permalink to startUrls in the form https://news.ycombinator.com/item?id=NNNNNNNN. That scrapes the single discussion rather than a listing page.

Does the Hacker News scraper support Show HN and the jobs board? Yes. https://news.ycombinator.com/show and https://news.ycombinator.com/jobs are both included in the default startUrls, and either can be used on its own.

What does enableCommentHierarchy change in the output? With it off, comments are returned as a flat list. With it on, replies are nested according to the indentation used on the site, which preserves the reply structure of the thread.

How many items can I scrape in one run? As many as maxItems allows, bounded by how far endPage lets the scraper paginate. Both are yours to set; the defaults are 50 items and one page.

Can I get the outbound article URL, not just the Hacker News permalink? Yes — url is the canonical Hacker News item link, and link is the destination the submission points to.

Does this Hacker News scraper need a browser or headless Chrome? No. It works with direct HTTP requests and HTML parsing, which makes runs faster and lighter than browser-based alternatives.

Do I need to configure proxies? No. Proxy rotation is handled automatically inside the actor; there is no proxy setting exposed in the input.

How do I add custom fields to the scraped Hacker News data? Use extendOutputFunction. It takes a JavaScript function and merges the object it returns into every output record.

Can I schedule the Hacker News scraper to run automatically? Yes. Use Apify's scheduler to run it hourly or daily, which is the right pattern for trend monitoring since front-page rankings change constantly.

How is age formatted in the output? It is the relative age string shown on the site itself, such as "1 day ago", captured at scrape time.

How do I avoid duplicate stories across sections? Deduplicate on the id field. The same submission legitimately appears on more than one listing page, so identical IDs are expected when you scrape several sections at once.

Can I export Hacker News data to CSV or Google Sheets? Yes. Export from the Dataset tab in JSON, CSV or Excel, or connect the dataset to Google Sheets, Zapier, Make or a webhook.

Is scraping Hacker News data legal? The actor collects only publicly visible pages. You remain responsible for how you use the data, including compliance with the site's terms of service and any applicable privacy law where usernames or comment content identify individuals.

---

🆘 Support & Feedback

If the Hacker News scraper misbehaves on a particular URL or section, open a report on the actor's Issues tab. Including the exact startUrls you used makes the problem reproducible and speeds up the fix considerably.

For custom work — extra fields, a tailored output shape, or an integration into an existing data pipeline — email scraperhubapi@gmail.com with what you have in mind.

If this scraper is useful to you, a review on the actor page is genuinely appreciated and helps others find it.

---

⚖️ Disclaimer

This Hacker News scraper collects only publicly accessible content from news.ycombinator.com. It does not log in, access private data, or circumvent any access control.

Responsibility for how the extracted data is used rests with you. Usernames, profile links and comment text can constitute personal data under GDPR, the UK GDPR, CCPA and similar regimes — if you store or process them, ensure you have a lawful basis, minimise what you retain, and honour deletion requests. Do not use scraped commentary to profile, target or harass individuals.

Your use must also comply with the Hacker News terms of service and with Apify's platform terms. Scrape at a reasonable rate and treat the source with the same courtesy you would want applied to your own site.

For data removal requests relating to content collected by this actor, contact scraperhubapi@gmail.com.