Hacker News Data Scraper

What does Hacker News Data Scraper do?

Hacker News Data Scraper collects structured public data from Hacker News for research, trend monitoring, content analysis, and hiring intelligence. Choose a Hacker News feed such as top stories, new stories, best stories, Ask HN, Show HN, or job stories, then set the maximum number of records to save. Results include story titles, authors, scores, timestamps, links, discussion metadata, and text when it is available.

The Actor is useful when you need a repeatable Hacker News data pipeline instead of manually copying posts from the website. Run it once for a research dataset, or schedule it to track changes in the technology community over time.

Why use Hacker News Data Scraper?

• Technology trend research - Collect popular and newly submitted stories to identify topics gaining attention among developers, founders, and technology professionals.
• Community engagement analysis - Compare scores, comment counts, authors, and posting times across different Hacker News categories.
• Startup and hiring intelligence - Collect job stories to study roles, companies, locations, and hiring activity shared by the Hacker News community.
• Ask HN and Show HN datasets - Build focused datasets for product research, founder discussions, launch analysis, and community feedback.
• Automation-ready results - Export the dataset as JSON, CSV, Excel, XML, or another format supported by Apify.
• Repeatable monitoring - Schedule runs, connect webhooks, or access completed datasets through the Apify API.

What data can you extract from Hacker News?

The Actor saves one dataset item for each available Hacker News item. Fields that are not published for a particular story type are returned as null or may be absent from the final record.

Field	Type	Description
id	Integer	Unique Hacker News item ID.
type	String	Item type, commonly story, job, poll, or another Hacker News item type.
title	String	Story or job title when available.
by	String	Hacker News username of the submitter or author when available.
score	Integer	Current score for the item.
descendants	Integer	Number of comments in the discussion thread when available.
time	Integer	Original Hacker News timestamp in Unix seconds.
timestamp	String	The same item time converted to an ISO 8601 timestamp.
url	String	External URL submitted with the item, when available.
text	String	Item text in the HTML format supplied by Hacker News, when available.
text_clean	String	Plain-text version of the item text, when available.
hn_url	String	Direct Hacker News discussion URL for the item.
kids	Array	IDs of direct comments associated with the item.
deleted	Boolean	Whether Hacker News marks the item as deleted.
dead	Boolean	Whether Hacker News marks the item as dead.
parent	Integer	Parent item ID for an item nested under another item, when available.
poll	Integer	Associated poll ID when the item belongs to a poll.
parts	Array	Related poll option IDs when available.

The Actor returns story metadata and comment references such as kids and descendants. It does not recursively save each comment as a separate dataset item.

How to scrape Hacker News data

1. Open Hacker News Data Scraper in Apify Console.
2. Choose a storyType from the available Hacker News feeds.
3. Set results_wanted to the maximum number of items to collect.
4. Start the run and review the dataset preview.
5. Download the results or connect the dataset to a schedule, webhook, API, spreadsheet, or analytics workflow.

The simplest run uses the default topstories feed and collects 20 items. For a focused dataset, change the feed to askstories, showstories, or jobstories before starting the run.

Input Parameters

Parameter	Type	Required	Default	Description
storyType	String	No	topstories	Hacker News feed to collect: topstories, newstories, beststories, askstories, showstories, or jobstories.
results_wanted	Integer	No	20	Maximum number of feed items to process. The Actor accepts values from 1 to 500.
proxyConfiguration	Object	No	Not set	Optional Apify Proxy configuration. Direct collection is normally sufficient for the public Hacker News data source.

Output Data

Field	Type	Description
id	Integer	Hacker News item identifier.
type	String	Hacker News item category.
by	String or null	Author username when available.
time	Integer	Unix timestamp from the source item.
timestamp	String	ISO 8601 representation of time.
title	String or null	Item title when available.
url	String or null	External submission URL when available.
text	String or null	Source HTML text when available.
text_clean	String or null	Clean text version of the item body.
score	Integer	Item score, or 0 when no score is available.
descendants	Integer	Discussion comment count, or 0 when unavailable.
kids	Array	Direct comment IDs, or an empty array when no comments are listed.
hn_url	String	Hacker News URL for the item discussion.
deleted	Boolean	Deleted status reported by Hacker News.
dead	Boolean	Dead status reported by Hacker News.
parent	Integer or null	Parent item ID when applicable.
poll	Integer or null	Poll ID when applicable.
parts	Array or null	Poll option IDs when applicable.

Usage Examples

Collect top stories

Collect up to 50 of the most prominent stories available in the top stories feed.

{
  "storyType": "topstories",
  "results_wanted": 50
}

Monitor new and recent submissions

Collect the latest 100 items from the new stories feed for a technology news or trend-monitoring dataset.

{
  "storyType": "newstories",
  "results_wanted": 100
}

Collect Ask HN discussions

Build a focused dataset of community questions and discussions from Ask HN.

{
  "storyType": "askstories",
  "results_wanted": 75
}

Track Hacker News job stories

Collect up to 200 job postings shared by the Hacker News community for hiring research and startup market analysis.

{
  "storyType": "jobstories",
  "results_wanted": 200
}

Sample Output

The following is an example of one dataset item. Actual values depend on the selected feed and the information published for each item.

{
  "id": 45006801,
  "type": "story",
  "by": "developer123",
  "time": 1771939200,
  "timestamp": "2026-02-24T00:00:00.000Z",
  "title": "Show HN: A practical tool for analyzing developer workflows",
  "url": "https://github.com/example/developer-workflow-tool",
  "text": "A project description published with the Hacker News submission.",
  "text_clean": "A project description published with the Hacker News submission.",
  "score": 245,
  "descendants": 67,
  "kids": [45006802, 45006803],
  "hn_url": "https://news.ycombinator.com/item?id=45006801",
  "deleted": false,
  "dead": false,
  "parent": null,
  "poll": null,
  "parts": null
}

Tips for best results

• Start with a small limit - Use 20 to 50 results to confirm the selected feed and output shape before running a larger collection.
• Match the feed to the question - Use topstories for popularity, newstories for recency, beststories for highly ranked content, askstories for questions, showstories for launches, and jobstories for hiring activity.
• Use text_clean for analysis - This field is easier to send into spreadsheets, search indexes, summaries, and natural-language processing workflows than the HTML text field.
• Use hn_url for attribution - Keep the discussion URL when building reports so users can return to the original conversation.
• Interpret missing fields correctly - A missing external URL or body is normal for some item types. Job stories and Ask HN posts can have different field combinations.
• Schedule repeat runs carefully - A scheduled run provides a fresh snapshot of the selected feed. Store the item ID and timestamp if you need to compare snapshots or identify new records.

Integrations and export formats

• Google Sheets - Review scores, titles, authors, and timestamps in a shared research sheet.
• Airtable - Create a searchable database of stories, discussions, or job posts.
• Slack or webhooks - Send completed run notifications or route new datasets to downstream services.
• Make or Zapier - Trigger summaries, alerts, or content workflows after a run completes.
• Apify API - Read dataset items programmatically and connect Hacker News records to your own applications or data warehouse.

Apify datasets can be downloaded as JSON, CSV, Excel, XML, and other supported formats.

Frequently Asked Questions

Which Hacker News feeds are supported?

The Actor supports topstories, newstories, beststories, askstories, showstories, and jobstories.

Does the Actor collect comments?

The Actor returns comment counts and direct comment IDs through descendants and kids. It does not fetch every comment as a separate dataset record.

Can I collect job postings from Hacker News?

Yes. Set storyType to jobstories to collect job items and their available titles, text, authors, timestamps, links, and discussion metadata.

Can I collect Ask HN or Show HN posts?

Yes. Use askstories for questions and community discussions, or showstories for product launches and project announcements.

How many items can I collect in one run?

Set results_wanted from 1 to 500. The number of available records can be lower when the selected Hacker News feed has fewer usable items or when items are deleted or marked dead.

Can I export Hacker News data to CSV or Excel?

Yes. After the run finishes, download the dataset in CSV, Excel, JSON, XML, or another format offered by Apify.

Can I run this Actor on a schedule?

Yes. Create an Apify schedule for hourly, daily, weekly, or custom recurring collection. Use item IDs and timestamps to compare results between runs.

What if some fields are empty?

Empty fields usually mean that Hacker News did not publish that value for the item. This is expected because stories, jobs, Ask HN posts, Show HN posts, and polls use different fields.

Is collecting Hacker News data legal?

The Actor is intended for responsible collection of publicly available Hacker News data. You are responsible for following Hacker News rules, applicable laws, and any requirements related to attribution, storage, privacy, and downstream use.

Related Actors

• RSS / XML Scraper - Collect RSS and Atom feed records for broader news monitoring and content aggregation workflows.
• Talent Job Scraper - Collect structured job listings for hiring research that complements Hacker News job stories.
• JobStreet Scraper - Gather additional job market data when comparing Hacker News hiring activity with external listings.

Support

For issues, output questions, or feature requests, use the Issues or Discussions area on the Actor page in Apify Console.

Legal Notice

This Actor collects publicly available Hacker News data for legitimate research, monitoring, and automation workflows. Users are responsible for complying with Hacker News policies, applicable laws, and any terms governing the use, storage, redistribution, or publication of collected data.