Hacker News Search Scraper

Hacker News Search Scraper — Story Discovery, Comments, and Engagement Metrics

Overview

Collect Hacker News stories and comments matching your search terms. Choose relevance-based or date-based ordering, refine results with tags and numeric filters, and export structured datasets in JSON, CSV, Excel, or other supported formats.

Each record can include the item title, author, points, comment count, timestamps, tags, and related story information when available.

Features

• Keyword search: Find Hacker News content matching your query.
• Relevance sorting: Discover results ranked by search relevance.
• Date sorting: Retrieve newer results first.
• Content filtering: Narrow results by tags such as stories or comments.
• Engagement metrics: Collect points and comment counts when available.
• Configurable collection: Set a maximum number of records.
• Structured exports: Save search results for analysis and reporting.
• Scheduled collection: Refresh search datasets periodically using Apify scheduling.

Collection Modes

1. Search by Relevance

Find stories and comments ranked according to their relevance to the search query.

2. Search by Date

Retrieve matching stories and comments ordered by date, with newer items appearing first.

Input Parameters

Parameter	Type	Default	Description
mode	String	search	Selects relevance-based search or date-based search.
query	String	—	Keyword or phrase to search for.
tags	String	—	Optional content filter, such as story or comment.
numericFilters	String	—	Optional numeric conditions, such as points>100, when supported by the actor input.
maxItems	Integer	100	Maximum number of records to collect.
requestDelaySecs	Number	1	Configured delay between requests, where applicable.
requestTimeoutSecs	Integer	30	Maximum waiting time for a request, in seconds.

Example Input

{
  "mode": "search",
  "query": "python",
  "tags": "story",
  "maxItems": 10,
  "requestDelaySecs": 1,
  "requestTimeoutSecs": 30
}

This configuration searches for Python-related stories and collects up to 10 matching records.

What Data Does This Actor Extract?

The actor collects searchable Hacker News items, including story and comment details, engagement metrics, tags, and timestamps where available.

Extracted Data Fields

Field	Description
objectID	Unique identifier for the search result.
storyTitle	Title associated with the story, when available.
author	Username of the item's author.
points	Points associated with the item.
numComments	Number of comments associated with the story, when available.
tagsList	List of tags associated with the result.
children	Related comment identifiers, when available.
createdAt	Creation date and time.
createdAtI	Creation timestamp in Unix format.
storyId	Identifier of the associated story, when available.
updatedAt	Last update timestamp, when available.
url	Link associated with the story, when available.
scrapedAt	Date and time when the record was collected.

Some fields may be absent depending on whether a result represents a story, comment, or another item type. The original title field is represented as storyTitle in the documented dataset structure.

Example Output

{
  "tagsList": [
    "story",
    "author_trotterdylan",
    "story_13319904"
  ],
  "author": "trotterdylan",
  "children": [
    13320144,
    13320174
  ],
  "createdAt": "2017-01-04T17:00:39Z",
  "createdAtI": 1483549239,
  "numComments": 451,
  "objectID": "13319904",
  "points": 1411,
  "storyId": 13319904,
  "storyTitle": "Grumpy: Go running Python",
  "updatedAt": "2026-06-14T03:38:24Z",
  "scrapedAt": "2026-10-26T00:00:00+00:00"
}

This example illustrates the record structure. Actual values and available fields vary by search result.

Use Cases

• Topic research: Find discussions and stories related to programming, startups, AI, and technology.
• Trend monitoring: Track newly published stories matching selected keywords.
• Engagement analysis: Identify stories with high point totals or comment counts.
• Dataset building: Export searchable Hacker News content for offline analysis.
• Community research: Explore topics and discussions across stories and comments.
• Content discovery: Find relevant technical discussions and interesting articles.

Working with Your Dataset

After the run finishes, open your dataset in Apify to inspect the collected records or export them.

• JSON: Use structured results in data-processing applications.
• CSV: Filter and analyze search results in spreadsheet software.
• Excel: Build summaries, charts, and engagement reports.
• Other supported formats: Choose from the dataset export formats available in Apify.

Important Notes

• Search results depend on the query, selected mode, and optional filters.
• The maxItems setting limits the number of records collected.
• Some fields are specific to stories and may not be present for comments.
• Story and comment counts can change over time.
• Search results represent the data available when the actor runs.

FAQ

Who is this actor for?

It is designed for developers, analysts, researchers, and technology enthusiasts who want searchable Hacker News data in a structured dataset.

What is the difference between the search modes?

Relevance-based search prioritizes results according to their relevance to your query. Date-based search orders matching results with newer items first.

Can I search only for stories or comments?

Yes. Use the tags parameter to narrow the results to supported content types, such as story or comment.

Can I filter by points?

You can use supported numeric filters, such as points>100, to focus on items meeting a points threshold.

What export formats are available?

You can export collected records as JSON, CSV, Excel, and other formats supported by Apify datasets.

Can I schedule recurring searches?

Yes. Configure an Apify schedule to refresh your search results at your preferred interval.

Disclaimer

This actor is an independent tool and is not affiliated with, endorsed by, sponsored by, or officially associated with Hacker News, Y Combinator, or Algolia. The collected information is provided for research and informational purposes. Users are responsible for ensuring that their collection and use of the data comply with applicable laws and relevant terms.

Is Web Scraping Legal?

Collecting information that is already public is generally legal. There is no single law that bans all forms of web scraping. The limits are the usual ones: do not break into accounts, do not take private or confidential data, and be careful with personal data and copyrighted works. A site's terms can also restrict how you access it. Facts such as a public name, price, or rating are not the same thing as republishing someone's article or photo.

This actor collects information from public pages. It does not log in and it does not bypass access controls.

For a longer explanation, see Apify's article Is web scraping legal?.