The Better Hacker News Scraper — Fast, Affordable HN Story & Comment Extractor

Scrape Hacker News at scale without the cost. The Better Hacker News Scraper is an affordable, efficient Hacker News scraper / HN data extractor that searches and exports Hacker News stories, comments, Ask HN and Show HN posts, points, authors, and links as structured JSON, CSV, or Excel. It runs entirely over plain HTTP against the public Algolia Hacker News Search API — no login, no API key, and no proxy required — so each run uses fewer compute units and stays cheap even when you pull thousands of records.

Whether you want to monitor Hacker News mentions of your product, build an HN dataset for research, track trending tech news and Show HN launches, or feed a news aggregator or LLM pipeline, this scraper gives you clean, normalized Hacker News data in seconds.

What it extracts

For every matching Hacker News item, the scraper returns:

• id — the Hacker News item id (Algolia objectID)
• title — the story/post title (falls back to the parent story title for comments)
• url — the outbound link (falls back to the HN discussion URL for text posts and comments)
• author — the HN username of the submitter/commenter
• points — the story score / upvote count
• numComments — number of comments on the story
• createdAt — submission time as an ISO 8601 timestamp
• text — clean plain-text body of Ask HN / Show HN posts and comments (HTML tags stripped, entities decoded). Regular link stories have no body text on Hacker News, so this is null for them — that's expected, not missing data (the content lives at url).
• hnUrl — the canonical Hacker News discussion URL (https://news.ycombinator.com/item?id=<id>)

Features

• HTTP-only, no browser — talks directly to the JSON API, so it is fast and light on compute.
• No proxy required — the Algolia HN Search API is public; runs work with zero proxy cost by default.
• Full-text search across all of Hacker News history via the query field.
• Filter by content type — stories, comments, polls, Show HN, Ask HN, or the current front page.
• Newest-first or relevance — flip byDate to switch between search_by_date and ranked search.
• Automatic pagination — fetches up to maxPages pages (100 items per page) and stops early when results run out.
• Normalized, predictable output — consistent field names ready for JSON, CSV, Excel, or an API dataset.
• Robust parsing — the core parser is a pure, unit-tested function with sensible fallbacks for missing fields.

Input

Field	Type	Default	Description
query	string	""	Full-text search term. Leave empty to fetch the latest/front-page items for the selected tag.
tags	string	"story"	Content type to return: story, comment, poll, show_hn, ask_hn, front_page. You can also target a user with author_<username>.
byDate	boolean	false	When true, uses the search_by_date endpoint for newest-first results. When false, results are ranked by relevance/popularity.
maxPages	integer	5	How many result pages to fetch (100 items per page, 1–200). More pages return more results and use slightly more compute.
proxyConfiguration	object	{ "useApifyProxy": false }	Optional. The API is public and works with no proxy — leave off to keep runs cheapest. Enable a datacenter proxy only if you hit rate limits at very high volume.

Example input

{
    "query": "openai",
    "tags": "story",
    "byDate": false,
    "maxPages": 3
}

Output

Each dataset item looks like this:

{
    "id": "38912345",
    "title": "Show HN: A tiny, fast Hacker News scraper",
    "url": "https://example.com/show-hn-scraper",
    "author": "pg",
    "points": 412,
    "numComments": 87,
    "createdAt": "2024-01-15T09:30:00.000Z",
    "text": null,
    "hnUrl": "https://news.ycombinator.com/item?id=38912345"
}

For an Ask HN or comment with no outbound link, url falls back to the hnUrl discussion page and text contains the post/comment body.

Example use cases

• Brand & product monitoring — track every Hacker News mention of your company, tool, or competitor.
• Tech trend research — build datasets of trending HN stories, points, and comment volume over time.
• Show HN / Ask HN analysis — study launch posts, engagement, and community feedback.
• Newsletter & aggregator feeds — pull the latest front-page or newest HN stories into your own product.
• LLM & RAG pipelines — collect Hacker News discussions as clean text for summarization or Q&A.
• Author tracking — follow everything a specific HN user has submitted or commented.
• Academic & market research — export large, structured HN corpora for analysis in Excel, pandas, or BigQuery.

How it works

1. The actor reads your input and builds a request to the public Algolia HN Search API — either /search (ranked) or /search_by_date (newest first), with your query and tags.
2. A lightweight CheerioCrawler fetches each page of JSON over HTTP (no browser is launched).
3. The pure parseHits parser normalizes every hit into the clean output shape above, filling sensible fallbacks for missing titles, URLs, and timestamps.
4. Results are pushed to the dataset, and the next page is enqueued until maxPages is reached or the results are exhausted.

Why it's efficient and affordable

• No browser, no rendering — it never spins up Chromium, so runs finish quickly and consume fewer compute units per run.
• No proxy cost by default — the Algolia HN Search API is public and free, so there is no residential-proxy bill; the default configuration uses no proxy at all.
• Compact JSON payloads — the API returns structured JSON, so there is no HTML to download or parse and less bandwidth per record.
• Early-exit pagination — the crawler stops as soon as results run out, so short queries do not waste requests.

Together these choices keep the cost per 1,000 Hacker News records low, which makes this a practical, low-cost Hacker News scraper for both one-off exports and scheduled monitoring.

FAQ

Do I need a Hacker News or Algolia API key? No. The scraper uses the public Algolia HN Search API, which requires no key, no account, and no login.

Do I need proxies to scrape Hacker News? No. The API is public and the actor works with no proxy, which is a big part of why it is so cheap to run. A datacenter proxy is available as an option only if you hit rate limits at very high volume.

How many results can I get? Each page returns up to 100 items and you can fetch up to 200 pages per run (maxPages), so a single run can return tens of thousands of Hacker News stories or comments.

Can I scrape comments as well as stories? Yes. Set tags to comment for comments, ask_hn or show_hn for those post types, or front_page for the current front page.

What formats can I export? Any format Apify datasets support — JSON, CSV, Excel, HTML, RSS, or via the API — so the data drops straight into spreadsheets, databases, or downstream code.

Legal & ethical note

This actor retrieves publicly available Hacker News data through the official public Algolia HN Search API. Please scrape responsibly: request only the data you need, respect Hacker News' and Algolia's Terms of Service and robots.txt, avoid excessive request rates, and comply with all applicable laws and data-protection regulations (including how you store and use any personal data such as usernames). You are responsible for how you use the data you collect.