Hacker News Historical Archive Scraper — Old Stories & Comments by Date

Download the entire Hacker News archive between any two dates. Point this scraper at a date range — a week, a year, or every day since HN launched in 2006 — and it bulk-exports the full chronological corpus of stories, comments, Ask HN, Show HN, polls and jobs as clean, flat rows. It runs on the keyless Algolia Hacker News Search API and uses a time-window pagination engine to slide backward through time and blow past Algolia's 1,000-results-per-query ceiling, so a single run can return hundreds of thousands of rows. No API key, no login, no cookies, no browser.

🏆 This is a bulk historical firehose — not a keyword search

Most HN scrapers answer "give me the top results for keyword X." This one answers "give me everything posted to Hacker News between date A and date B." Leave the keyword empty and you get the whole archive for the window — every story and every comment, in chronological order, ready for AI training corpora, trend research and dataset building. Export to JSON / CSV / Excel.

---

✨ What does this Actor do?

This Actor is a date-bounded chronological exporter for the complete Hacker News corpus. You give it a startDate, an endDate and a contentType (stories, comments, Ask/Show HN, polls, jobs, or all combined) and it pulls every matching item in that window from the public Algolia HN Search API (search_by_date), newest to oldest.

The engine's core trick is time-window pagination. Algolia caps any single query at ~1,000 results (page × hitsPerPage ≤ 1000), so a naive page++ loop silently stops at 1,000 rows and can never export a real range. Instead, this Actor requests a page of 1,000, records the oldest created_at_i in the batch, then re-queries with that timestamp as the new upper bound — sliding the window strictly backward through time and repeating until the range is exhausted. That's how it reaches hundreds of thousands of rows from one date range. Items are de-duplicated by objectID. It's keyless, fast, needs no browser, and covers HN from its 2006 launch to today.

👥 Who is it for?

• AI / ML & NLP teams building domain-specific training corpora from years of high-signal tech discussion and comment threads.
• Data scientists & quantitative researchers who need a complete, date-bounded HN dataset for trend, topic and sentiment modelling.
• Academic researchers & social scientists studying the tech community, startup culture or online discourse longitudinally.
• Data journalists & analysts reconstructing how a topic, company or technology was discussed across a precise historical window.
• Brand, PR & market-intelligence teams monitoring mentions of a product, company or person across the full HN back-catalogue over time.

🎯 Use cases

• Build an AI training corpus — export every comment and story in a multi-year window to fine-tune or evaluate language models on real technical dialogue.
• Tech-trend & sentiment analysis — pull all stories mentioning a technology across several years and chart how attention and engagement shifted.
• Academic study of a community — collect a complete date-bounded slice of HN to study discourse, moderation or topic diffusion.
• Brand & reputation monitoring over time — retrieve every historical mention of your company or product to measure share-of-voice and sentiment.
• Dataset building & retrospective analysis — snapshot an entire era of Hacker News (e.g. the 2021 crypto run-up, the 2023 AI boom) as a reusable dataset.

💡 Why use this Actor?

• 🔑 100% keyless — no API key, no login, no cookies, no client secret. It reads the public Algolia HN endpoint directly.
• 🌊 True bulk firehose — leave the keyword empty and export every item in your date window, not just a capped top-N.
• ♾️ Beats the 1,000-result cap — the time-window engine slides backward through time to return hundreds of thousands of rows across a wide range.
• 🧾 14 clean fields per row — id, type, title, url, author, points, comment count, HTML-stripped text, timestamps, story/parent IDs, HN permalink and tags.
• 🎛️ Precise filters — content type, date window, optional keyword, author allow-list, minimum points and minimum comments.
• 📤 Export anywhere — download as JSON, CSV, Excel, HTML, XML or JSONL, or pull via the Apify API and webhooks. Pay-per-result pricing.

📊 What data can you extract?

Every story or comment comes back as one flat row. Story rows carry title, url, points and numComments; comment rows carry the text (HTML-stripped) plus storyId and parentId linking them back into their thread.

Field	Type	Description
objectID	string	Hacker News item ID (unique per story/comment).
type	string	story, comment, poll, job, show_hn or ask_hn.
title	string	Story / poll / Ask / Show title (empty for comments).
url	string	Outbound link of a story (empty for text posts and comments).
author	string	Hacker News username of the poster.
points	number	Upvote score (stories/polls; null for comments/jobs).
numComments	number	Number of comments on the story/poll.
text	string	HTML-stripped body — comment text or story self-text.
createdAt	string	ISO 8601 UTC timestamp of posting.
createdAtUnix	number	Creation time as UNIX epoch seconds.
storyId	string	ID of the parent story (comments and Ask/Show sub-items).
parentId	string	ID of the direct parent item (comments).
hnUrl	string	Permalink on news.ycombinator.com.
tags	string	Algolia item tags, comma-joined (e.g. story,author_pg,story_38836483).

Example row (story)

{
  "objectID": "38836483",
  "type": "story",
  "title": "The Rise of Unix. The Seeds of Its Fall",
  "url": "https://www.youtube.com/watch?v=HADp3emVABg",
  "author": "tux1968",
  "points": 6,
  "numComments": 0,
  "text": "",
  "createdAt": "2024-01-01T23:58:35Z",
  "createdAtUnix": 1704153515,
  "storyId": "38836483",
  "parentId": "",
  "hnUrl": "https://news.ycombinator.com/item?id=38836483",
  "tags": "story,author_tux1968,story_38836483"
}

🚀 How to use

Option A — Export the whole archive for a date window (firehose)

The default, most powerful mode. Set a content type and a date range and leave the keyword empty to pull everything.

1. Set Content Type to Stories (or Comments, All, etc.).
2. Set Start Date and End Date (YYYY-MM-DD, UTC, inclusive).
3. Raise Max Items to the corpus size you want (e.g. 100000).
4. Click Start, then export from the Output tab.

{
  "contentType": "story",
  "startDate": "2023-01-01",
  "endDate": "2023-12-31",
  "query": "",
  "maxItems": 100000,
  "proxyConfiguration": { "useApifyProxy": true }
}

Option B — A filtered historical slice (topic, author or high-engagement)

Add a keyword, an author allow-list, or point/comment thresholds to carve a targeted slice out of the window.

1. Keep your date range.
2. Add a Keyword filter (e.g. openai), and/or Authors (e.g. pg, dang), and/or Minimum Points.
3. Run and export.

{
  "contentType": "story",
  "startDate": "2022-01-01",
  "endDate": "2024-12-31",
  "query": "openai",
  "minPoints": 50,
  "maxItems": 5000
}

⚙️ Input parameters

Parameter	Type	Default	Description
contentType	select	story	story, comment, show_hn, ask_hn, poll, job, front_page, or all (stories + comments + polls + jobs combined).
startDate	string	2024-01-01	Oldest date to include (YYYY-MM-DD, UTC, inclusive). HN data starts in 2006. Required.
endDate	string	2024-01-08	Newest date to include (YYYY-MM-DD, UTC, inclusive to end-of-day). Required.
query	string	""	Optional keyword filter (title/text/comment). Leave empty for the full firehose.
authors	array	[]	Optional list of HN usernames to restrict to (no @). Each is queried separately.
minPoints	integer	0	Only items with at least this many points. 0 = no filter.
minComments	integer	0	Only items with at least this many comments. 0 = no filter.
maxItems	integer	5000	Global cap on rows. Raise for a full corpus; 0 = unlimited (bounded by window + timeout).
proxyConfiguration	object	{ "useApifyProxy": true }	Apify Proxy. Datacenter is plenty for this public CDN endpoint.

Full input example

{
  "contentType": "all",
  "startDate": "2021-01-01",
  "endDate": "2021-03-31",
  "query": "",
  "authors": [],
  "minPoints": 0,
  "minComments": 0,
  "maxItems": 50000,
  "proxyConfiguration": { "useApifyProxy": true }
}

📤 Output example

A comment row, showing how comments link back into their thread via storyId / parentId:

{
  "objectID": "38836489",
  "type": "comment",
  "title": "",
  "url": "",
  "author": "elijahbenizzy",
  "points": null,
  "numComments": null,
  "text": "Oh don't get me started on outdoor cats...",
  "createdAt": "2024-01-01T23:59:50Z",
  "createdAtUnix": 1704153590,
  "storyId": "38835845",
  "parentId": "38836469",
  "hnUrl": "https://news.ycombinator.com/item?id=38836489",
  "tags": "comment,author_elijahbenizzy,story_38835845"
}

🧠 Tips for best results

• Leave query empty for a true firehose. Any keyword turns the run into a filtered slice instead of a full-archive export.
• Size maxItems to your window. A single busy day is ~600 stories and ~6,000+ comments; a full year of comments is well into the millions. Set maxItems: 0 (unlimited) only when you also allow enough run timeout.
• Split enormous ranges into chunks (e.g. one run per year) for easier restarts, smaller datasets and cheaper re-runs.
• Use contentType: "all" to interleave stories and comments in one chronological stream; use comment alone to build a pure dialogue corpus.
• minPoints filters to high-signal content — great for "only stories that actually got traction". Remember comments and jobs have no points, so a non-zero value effectively drops them.
• authors builds per-user corpora — list several usernames to pull each author's full history inside the window.
• Comments carry storyId — group rows by it to reconstruct threads, or join against a separate stories export.
• Timestamps are UTC. Convert to your local zone downstream if needed; createdAtUnix is handy for exact math and sorting.
• Dates are inclusive. endDate covers the whole final day (to 23:59:59 UTC).
• Re-run on a schedule to keep a rolling, up-to-date archive of a topic or the whole firehose.

🔌 Integrations

Wire this Actor into your stack with Apify's built-in connectors and API:

• Google Sheets — push exported rows straight into a sheet for analysis or sharing.
• Slack — get notified when a scheduled archive run finishes.
• Zapier / Make / n8n — trigger downstream workflows on new results.
• Webhooks — POST run + dataset events to your own service.
• Schedule — run daily/weekly to maintain a fresh, growing HN corpus.
• Apify API & clients — pull datasets programmatically in JSON, CSV, Excel, XML or JSONL.

🧑‍💻 API usage

Run the Actor and fetch results programmatically. Get your token from Settings → Integrations in the Apify Console.

cURL

curl -X POST "https://api.apify.com/v2/acts/logiover~hacker-news-historical-archive-scraper/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"contentType":"story","startDate":"2024-01-01","endDate":"2024-01-31","maxItems":10000}'

Node.js (apify-client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });

const run = await client.actor('logiover/hacker-news-historical-archive-scraper').call({
  contentType: 'comment',
  startDate: '2023-01-01',
  endDate: '2023-06-30',
  maxItems: 50000,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.length, 'HN items');

Python (apify-client)

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

run = client.actor("logiover/hacker-news-historical-archive-scraper").call(run_input={
    "contentType": "story",
    "startDate": "2022-01-01",
    "endDate": "2022-12-31",
    "query": "rust",
    "maxItems": 5000,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["createdAt"], item["title"])

🤖 Use with AI agents (MCP)

This Actor works as a tool for LLM agents through the Apify MCP (Model Context Protocol) server, so an agent can pull historical Hacker News data on demand to answer research questions or build corpora. Point your MCP-capable client at Apify's MCP server and call this Actor by name.

Example prompt: "Using the Hacker News Historical Archive Scraper, export all stories mentioning 'LLM' between 2023-01-01 and 2023-12-31 with at least 100 points, then summarise the ten most-discussed themes."

❓ FAQ

Is this really keyless — no API key or login?

Yes. It reads the public Algolia Hacker News Search API directly. No API key, no HN account, no OAuth, no cookies and no client secret are required.

How is this different from a Hacker News keyword-search scraper?

A keyword-search scraper returns a capped set of the best matches for a search term. This Actor is a bulk, date-bounded firehose: leave the keyword empty and it exports every story and comment in your date window in chronological order — the whole archive slice, not a top-N. Use it when you want completeness over a time range, not relevance ranking for one query.

How far back does the data go?

Hacker News launched in 2006, and Algolia indexes it from the start, so any startDate from 2006-10 onward is reachable, right up to the present.

How much data can one run return? Isn't Algolia capped at 1,000 results?

Algolia caps a single query at ~1,000 results, but this Actor uses time-window pagination — it slides the query window backward through time using the oldest timestamp from each batch — so a run can return hundreds of thousands of rows across a wide date range. The 1,000 cap does not limit your export.

Why did my run return zero items?

Usually the date window brackets no activity, or the filters are too strict. Check that startDate/endDate cover real dates (2006→now), and relax query, minPoints or minComments. Comments and jobs have no points, so a non-zero minPoints will exclude them.

Why are some fields empty or null?

By design. Comments have no title, url, points or numComments; text posts have no outbound url; jobs have no points. Empty string / null means the field doesn't apply to that item type.

Can I export to CSV, Excel or JSON?

Yes. Every row is flat, so you can download the dataset as CSV, JSON, Excel, HTML, XML or JSONL from the Apify Console, or pull it via the API. It imports cleanly into spreadsheets, warehouses and notebooks.

How do I pull only comments (to build a dialogue corpus)?

Set Content Type to Comments. Each row is one comment with its HTML-stripped text, author, storyId and parentId, so you can reconstruct threads or feed the text straight into an NLP pipeline.

How do I export a specific user's Hacker News history?

Add their username(s) to Authors and set a wide date range. Each user is queried across the whole window for the chosen content type.

How fast is it, and how much does it cost?

It's a lightweight JSON API with no browser, so it's fast and cheap. Cost scales with the number of rows on the pay-per-result model — see the Pricing tab. Split very large ranges into per-year runs to control dataset size and cost.

Is scraping Hacker News legal?

The Actor reads publicly available Hacker News content through a public API. You are responsible for using the data in line with Hacker News' / Y Combinator's terms and any applicable laws such as GDPR. See Is it legal? below.

How often is the data updated?

The Algolia index tracks Hacker News in near real time, so recent items appear within minutes of posting. Schedule the Actor to keep a rolling archive current.

⚖️ Is it legal?

This Actor extracts only publicly available Hacker News data via a public API and is intended for legitimate research, analytics, archival and monitoring use. It requires no login and accesses no private or gated content. You are responsible for complying with Hacker News' / Y Combinator's terms of service, the API provider's terms, GDPR and any other applicable local laws and privacy regulations when collecting and using the data.

🔗 Related scrapers by logiover

• Hacker News Search Scraper — targeted keyword search across HN (stories, comments, Ask/Show), when you want ranked results for a query rather than a full-archive firehose.
• Reddit Historical Archive Scraper — the same date-bounded archive approach for Reddit posts and comments.
• Substack Newsletter Scraper — export newsletter posts and archives for tech-writing corpora.

👉 Browse all logiover scrapers on Apify Store — 180+ actors across news, developer tools, social media, real estate, jobs, crypto and B2B data.

---

📝 Changelog

2026-07-22

• 🚀 Initial release. Bulk historical Hacker News export between any two dates via the keyless Algolia HN Search API. Time-window pagination beats the 1,000-results-per-query cap and returns hundreds of thousands of rows. Content types: stories, comments, Ask/Show HN, polls, jobs, front-page and all-combined. Filters for keyword, authors, minimum points and minimum comments. 14 flat fields per row, HTML-stripped comment/story text, de-duplication by item ID, datacenter proxy with retry + IP rotation.