Hacker News Historical Archive Scraper — Old Stories & Comments by Date

Export Hacker News records between two UTC boundaries as clean, flat rows. The Actor reads the keyless Algolia Hacker News Search API and uses backward timestamp windows plus a boundary-second recovery query to move beyond the 1,000-results-per-query ceiling. Output is newest to oldest and is bounded by maxItems, runtime, selected filters and the records Algolia currently indexes. No API key, login, cookies or browser.

🏆 This is a bulk historical firehose — not a keyword search

Most HN search tools return one capped relevance page. This Actor walks the selected date window newest-to-oldest. Leave the keyword empty for all matching records exposed by Algolia, subject to your row cap and runtime budget. Export to JSON / CSV / Excel.

---

✨ What does this Actor do?

This Actor is a date-bounded reverse-chronological exporter. You give it a startDate, an endDate and a contentType; it pulls matching records from the public Algolia HN Search API (search_by_date) newest to oldest. maxItems, runtime, filters and upstream index coverage define the actual completeness boundary.

Algolia caps a single result page near 1,000 rows. The Actor records each page's oldest created_at_i, separately recovers matches sharing that boundary second, then continues with an exclusive upper timestamp. Records are de-duplicated by objectID. If an extreme boundary second itself exceeds Algolia's retrievable page, the run emits an explicit warning instead of claiming silent completeness.

👥 Who is it for?

• AI / ML & NLP teams building domain-specific training corpora from years of high-signal tech discussion and comment threads.
• Data scientists & quantitative researchers who need a complete, date-bounded HN dataset for trend, topic and sentiment modelling.
• Academic researchers & social scientists studying the tech community, startup culture or online discourse longitudinally.
• Data journalists & analysts reconstructing how a topic, company or technology was discussed across a precise historical window.
• Brand, PR & market-intelligence teams monitoring mentions of a product, company or person across the full HN back-catalogue over time.

🎯 Use cases

• Build an AI training corpus — export every comment and story in a multi-year window to fine-tune or evaluate language models on real technical dialogue.
• Tech-trend & sentiment analysis — pull all stories mentioning a technology across several years and chart how attention and engagement shifted.
• Academic study of a community — collect a complete date-bounded slice of HN to study discourse, moderation or topic diffusion.
• Brand & reputation monitoring over time — retrieve every historical mention of your company or product to measure share-of-voice and sentiment.
• Dataset building & retrospective analysis — snapshot an entire era of Hacker News (e.g. the 2021 crypto run-up, the 2023 AI boom) as a reusable dataset.

💡 Why use this Actor?

• 🔑 100% keyless — no API key, no login, no cookies, no client secret. It reads the public Algolia HN endpoint directly.
• 🌊 Date-window export — leave the keyword empty to avoid relevance filtering; maxItems, runtime and Algolia's index still bound the result.
• ♾️ Beats the 1,000-result cap — the time-window engine slides backward through time to return hundreds of thousands of rows across a wide range.
• 🧾 14 clean fields per row — id, type, title, url, author, points, comment count, HTML-stripped text, timestamps, story/parent IDs, HN permalink and tags.
• 🎛️ Precise filters — content type, date window, optional keyword, author allow-list, minimum points and minimum comments.
• 📤 Export anywhere — download as JSON, CSV, Excel, HTML, XML or JSONL, or pull via the Apify API and webhooks. Pay-per-result pricing.

📊 What data can you extract?

Every story or comment comes back as one flat row. Story rows carry title, url, points and numComments; comment rows carry the text (HTML-stripped) plus storyId and parentId linking them back into their thread.

Field	Type	Description
objectID	string	Hacker News item ID (unique per story/comment).
type	string	story, comment, poll, job, show_hn or ask_hn.
title	string	Story / poll / Ask / Show title (empty for comments).
url	string	Outbound link of a story (empty for text posts and comments).
author	string	Hacker News username of the poster.
points	number	Upvote score (stories/polls; null for comments/jobs).
numComments	number	Number of comments on the story/poll.
text	string	HTML-stripped body — comment text or story self-text.
createdAt	string	ISO 8601 UTC timestamp of posting.
createdAtUnix	number	Creation time as UNIX epoch seconds.
storyId	string	ID of the parent story (comments and Ask/Show sub-items).
parentId	string	ID of the direct parent item (comments).
hnUrl	string	Permalink on news.ycombinator.com.
tags	string	Algolia item tags, comma-joined (e.g. story,author_pg,story_38836483).

Example row (story)

{
  "objectID": "38836483",
  "type": "story",
  "title": "The Rise of Unix. The Seeds of Its Fall",
  "url": "https://www.youtube.com/watch?v=HADp3emVABg",
  "author": "tux1968",
  "points": 6,
  "numComments": 0,
  "text": "",
  "createdAt": "2024-01-01T23:58:35Z",
  "createdAtUnix": 1704153515,
  "storyId": "38836483",
  "parentId": "",
  "hnUrl": "https://news.ycombinator.com/item?id=38836483",
  "tags": "story,author_tux1968,story_38836483"
}

🚀 How to use

Option A — Export the whole archive for a date window (firehose)

Set a content type and date range and leave the keyword empty to avoid relevance filtering. The result remains bounded by maxItems, timeBudgetSecs, Actor timeout and Algolia's index.

1. Set Content Type to Stories (or Comments, All, etc.).
2. Set Start Date and End Date (YYYY-MM-DD, UTC, inclusive).
3. Raise Max Items to the corpus size you want (e.g. 100000).
4. Click Start, then export from the Output tab.

{
  "contentType": "story",
  "startDate": "2023-01-01",
  "endDate": "2023-12-31",
  "query": "",
  "maxItems": 100000,
  "proxyConfiguration": { "useApifyProxy": true }
}

Option B — A filtered historical slice (topic, author or high-engagement)

Add a keyword, an author allow-list, or point/comment thresholds to carve a targeted slice out of the window.

1. Keep your date range.
2. Add a Keyword filter (e.g. openai), and/or Authors (e.g. pg, dang), and/or Minimum Points.
3. Run and export.

{
  "contentType": "story",
  "startDate": "2022-01-01",
  "endDate": "2024-12-31",
  "query": "openai",
  "minPoints": 50,
  "maxItems": 5000
}

⚙️ Input parameters

Parameter	Type	Default	Description
contentType	select	story	story, comment, show_hn, ask_hn, poll, job, front_page, or all (stories + comments + polls + jobs combined).
startDate	string	last 3 days	Optional canonical YYYY-MM-DD (UTC day, inclusive) or YYYY-MM-DDTHH:mm:ss[.sss]Z (inclusive instant). If one edge is missing, a 3-day window is derived.
endDate	string	now	Optional canonical YYYY-MM-DD (full UTC day, inclusive) or YYYY-MM-DDTHH:mm:ss[.sss]Z (exclusive upper instant).
query	string	""	Optional keyword filter (title/text/comment). Leave empty for the full firehose.
authors	array	[]	Optional list of HN usernames to restrict to (no @). Each is queried separately.
minPoints	integer	0	Only items with at least this many points. 0 = no filter.
minComments	integer	0	Only items with at least this many comments. 0 = no filter.
maxItems	integer	5000	Global row cap. 0 removes this cap; window, time budget, Actor timeout and upstream coverage still apply.
timeBudgetSecs	integer	270	Algolia-work budget. Raise it together with the Actor run timeout for larger exports (30–86400).
proxyConfiguration	object	{ "useApifyProxy": true }	Apify Proxy. Datacenter is plenty for this public CDN endpoint.

Full input example

{
  "contentType": "all",
  "startDate": "2021-01-01",
  "endDate": "2021-03-31",
  "query": "",
  "authors": [],
  "minPoints": 0,
  "minComments": 0,
  "maxItems": 50000,
  "proxyConfiguration": { "useApifyProxy": true }
}

📤 Output example

A comment row, showing how comments link back into their thread via storyId / parentId:

{
  "objectID": "38836489",
  "type": "comment",
  "title": "",
  "url": "",
  "author": "elijahbenizzy",
  "points": null,
  "numComments": null,
  "text": "Oh don't get me started on outdoor cats...",
  "createdAt": "2024-01-01T23:59:50Z",
  "createdAtUnix": 1704153590,
  "storyId": "38835845",
  "parentId": "38836469",
  "hnUrl": "https://news.ycombinator.com/item?id=38836489",
  "tags": "comment,author_elijahbenizzy,story_38835845"
}

🧠 Tips for best results

• Leave query empty for a true firehose. Any keyword turns the run into a filtered slice instead of a full-archive export.
• Size both limits to your window. maxItems: 0 removes only the row cap. Raise timeBudgetSecs and the Actor run timeout together for larger exports.
• Split enormous ranges into chunks (e.g. one run per year) for easier restarts, smaller datasets and cheaper re-runs.
• Use contentType: "all" for one reverse-chronological mixed stream. With multiple authors, each author is queried as a separate block, so globally sort by createdAtUnix downstream if cross-author order matters; the global cap can be consumed by earlier authors.
• minPoints filters to high-signal content — great for "only stories that actually got traction". Remember comments and jobs have no points, so a non-zero value effectively drops them.
• authors builds per-user corpora — list several usernames to pull each author's full history inside the window.
• Comments carry storyId — group rows by it to reconstruct threads, or join against a separate stories export.
• Timestamps are UTC. Convert to your local zone downstream if needed; createdAtUnix is handy for exact math and sorting.
• Dates are inclusive. A bare endDate covers the full UTC calendar day by using the next midnight as an exclusive query boundary.
• Re-run on a schedule to keep a rolling, up-to-date archive of a topic or the whole firehose.

🔌 Integrations

Wire this Actor into your stack with Apify's built-in connectors and API:

• Google Sheets — push exported rows straight into a sheet for analysis or sharing.
• Slack — get notified when a scheduled archive run finishes.
• Zapier / Make / n8n — trigger downstream workflows on new results.
• Webhooks — POST run + dataset events to your own service.
• Schedule — run daily/weekly to maintain a fresh, growing HN corpus.
• Apify API & clients — pull datasets programmatically in JSON, CSV, Excel, XML or JSONL.

🧑‍💻 API usage

Run the Actor and fetch results programmatically. Get your token from Settings → Integrations in the Apify Console.

cURL

curl -X POST "https://api.apify.com/v2/acts/logiover~hacker-news-historical-archive-scraper/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"contentType":"story","startDate":"2024-01-01","endDate":"2024-01-31","maxItems":10000}'

Node.js (apify-client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });

const run = await client.actor('logiover/hacker-news-historical-archive-scraper').call({
  contentType: 'comment',
  startDate: '2023-01-01',
  endDate: '2023-06-30',
  maxItems: 50000,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.length, 'HN items');

Python (apify-client)

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")

run = client.actor("logiover/hacker-news-historical-archive-scraper").call(run_input={
    "contentType": "story",
    "startDate": "2022-01-01",
    "endDate": "2022-12-31",
    "query": "rust",
    "maxItems": 5000,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["createdAt"], item["title"])

🤖 Use with AI agents (MCP)

This Actor works as a tool for LLM agents through the Apify MCP (Model Context Protocol) server, so an agent can pull historical Hacker News data on demand to answer research questions or build corpora. Point your MCP-capable client at Apify's MCP server and call this Actor by name.

Example prompt: "Using the Hacker News Historical Archive Scraper, export all stories mentioning 'LLM' between 2023-01-01 and 2023-12-31 with at least 100 points, then summarise the ten most-discussed themes."

❓ FAQ

Is this really keyless — no API key or login?

Yes. It reads the public Algolia Hacker News Search API directly. No API key, no HN account, no OAuth, no cookies and no client secret are required.

How is this different from a Hacker News keyword-search scraper?

A keyword-search scraper returns one capped relevance page. This Actor walks a date-bounded stream newest-to-oldest: leave the keyword empty to retrieve the matching Algolia index slice, subject to maxItems, runtime and upstream coverage.

How far back does the data go?

Hacker News launched in 2006, and Algolia indexes it from the start, so any startDate from 2006-10 onward is reachable, right up to the present.

How much data can one run return? Isn't Algolia capped at 1,000 results?

Algolia caps a single result page near 1,000 rows. Timestamp-window pagination plus a boundary-second recovery query continues beyond that page; your maxItems, Actor runtime and any exceptional upstream retrieval limit still apply.

Why did my run return zero items?

Usually the date window brackets no activity, or the filters are too strict. Check that startDate/endDate cover real dates (2006→now), and relax query, minPoints or minComments. Comments and jobs have no points, so a non-zero minPoints will exclude them.

Why are some fields empty or null?

By design. Comments have no title, url, points or numComments; text posts have no outbound url; jobs have no points. Empty string / null means the field doesn't apply to that item type.

Can I export to CSV, Excel or JSON?

Yes. Every row is flat, so you can download the dataset as CSV, JSON, Excel, HTML, XML or JSONL from the Apify Console, or pull it via the API. It imports cleanly into spreadsheets, warehouses and notebooks.

How do I pull only comments (to build a dialogue corpus)?

Set Content Type to Comments. Each row is one comment with its HTML-stripped text, author, storyId and parentId, so you can reconstruct threads or feed the text straight into an NLP pipeline.

How do I export a specific user's Hacker News history?

Add their username(s) to Authors and set a wide date range. Each user is queried across the whole window for the chosen content type.

How fast is it, and how much does it cost?

It's a lightweight JSON API with no browser, so it's fast and cheap. Cost scales with the number of rows on the pay-per-result model — see the Pricing tab. Split very large ranges into per-year runs to control dataset size and cost.

Is scraping Hacker News legal?

The Actor reads publicly available Hacker News content through a public API. You are responsible for using the data in line with Hacker News' / Y Combinator's terms and any applicable laws such as GDPR. See Is it legal? below.

How often is the data updated?

The Algolia index tracks Hacker News in near real time, so recent items appear within minutes of posting. Schedule the Actor to keep a rolling archive current.

⚖️ Is it legal?

This Actor extracts only publicly available Hacker News data via a public API and is intended for legitimate research, analytics, archival and monitoring use. It requires no login and accesses no private or gated content. You are responsible for complying with Hacker News' / Y Combinator's terms of service, the API provider's terms, GDPR and any other applicable local laws and privacy regulations when collecting and using the data.

🔗 Related scrapers by logiover

• Hacker News Search Scraper — targeted keyword search across HN (stories, comments, Ask/Show), when you want ranked results for a query rather than a full-archive firehose.
• Reddit Historical Archive Scraper — the same date-bounded archive approach for Reddit posts and comments.
• Substack Newsletter Scraper — export newsletter posts and archives for tech-writing corpora.

👉 Browse all logiover scrapers on Apify Store — 180+ actors across news, developer tools, social media, real estate, jobs, crypto and B2B data.

---

📝 Changelog

2026-08-01

• Completed the August 2026 full health check: verified empty/programmatic default, Console UI default, and two source-informed alternative inputs on Apify.
• Confirmed successful live execution, non-empty structured output, dataset-field/type integrity, and logical sample quality within the 5-minute quality window.
• Replaced the field-level output schema with a canonical results dataset link so run results open correctly in the Apify Console.
• Removed mandatory input fields (startDate, endDate); empty input now reaches the Actor's documented broad/default mode.
• Declared 14 dataset fields from typed live cloud samples so the output contract is no longer an empty placeholder.
• Declared 14 nullable dataset fields from typed live cloud samples so the output contract is no longer an empty placeholder or brittle to sparse modes.

2026-08-01 — Health-check remediation

• August 2026 monthly health-check remediation is in progress; this build contains fixes verified from empty/default, Console-default, targeted live probes, or field-level semantic review.
• Replaced the field-level output schema with a canonical results dataset link so run results open correctly in the Apify Console.
• Removed mandatory input fields (startDate, endDate); empty input now reaches the Actor's documented broad/default mode.
• Fixed optional/single-edge date windows, inclusive UTC-day boundaries, strict date validation, reversed ranges and boundary-second pagination recovery; added an explicit configurable API-work budget.
• The final four-input matrix verdict will be appended after post-build cloud revalidation.

2026-07-22

• 🚀 Initial release. Bulk historical Hacker News export between any two dates via the keyless Algolia HN Search API. Time-window pagination beats the 1,000-results-per-query cap and returns hundreds of thousands of rows. Content types: stories, comments, Ask/Show HN, polls, jobs, front-page and all-combined. Filters for keyword, authors, minimum points and minimum comments. 14 flat fields per row, HTML-stripped comment/story text, de-duplication by item ID, datacenter proxy with retry + IP rotation.