Hacker News Tech Stories Scraper

Searches public Hacker News stories through the Algolia HN search API, using a configurable technology query such as technology, AI, programming, or open source. Records include stable story IDs, titles, authors, normalized timestamps, points, comment counts, canonical HTTPS Hacker News URLs, optional HTTPS story URLs, domains, story text, query/rank metadata, tags, Algolia highlights, child IDs, and optional real top-level Firebase comments.

Input

{
  "query": "technology",
  "maxItems": 25,
  "sortByDate": false,
  "includeComments": true,
  "maxCommentsPerStory": 2
}

Requests are bounded, retried and delayed; deleted or incomplete comments are omitted, insecure story URLs are not followed or emitted, and no search result is fabricated. Search and comment failures are summarized in the OUTPUT key only; they never create dataset rows. A comment-enrichment issue after stories are saved keeps the run successful and marks its data PARTIAL.

Output

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

Each dataset item is a source-backed story with a stable recordId and recordType: "story". Story items contain Algolia story identity, title, author, timestamps, points, comment count, a canonical Hacker News URL, an optional HTTPS story URL, domain, text, query and rank metadata, tags, highlights, child IDs, and optional Firebase topComments. No diagnostic or placeholder rows are written.

Example input: {"query":"technology","maxItems":10,"sortByDate":false,"includeComments":true,"maxCommentsPerStory":2}

Example output: {"storyId":"12345678","titleText":"Example technology story","author":"example-user","points":180,"commentCount":32,"storyUrl":"https://example.com/story","searchQuery":"technology","searchRank":1,"tags":["story","author_example-user"],"hackerNewsUrl":"https://news.ycombinator.com/item?id=12345678"}

Key-value store

The exact OUTPUT key contains a count-only summary with status and runStatus (SUCCEEDED or FAILED) plus dataStatus (COMPLETE, PARTIAL, NO_RESULTS, or FAILED). A run with saved stories and later comment issues is SUCCEEDED/PARTIAL; an empty clean search is SUCCEEDED/NO_RESULTS; a failed search with no saved stories is FAILED/FAILED.

Cost and limits

The main cost is one Algolia request plus up to four bounded Firebase comment requests per story. Keep maxItems and maxCommentsPerStory modest.

Local run

apify run --purge --entrypoint src/main.js --input '{"query":"technology","maxItems":5,"includeComments":true,"maxCommentsPerStory":2}'
npm run validate

The validator checks stable IDs, source-backed story records only, HTTPS evidence URLs, timestamps, duplicate story identities, canonical statuses, and consistency between dataset counts and OUTPUT. It also accepts an empty dataset when the summary reports a clean NO_RESULTS outcome or a true FAILED outcome.

Generated by Codex with GPT-5.6 Sol.