# Hacker News Historical Archive Scraper β€” Old Stories by Date (`logiover/hacker-news-historical-archive-scraper`) Actor Bulk-export the entire Hacker News archive between any two dates in chronological order. Stories, comments, Ask/Show HN, polls and jobs since 2006 via the keyless Algolia HN API. Time-windowed firehose yields hundreds of thousands of rows. No API key, no login. - **URL**: https://apify.com/logiover/hacker-news-historical-archive-scraper.md - **Developed by:** [Logiover](https://apify.com/logiover) (community) - **Categories:** News, Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.50 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Hacker News Historical Archive Scraper β€” Old Stories & Comments by Date ![Apify Actor](https://img.shields.io/badge/Apify-Actor-00A67E?logo=apify&logoColor=white) ![No API key](https://img.shields.io/badge/No%20API%20key-required-2ea44f) ![Pay per result](https://img.shields.io/badge/Pricing-Pay%20per%20result-1C7ED6) ![News](https://img.shields.io/badge/Category-News-FF6600) ![Export](https://img.shields.io/badge/Export-JSON%20%7C%20CSV%20%7C%20Excel-F59E0B) **Download the entire Hacker News archive between any two dates.** Point this scraper at a date range β€” a week, a year, or every day since HN launched in 2006 β€” and it bulk-exports the full chronological corpus of **stories, comments, Ask HN, Show HN, polls and jobs** as clean, flat rows. It runs on the **keyless Algolia Hacker News Search API** and uses a **time-window pagination engine** to slide backward through time and blow past Algolia's 1,000-results-per-query ceiling, so a single run can return **hundreds of thousands of rows**. **No API key, no login, no cookies, no browser.** > ### πŸ† This is a bulk historical firehose β€” not a keyword search > Most HN scrapers answer *"give me the top results for keyword X."* This one answers *"give me **everything** posted to Hacker News between date A and date B."* Leave the keyword empty and you get the whole archive for the window β€” every story and every comment, in chronological order, ready for AI training corpora, trend research and dataset building. Export to **JSON / CSV / Excel**. --- ### ✨ What does this Actor do? This Actor is a **date-bounded chronological exporter** for the complete Hacker News corpus. You give it a `startDate`, an `endDate` and a `contentType` (stories, comments, Ask/Show HN, polls, jobs, or all combined) and it pulls every matching item in that window from the public **Algolia HN Search API** (`search_by_date`), newest to oldest. The engine's core trick is **time-window pagination**. Algolia caps any single query at ~1,000 results (`page Γ— hitsPerPage ≀ 1000`), so a naive `page++` loop silently stops at 1,000 rows and can never export a real range. Instead, this Actor requests a page of 1,000, records the **oldest** `created_at_i` in the batch, then re-queries with that timestamp as the new upper bound β€” sliding the window strictly backward through time and repeating until the range is exhausted. That's how it reaches **hundreds of thousands of rows** from one date range. Items are de-duplicated by `objectID`. It's **keyless**, fast, needs no browser, and covers HN from its **2006 launch to today**. ### πŸ‘₯ Who is it for? - **AI / ML & NLP teams** building domain-specific training corpora from years of high-signal tech discussion and comment threads. - **Data scientists & quantitative researchers** who need a complete, date-bounded HN dataset for trend, topic and sentiment modelling. - **Academic researchers & social scientists** studying the tech community, startup culture or online discourse longitudinally. - **Data journalists & analysts** reconstructing how a topic, company or technology was discussed across a precise historical window. - **Brand, PR & market-intelligence teams** monitoring mentions of a product, company or person across the full HN back-catalogue over time. ### 🎯 Use cases - **Build an AI training corpus** β€” export every comment and story in a multi-year window to fine-tune or evaluate language models on real technical dialogue. - **Tech-trend & sentiment analysis** β€” pull all stories mentioning a technology across several years and chart how attention and engagement shifted. - **Academic study of a community** β€” collect a complete date-bounded slice of HN to study discourse, moderation or topic diffusion. - **Brand & reputation monitoring over time** β€” retrieve every historical mention of your company or product to measure share-of-voice and sentiment. - **Dataset building & retrospective analysis** β€” snapshot an entire era of Hacker News (e.g. the 2021 crypto run-up, the 2023 AI boom) as a reusable dataset. ### πŸ’‘ Why use this Actor? - πŸ”‘ **100% keyless** β€” no API key, no login, no cookies, no client secret. It reads the public Algolia HN endpoint directly. - 🌊 **True bulk firehose** β€” leave the keyword empty and export **every** item in your date window, not just a capped top-N. - ♾️ **Beats the 1,000-result cap** β€” the time-window engine slides backward through time to return hundreds of thousands of rows across a wide range. - 🧾 **14 clean fields per row** β€” id, type, title, url, author, points, comment count, HTML-stripped text, timestamps, story/parent IDs, HN permalink and tags. - πŸŽ›οΈ **Precise filters** β€” content type, date window, optional keyword, author allow-list, minimum points and minimum comments. - πŸ“€ **Export anywhere** β€” download as JSON, CSV, Excel, HTML, XML or JSONL, or pull via the Apify API and webhooks. **Pay-per-result** pricing. ### πŸ“Š What data can you extract? Every story or comment comes back as one flat row. Story rows carry `title`, `url`, `points` and `numComments`; comment rows carry the `text` (HTML-stripped) plus `storyId` and `parentId` linking them back into their thread. | Field | Type | Description | |-------|------|-------------| | `objectID` | string | Hacker News item ID (unique per story/comment). | | `type` | string | `story`, `comment`, `poll`, `job`, `show_hn` or `ask_hn`. | | `title` | string | Story / poll / Ask / Show title (empty for comments). | | `url` | string | Outbound link of a story (empty for text posts and comments). | | `author` | string | Hacker News username of the poster. | | `points` | number | Upvote score (stories/polls; `null` for comments/jobs). | | `numComments` | number | Number of comments on the story/poll. | | `text` | string | HTML-stripped body β€” comment text or story self-text. | | `createdAt` | string | ISO 8601 UTC timestamp of posting. | | `createdAtUnix` | number | Creation time as UNIX epoch seconds. | | `storyId` | string | ID of the parent story (comments and Ask/Show sub-items). | | `parentId` | string | ID of the direct parent item (comments). | | `hnUrl` | string | Permalink on `news.ycombinator.com`. | | `tags` | string | Algolia item tags, comma-joined (e.g. `story,author_pg,story_38836483`). | #### Example row (story) ```json { "objectID": "38836483", "type": "story", "title": "The Rise of Unix. The Seeds of Its Fall", "url": "https://www.youtube.com/watch?v=HADp3emVABg", "author": "tux1968", "points": 6, "numComments": 0, "text": "", "createdAt": "2024-01-01T23:58:35Z", "createdAtUnix": 1704153515, "storyId": "38836483", "parentId": "", "hnUrl": "https://news.ycombinator.com/item?id=38836483", "tags": "story,author_tux1968,story_38836483" } ```` ### πŸš€ How to use #### Option A β€” Export the whole archive for a date window (firehose) The default, most powerful mode. Set a content type and a date range and leave the keyword empty to pull **everything**. 1. Set **Content Type** to `Stories` (or `Comments`, `All`, etc.). 2. Set **Start Date** and **End Date** (`YYYY-MM-DD`, UTC, inclusive). 3. Raise **Max Items** to the corpus size you want (e.g. `100000`). 4. Click **Start**, then export from the **Output** tab. ```json { "contentType": "story", "startDate": "2023-01-01", "endDate": "2023-12-31", "query": "", "maxItems": 100000, "proxyConfiguration": { "useApifyProxy": true } } ``` #### Option B β€” A filtered historical slice (topic, author or high-engagement) Add a keyword, an author allow-list, or point/comment thresholds to carve a targeted slice out of the window. 1. Keep your date range. 2. Add a **Keyword filter** (e.g. `openai`), and/or **Authors** (e.g. `pg`, `dang`), and/or **Minimum Points**. 3. Run and export. ```json { "contentType": "story", "startDate": "2022-01-01", "endDate": "2024-12-31", "query": "openai", "minPoints": 50, "maxItems": 5000 } ``` ### βš™οΈ Input parameters | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `contentType` | select | `story` | `story`, `comment`, `show_hn`, `ask_hn`, `poll`, `job`, `front_page`, or `all` (stories + comments + polls + jobs combined). | | `startDate` | string | `2024-01-01` | Oldest date to include (`YYYY-MM-DD`, UTC, inclusive). HN data starts in 2006. **Required.** | | `endDate` | string | `2024-01-08` | Newest date to include (`YYYY-MM-DD`, UTC, inclusive to end-of-day). **Required.** | | `query` | string | `""` | Optional keyword filter (title/text/comment). Leave empty for the full firehose. | | `authors` | array | `[]` | Optional list of HN usernames to restrict to (no `@`). Each is queried separately. | | `minPoints` | integer | `0` | Only items with at least this many points. `0` = no filter. | | `minComments` | integer | `0` | Only items with at least this many comments. `0` = no filter. | | `maxItems` | integer | `5000` | Global cap on rows. Raise for a full corpus; `0` = unlimited (bounded by window + timeout). | | `proxyConfiguration` | object | `{ "useApifyProxy": true }` | Apify Proxy. Datacenter is plenty for this public CDN endpoint. | #### Full input example ```json { "contentType": "all", "startDate": "2021-01-01", "endDate": "2021-03-31", "query": "", "authors": [], "minPoints": 0, "minComments": 0, "maxItems": 50000, "proxyConfiguration": { "useApifyProxy": true } } ``` ### πŸ“€ Output example A comment row, showing how comments link back into their thread via `storyId` / `parentId`: ```json { "objectID": "38836489", "type": "comment", "title": "", "url": "", "author": "elijahbenizzy", "points": null, "numComments": null, "text": "Oh don't get me started on outdoor cats...", "createdAt": "2024-01-01T23:59:50Z", "createdAtUnix": 1704153590, "storyId": "38835845", "parentId": "38836469", "hnUrl": "https://news.ycombinator.com/item?id=38836489", "tags": "comment,author_elijahbenizzy,story_38835845" } ``` ### 🧠 Tips for best results - **Leave `query` empty for a true firehose.** Any keyword turns the run into a filtered slice instead of a full-archive export. - **Size `maxItems` to your window.** A single busy day is ~600 stories and ~6,000+ comments; a full year of comments is well into the millions. Set `maxItems: 0` (unlimited) only when you also allow enough run timeout. - **Split enormous ranges into chunks** (e.g. one run per year) for easier restarts, smaller datasets and cheaper re-runs. - **Use `contentType: "all"`** to interleave stories and comments in one chronological stream; use `comment` alone to build a pure dialogue corpus. - **`minPoints` filters to high-signal content** β€” great for "only stories that actually got traction". Remember comments and jobs have no points, so a non-zero value effectively drops them. - **`authors` builds per-user corpora** β€” list several usernames to pull each author's full history inside the window. - **Comments carry `storyId`** β€” group rows by it to reconstruct threads, or join against a separate stories export. - **Timestamps are UTC.** Convert to your local zone downstream if needed; `createdAtUnix` is handy for exact math and sorting. - **Dates are inclusive.** `endDate` covers the whole final day (to 23:59:59 UTC). - **Re-run on a schedule** to keep a rolling, up-to-date archive of a topic or the whole firehose. ### πŸ”Œ Integrations Wire this Actor into your stack with Apify's built-in connectors and API: - **Google Sheets** β€” push exported rows straight into a sheet for analysis or sharing. - **Slack** β€” get notified when a scheduled archive run finishes. - **Zapier / Make / n8n** β€” trigger downstream workflows on new results. - **Webhooks** β€” POST run + dataset events to your own service. - **Schedule** β€” run daily/weekly to maintain a fresh, growing HN corpus. - **Apify API & clients** β€” pull datasets programmatically in JSON, CSV, Excel, XML or JSONL. ### πŸ§‘β€πŸ’» API usage Run the Actor and fetch results programmatically. Get your token from **Settings β†’ Integrations** in the Apify Console. **cURL** ```bash curl -X POST "https://api.apify.com/v2/acts/logiover~hacker-news-historical-archive-scraper/runs?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"contentType":"story","startDate":"2024-01-01","endDate":"2024-01-31","maxItems":10000}' ``` **Node.js** (`apify-client`) ```javascript import { ApifyClient } from 'apify-client'; const client = new ApifyClient({ token: 'YOUR_TOKEN' }); const run = await client.actor('logiover/hacker-news-historical-archive-scraper').call({ contentType: 'comment', startDate: '2023-01-01', endDate: '2023-06-30', maxItems: 50000, }); const { items } = await client.dataset(run.defaultDatasetId).listItems(); console.log(items.length, 'HN items'); ``` **Python** (`apify-client`) ```python from apify_client import ApifyClient client = ApifyClient("YOUR_TOKEN") run = client.actor("logiover/hacker-news-historical-archive-scraper").call(run_input={ "contentType": "story", "startDate": "2022-01-01", "endDate": "2022-12-31", "query": "rust", "maxItems": 5000, }) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item["createdAt"], item["title"]) ``` ### πŸ€– Use with AI agents (MCP) This Actor works as a tool for LLM agents through the **Apify MCP (Model Context Protocol)** server, so an agent can pull historical Hacker News data on demand to answer research questions or build corpora. Point your MCP-capable client at Apify's MCP server and call this Actor by name. > Example prompt: *"Using the Hacker News Historical Archive Scraper, export all stories mentioning 'LLM' between 2023-01-01 and 2023-12-31 with at least 100 points, then summarise the ten most-discussed themes."* ### ❓ FAQ #### Is this really keyless β€” no API key or login? Yes. It reads the public **Algolia Hacker News Search API** directly. No API key, no HN account, no OAuth, no cookies and no client secret are required. #### How is this different from a Hacker News keyword-search scraper? A keyword-search scraper returns a capped set of the best matches for a search term. This Actor is a **bulk, date-bounded firehose**: leave the keyword empty and it exports **every** story and comment in your date window in chronological order β€” the whole archive slice, not a top-N. Use it when you want completeness over a time range, not relevance ranking for one query. #### How far back does the data go? Hacker News launched in **2006**, and Algolia indexes it from the start, so any `startDate` from **2006-10** onward is reachable, right up to the present. #### How much data can one run return? Isn't Algolia capped at 1,000 results? Algolia caps a *single query* at ~1,000 results, but this Actor uses **time-window pagination** β€” it slides the query window backward through time using the oldest timestamp from each batch β€” so a run can return **hundreds of thousands of rows** across a wide date range. The 1,000 cap does not limit your export. #### Why did my run return zero items? Usually the date window brackets no activity, or the filters are too strict. Check that `startDate`/`endDate` cover real dates (2006β†’now), and relax `query`, `minPoints` or `minComments`. Comments and jobs have no `points`, so a non-zero `minPoints` will exclude them. #### Why are some fields empty or null? By design. Comments have no `title`, `url`, `points` or `numComments`; text posts have no outbound `url`; jobs have no `points`. Empty string / `null` means the field doesn't apply to that item type. #### Can I export to CSV, Excel or JSON? Yes. Every row is flat, so you can download the dataset as **CSV, JSON, Excel, HTML, XML or JSONL** from the Apify Console, or pull it via the API. It imports cleanly into spreadsheets, warehouses and notebooks. #### How do I pull only comments (to build a dialogue corpus)? Set **Content Type** to `Comments`. Each row is one comment with its HTML-stripped `text`, `author`, `storyId` and `parentId`, so you can reconstruct threads or feed the text straight into an NLP pipeline. #### How do I export a specific user's Hacker News history? Add their username(s) to **Authors** and set a wide date range. Each user is queried across the whole window for the chosen content type. #### How fast is it, and how much does it cost? It's a lightweight JSON API with no browser, so it's fast and cheap. Cost scales with the number of rows on the **pay-per-result** model β€” see the **Pricing** tab. Split very large ranges into per-year runs to control dataset size and cost. #### Is scraping Hacker News legal? The Actor reads **publicly available** Hacker News content through a public API. You are responsible for using the data in line with Hacker News' / Y Combinator's terms and any applicable laws such as GDPR. See **Is it legal?** below. #### How often is the data updated? The Algolia index tracks Hacker News in near real time, so recent items appear within minutes of posting. Schedule the Actor to keep a rolling archive current. ### βš–οΈ Is it legal? This Actor extracts only **publicly available** Hacker News data via a public API and is intended for legitimate research, analytics, archival and monitoring use. It requires no login and accesses no private or gated content. You are responsible for complying with Hacker News' / Y Combinator's terms of service, the API provider's terms, GDPR and any other applicable local laws and privacy regulations when collecting and using the data. ### πŸ”— Related scrapers by logiover - [Hacker News Search Scraper](https://apify.com/logiover/hacker-news-search-scraper) β€” targeted **keyword search** across HN (stories, comments, Ask/Show), when you want ranked results for a query rather than a full-archive firehose. - [Reddit Historical Archive Scraper](https://apify.com/logiover/reddit-historical-archive-scraper) β€” the same date-bounded archive approach for **Reddit** posts and comments. - [Substack Newsletter Scraper](https://apify.com/logiover/substack-newsletter-scraper) β€” export newsletter posts and archives for tech-writing corpora. πŸ‘‰ Browse all **[logiover scrapers on Apify Store](https://apify.com/logiover)** β€” 180+ actors across news, developer tools, social media, real estate, jobs, crypto and B2B data. *** ### πŸ“ Changelog #### 2026-07-22 - πŸš€ Initial release. Bulk historical Hacker News export between any two dates via the keyless Algolia HN Search API. Time-window pagination beats the 1,000-results-per-query cap and returns hundreds of thousands of rows. Content types: stories, comments, Ask/Show HN, polls, jobs, front-page and all-combined. Filters for keyword, authors, minimum points and minimum comments. 14 flat fields per row, HTML-stripped comment/story text, de-duplication by item ID, datacenter proxy with retry + IP rotation. # Actor input Schema ## `contentType` (type: `string`): Which kind of Hacker News item to export. 'Stories' = link + text posts (the classic front-page firehose). 'Comments' = every comment body with its parent/story. 'Show HN' / 'Ask HN' = those two dedicated categories. 'Polls', 'Jobs' = those item types. 'All' = stories + comments + polls + jobs combined in one chronological stream. ## `startDate` (type: `string`): The oldest date to include, as YYYY-MM-DD (UTC). Everything created on or after this date is exported. HN launched in 2006, so any date from 2006-10-01 onward is reachable. Example: '2024-01-01'. ## `endDate` (type: `string`): The newest date to include, as YYYY-MM-DD (UTC). Everything created on or before the END of this day is exported. Combine with Start Date to define your window. Example: '2024-01-08'. ## `query` (type: `string`): OPTIONAL keyword to narrow the export (matched against titles, text and comment bodies). Leave EMPTY for a pure firehose that returns every item in the date window. Use this only if you want a topic slice, e.g. 'openai', 'rust', 'layoffs'. ## `authors` (type: `array`): OPTIONAL list of Hacker News usernames to restrict the export to (no @ prefix). Each username is queried separately across the date window. Leave empty to include all authors. Example: \['pg', 'dang', 'patio11']. ## `minPoints` (type: `integer`): Only include items with at least this many points (upvotes). 0 = no filter. Useful to pull only high-engagement stories. Note: comments and jobs do not carry points, so a non-zero value effectively limits results to stories/polls. ## `minComments` (type: `integer`): Only include items with at least this many comments. 0 = no filter. Applies to stories/Ask/Show/polls (comments themselves have no comment count). ## `maxItems` (type: `integer`): Global hard cap on rows returned by the run. Default 5000 is a sane starting point; raise it (e.g. 100000+) to pull a full multi-year corpus. 0 = unlimited (bounded only by the date window and the run timeout). ## `proxyConfiguration` (type: `object`): Apify Proxy configuration. The Algolia HN Search API is a fast public CDN endpoint that works over a direct connection, so datacenter proxy (the default) is plenty and keeps cost low. A fresh proxy session is minted per retry to rotate away from any throttled IP. ## Actor input object example ```json { "contentType": "story", "startDate": "2024-01-01", "endDate": "2024-01-08", "query": "", "authors": [], "minPoints": 0, "minComments": 0, "maxItems": 5000, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `objectID` (type: `string`): Hacker News item ID. ## `type` (type: `string`): Item type: story, comment, poll, job, show\_hn or ask\_hn. ## `title` (type: `string`): Story / poll / Ask / Show title (empty for comments). ## `url` (type: `string`): Outbound link of a story (empty for text posts and comments). ## `author` (type: `string`): Hacker News username of the poster. ## `points` (type: `string`): Upvote score (stories/polls). ## `numComments` (type: `string`): Number of comments on the story/poll. ## `text` (type: `string`): HTML-stripped body β€” comment text or story self-text. ## `createdAt` (type: `string`): ISO 8601 UTC timestamp when the item was posted. ## `createdAtUnix` (type: `string`): Creation time as UNIX epoch seconds. ## `storyId` (type: `string`): ID of the parent story (for comments and Ask/Show sub-items). ## `parentId` (type: `string`): ID of the direct parent item (for comments). ## `hnUrl` (type: `string`): Permalink to the item on news.ycombinator.com. ## `tags` (type: `string`): Algolia item tags, comma-joined. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startDate": "2024-01-01", "endDate": "2024-01-08", "proxyConfiguration": { "useApifyProxy": true } }; // Run the Actor and wait for it to finish const run = await client.actor("logiover/hacker-news-historical-archive-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startDate": "2024-01-01", "endDate": "2024-01-08", "proxyConfiguration": { "useApifyProxy": True }, } # Run the Actor and wait for it to finish run = client.actor("logiover/hacker-news-historical-archive-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startDate": "2024-01-01", "endDate": "2024-01-08", "proxyConfiguration": { "useApifyProxy": true } }' | apify call logiover/hacker-news-historical-archive-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=logiover/hacker-news-historical-archive-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Hacker News Historical Archive Scraper β€” Old Stories by Date", "description": "Bulk-export the entire Hacker News archive between any two dates in chronological order. Stories, comments, Ask/Show HN, polls and jobs since 2006 via the keyless Algolia HN API. Time-windowed firehose yields hundreds of thousands of rows. No API key, no login.", "version": "1.0", "x-build-id": "HaFwMFbm9cFpcymYn" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/logiover~hacker-news-historical-archive-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-logiover-hacker-news-historical-archive-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/logiover~hacker-news-historical-archive-scraper/runs": { "post": { "operationId": "runs-sync-logiover-hacker-news-historical-archive-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/logiover~hacker-news-historical-archive-scraper/run-sync": { "post": { "operationId": "run-sync-logiover-hacker-news-historical-archive-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startDate", "endDate" ], "properties": { "contentType": { "title": "Content Type", "enum": [ "story", "comment", "show_hn", "ask_hn", "poll", "job", "front_page", "all" ], "type": "string", "description": "Which kind of Hacker News item to export. 'Stories' = link + text posts (the classic front-page firehose). 'Comments' = every comment body with its parent/story. 'Show HN' / 'Ask HN' = those two dedicated categories. 'Polls', 'Jobs' = those item types. 'All' = stories + comments + polls + jobs combined in one chronological stream.", "default": "story" }, "startDate": { "title": "Start Date (inclusive, UTC)", "type": "string", "description": "The oldest date to include, as YYYY-MM-DD (UTC). Everything created on or after this date is exported. HN launched in 2006, so any date from 2006-10-01 onward is reachable. Example: '2024-01-01'." }, "endDate": { "title": "End Date (inclusive, UTC)", "type": "string", "description": "The newest date to include, as YYYY-MM-DD (UTC). Everything created on or before the END of this day is exported. Combine with Start Date to define your window. Example: '2024-01-08'." }, "query": { "title": "Keyword filter (optional)", "type": "string", "description": "OPTIONAL keyword to narrow the export (matched against titles, text and comment bodies). Leave EMPTY for a pure firehose that returns every item in the date window. Use this only if you want a topic slice, e.g. 'openai', 'rust', 'layoffs'.", "default": "" }, "authors": { "title": "Authors (optional)", "type": "array", "description": "OPTIONAL list of Hacker News usernames to restrict the export to (no @ prefix). Each username is queried separately across the date window. Leave empty to include all authors. Example: ['pg', 'dang', 'patio11'].", "default": [], "items": { "type": "string" } }, "minPoints": { "title": "Minimum Points", "minimum": 0, "type": "integer", "description": "Only include items with at least this many points (upvotes). 0 = no filter. Useful to pull only high-engagement stories. Note: comments and jobs do not carry points, so a non-zero value effectively limits results to stories/polls.", "default": 0 }, "minComments": { "title": "Minimum Comments", "minimum": 0, "type": "integer", "description": "Only include items with at least this many comments. 0 = no filter. Applies to stories/Ask/Show/polls (comments themselves have no comment count).", "default": 0 }, "maxItems": { "title": "Max Items", "minimum": 0, "type": "integer", "description": "Global hard cap on rows returned by the run. Default 5000 is a sane starting point; raise it (e.g. 100000+) to pull a full multi-year corpus. 0 = unlimited (bounded only by the date window and the run timeout).", "default": 5000 }, "proxyConfiguration": { "title": "Proxy", "type": "object", "description": "Apify Proxy configuration. The Algolia HN Search API is a fast public CDN endpoint that works over a direct connection, so datacenter proxy (the default) is plenty and keeps cost low. A fresh proxy session is minted per retry to rotate away from any throttled IP.", "default": { "useApifyProxy": true } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```