# Hacker News, Product Hunt & Substack Scraper (`bachiembmt/hacker-news-product-hunt-substack-scraper`) Actor

Hacker News scraper plus Product Hunt launches and Substack posts in one unified schema. Search stories and comments by keyword, tag, points and date, with only-new mode for trend monitoring and startup research. No login or API key.

- **URL**: https://apify.com/bachiembmt/hacker-news-product-hunt-substack-scraper.md
- **Developed by:** [Ha Phan Thanh Hai](https://apify.com/bachiembmt) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 items

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Hacker News, Product Hunt & Substack Scraper

Scrape **Hacker News** stories and comments, **Product Hunt** launches and **Substack** posts in one run, in a single schema, for trend monitoring and startup research. No login, no API key.

### What does Hacker News, Product Hunt & Substack Scraper do?

- Pull the Hacker News front page (top / new / best) or search it by keyword, tag (`story`, `show_hn`, `ask_hn`, `job`, `comment`), minimum points and date.
- Get the newest Product Hunt launches from the public feed, filtered by keyword.
- Pull the archive of any Substack publication, with full text for free posts.
- Add top-level Hacker News comments as separate items linked by `parentId`.
- Use `onlyNew` to skip anything returned in earlier runs.

### Why use it

- **Three sources, one schema**: points, author, date, URL and text in the same fields.
- **Search and filters** for Hacker News via Algolia (keywords, tags, points, dates).
- **Scheduled monitoring** with `onlyNew`.
- **Low price**: $0.80 per 1,000 items.
- **No login, no API key, no proxy.**

### Use cases

- Trend and launch monitoring for founders and investors.
- Competitor and keyword mentions on Hacker News.
- Newsletter curation from Substack publications.
- Startup and product research, lead-gen research.
- Datasets of tech discussions for LLM analysis.

### How to use

1. Open the Actor on Apify and click **Try for free** (or call it through the API).
2. Pick **sources** (`hackernews`, `producthunt`, `substack`) and add **keywords** or Substack publications.
3. Click **Start**, then download the dataset as JSON, CSV, Excel or HTML, or read it through the API.

#### Input example

```json
{
  "sources": ["hackernews", "producthunt", "substack"],
  "keywords": ["ai agents"],
  "hnTags": ["story", "show_hn"],
  "minPoints": 20,
  "since": "2026-10-01",
  "substackPublications": ["platformer", "www.lennysnewsletter.com"],
  "includeComments": true,
  "maxCommentsPerStory": 10,
  "maxItems": 50,
  "onlyNew": true
}
```

The default input (`{}`) returns the 20 top Hacker News stories plus the Product Hunt feed in about 30 seconds. `maxItems` applies per source (per publication for Substack); comments are extra.

### Output example

```json
{
  "source": "hackernews",
  "id": "50175070",
  "type": "story",
  "title": "The Future of CSS",
  "url": "https://www.joshwcomeau.com/css/interop-2027/",
  "author": "soheilpro",
  "points": 360,
  "commentsCount": 140,
  "publishedAt": "2026-10-21T01:47:28+00:00",
  "text": null,
  "tags": ["story"],
  "publication": null,
  "sourceUrl": "https://news.ycombinator.com/item?id=50175070",
  "parentId": null
}
```

`type` is `story`, `job`, `comment`, `launch` or `post`. `sourceUrl` is the page on the source platform; `url` is the linked target (the story link on Hacker News, the product page on Product Hunt, the post URL on Substack). `publication` is set for Substack only; Substack `points` = reactions.

### Pricing

Pay per event, no subscription. You pay only for what is returned:

| Event | Price per 1,000 | 1,000 | 10,000 | 100,000 |
|---|---|---|---|---|
| item | $0.80 | $0.80 | $8.00 | $80.00 |

Example: 1,000 items cost $0.80; 10,000 cost $8.00.

Hacker News comments count as items.

Platform usage (compute) is included in the event price; there is nothing else to pay for.

### Integrations & API

Run it from code with the Apify API. Replace `<APIFY_TOKEN>` with your API token.

**Python** (`pip install apify-client`)

```python
from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("bachiembmt/hacker-news-product-hunt-substack-scraper").call(run_input={"sources": ["hackernews"], "keywords": ["ai agents"], "minPoints": 20, "maxItems": 50})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)
```

**cURL** (run synchronously and get dataset items)

```bash
curl -X POST "https://api.apify.com/v2/acts/bachiembmt~hacker-news-product-hunt-substack-scraper/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"sources": ["hackernews"], "keywords": ["ai agents"], "minPoints": 20, "maxItems": 50}'
```

Works with:

- **Make** and **Zapier**: use the Apify app/module to run this Actor and pass the dataset on.
- **n8n**: use the Apify node or an HTTP Request node with the API call above.
- **Google Sheets**: export the dataset as CSV, or use the Apify integration to append rows automatically.
- **Webhooks**: trigger your own endpoint when a run finishes; combine with Apify schedules for recurring jobs.
- **AI agents**: expose the Actor as a tool through the [Apify MCP server](https://mcp.apify.com).

### FAQ

**Does Product Hunt return votes?**
No. The public feed has no votes or comment counts (those need a login token), so `points` and `commentsCount` are `null` for Product Hunt.

**Do I get paid Substack posts?**
Only the public preview; full text is returned for free posts.

**Can I run it on a schedule?**
Yes, enable `onlyNew` and each run returns just the new items.

**Do I need an account or API key for the source?**
No. The Actor needs no login or source API key. You only need an Apify account to run it.

**Is it legal to scrape this data?**
Only publicly visible data is collected. Respect each platform's terms and authors' copyright when republishing content. This is general information, not legal advice.

### Limitations

- **Product Hunt**: only the public Atom feed (about 20 most recently updated launches). No votes or comment counts (those need a login token), so `points` and `commentsCount` are `null`. Keyword filtering applies to the visible launch text only.
- **Substack**: full text only for free posts; paid posts return the public preview. Custom-domain publications work when entered as host or URL.
- **Hacker News**: search mode sorts newest first. `minPoints`, `since`, keywords and tags switch from the front-page feed to Algolia search. Only top-level comments are collected.
- Requests are throttled to about 1 per second per run, so large runs take time.
- Only publicly visible data is collected.

# Actor input Schema

## `sources` (type: `array`):

Which sources to scrape: hackernews, producthunt, substack.

## `keywords` (type: `array`):

Match any keyword (title/text). Hacker News runs one search per keyword; Product Hunt and Substack are filtered by text. Empty = no keyword filter.

## `hnFeed` (type: `string`):

Used when no keywords, tags, minimum points or date are set: which front-page list to read.

## `hnTags` (type: `array`):

Search only these kinds (OR): story, show_hn, ask_hn, job, comment. Switches Hacker News to search mode (newest first).

## `minPoints` (type: `integer`):

Hacker News only: skip items with fewer points.

## `since` (type: `string`):

Only items published on/after this date (YYYY-MM-DD or ISO datetime). Optional.

## `substackPublications` (type: `array`):

Subdomain (platformer), host (www.lennysnewsletter.com) or URL. Used when 'substack' is in Sources.

## `includeComments` (type: `boolean`):

Also output the top-level comments of each story as separate items with parentId.

## `maxCommentsPerStory` (type: `integer`):

Top-level comments per story (when comments are included). Comments are billed as items.

## `maxItems` (type: `integer`):

Stories / launches / posts per source (per publication for Substack). Comments are extra.

## `onlyNew` (type: `boolean`):

Skip items already returned by earlier runs (remembered in the named key-value store hn-ph-substack-seen).

## Actor input object example

```json
{
  "sources": [
    "hackernews",
    "producthunt"
  ],
  "hnFeed": "top",
  "minPoints": 0,
  "includeComments": false,
  "maxCommentsPerStory": 10,
  "maxItems": 20,
  "onlyNew": false
}
```

# Actor output Schema

## `items` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sources": [
        "hackernews",
        "producthunt"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("bachiembmt/hacker-news-product-hunt-substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "sources": [
        "hackernews",
        "producthunt",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("bachiembmt/hacker-news-product-hunt-substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sources": [
    "hackernews",
    "producthunt"
  ]
}' |
apify call bachiembmt/hacker-news-product-hunt-substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,bachiembmt/hacker-news-product-hunt-substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Su96OVkEwiM0GZdmE/builds/pPm9ETIPzvwZLQfu9/openapi.json