Hacker News, Product Hunt & Substack Scraper

Scrape Hacker News stories and comments, Product Hunt launches and Substack posts in one run, in a single schema, for trend monitoring and startup research. No login, no API key.

What does Hacker News, Product Hunt & Substack Scraper do?

• Pull the Hacker News front page (top / new / best) or search it by keyword, tag (story, show_hn, ask_hn, job, comment), minimum points and date.
• Get the newest Product Hunt launches from the public feed, filtered by keyword.
• Pull the archive of any Substack publication, with full text for free posts.
• Add top-level Hacker News comments as separate items linked by parentId.
• Use onlyNew to skip anything returned in earlier runs.

Why use it

• Three sources, one schema: points, author, date, URL and text in the same fields.
• Search and filters for Hacker News via Algolia (keywords, tags, points, dates).
• Scheduled monitoring with onlyNew.
• Low price: $0.80 per 1,000 items.
• No login, no API key, no proxy.

Use cases

• Trend and launch monitoring for founders and investors.
• Competitor and keyword mentions on Hacker News.
• Newsletter curation from Substack publications.
• Startup and product research, lead-gen research.
• Datasets of tech discussions for LLM analysis.

How to use

1. Open the Actor on Apify and click Try for free (or call it through the API).
2. Pick sources (hackernews, producthunt, substack) and add keywords or Substack publications.
3. Click Start, then download the dataset as JSON, CSV, Excel or HTML, or read it through the API.

Input example

{
  "sources": ["hackernews", "producthunt", "substack"],
  "keywords": ["ai agents"],
  "hnTags": ["story", "show_hn"],
  "minPoints": 20,
  "since": "2026-10-01",
  "substackPublications": ["platformer", "www.lennysnewsletter.com"],
  "includeComments": true,
  "maxCommentsPerStory": 10,
  "maxItems": 50,
  "onlyNew": true
}

The default input ({}) returns the 20 top Hacker News stories plus the Product Hunt feed in about 30 seconds. maxItems applies per source (per publication for Substack); comments are extra.

Output example

{
  "source": "hackernews",
  "id": "50175070",
  "type": "story",
  "title": "The Future of CSS",
  "url": "https://www.joshwcomeau.com/css/interop-2027/",
  "author": "soheilpro",
  "points": 360,
  "commentsCount": 140,
  "publishedAt": "2026-10-21T01:47:28+00:00",
  "text": null,
  "tags": ["story"],
  "publication": null,
  "sourceUrl": "https://news.ycombinator.com/item?id=50175070",
  "parentId": null
}

type is story, job, comment, launch or post. sourceUrl is the page on the source platform; url is the linked target (the story link on Hacker News, the product page on Product Hunt, the post URL on Substack). publication is set for Substack only; Substack points = reactions.

Pricing

Pay per event, no subscription. You pay only for what is returned:

Event	Price per 1,000	1,000	10,000	100,000
item	$0.80	$0.80	$8.00	$80.00

Example: 1,000 items cost $0.80; 10,000 cost $8.00.

Hacker News comments count as items.

Platform usage (compute) is included in the event price; there is nothing else to pay for.

Integrations & API

Run it from code with the Apify API. Replace <APIFY_TOKEN> with your API token.

Python (pip install apify-client)

from apify_client import ApifyClient

client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("bachiembmt/hacker-news-product-hunt-substack-scraper").call(run_input={"sources": ["hackernews"], "keywords": ["ai agents"], "minPoints": 20, "maxItems": 50})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

cURL (run synchronously and get dataset items)

curl -X POST "https://api.apify.com/v2/acts/bachiembmt~hacker-news-product-hunt-substack-scraper/run-sync-get-dataset-items?token=<APIFY_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{"sources": ["hackernews"], "keywords": ["ai agents"], "minPoints": 20, "maxItems": 50}'

Works with:

• Make and Zapier: use the Apify app/module to run this Actor and pass the dataset on.
• n8n: use the Apify node or an HTTP Request node with the API call above.
• Google Sheets: export the dataset as CSV, or use the Apify integration to append rows automatically.
• Webhooks: trigger your own endpoint when a run finishes; combine with Apify schedules for recurring jobs.
• AI agents: expose the Actor as a tool through the Apify MCP server.

FAQ

Does Product Hunt return votes? No. The public feed has no votes or comment counts (those need a login token), so points and commentsCount are null for Product Hunt.

Do I get paid Substack posts? Only the public preview; full text is returned for free posts.

Can I run it on a schedule? Yes, enable onlyNew and each run returns just the new items.

Do I need an account or API key for the source? No. The Actor needs no login or source API key. You only need an Apify account to run it.

Is it legal to scrape this data? Only publicly visible data is collected. Respect each platform's terms and authors' copyright when republishing content. This is general information, not legal advice.

Limitations

• Product Hunt: only the public Atom feed (about 20 most recently updated launches). No votes or comment counts (those need a login token), so points and commentsCount are null. Keyword filtering applies to the visible launch text only.
• Substack: full text only for free posts; paid posts return the public preview. Custom-domain publications work when entered as host or URL.
• Hacker News: search mode sorts newest first. minPoints, since, keywords and tags switch from the front-page feed to Algolia search. Only top-level comments are collected.
• Requests are throttled to about 1 per second per run, so large runs take time.
• Only publicly visible data is collected.