Hacker News Search Scraper

Search public Hacker News stories, comments and author activity, then export the results as CSV, Excel or JSON. Use keyword, item-type and date filters to research products, technical discussions or recurring mentions.

Quick start

Start with a story search and a small result cap:

{
  "query": "apify",
  "tags": ["story"],
  "includeComments": false,
  "sortBy": "relevance",
  "maxItems": 5
}

Run the Actor and open its dataset. maxItems is a saved-result cap. A matching result is a story or comment record, not an entire discussion thread.

Prefer a video guide? This short Apify walkthrough shows you how to set up and run an Actor. It uses a different Actor, but the setup steps are the same.

Ready-to-run examples

Choose a public example to see its settings before starting a run:

• Export public Hacker News activity for an author
• Export Hacker News comments about a technology topic
• Search Hacker News stories for a product or company

What you get

One matching public story or comment record, with its item ID, Hacker News link, type and available author, date, text and engagement fields. Story and parent identifiers can connect records for later review, but a result row is not a whole discussion thread. Missing comment fields are not treated as zero engagement.

Who is it for?

• Developer-relations teams reviewing public mentions of products or companies.
• Product researchers reading matching technical discussion excerpts before source review.
• Community analysts inspecting public author activity and item context.
• Data teams maintaining their own repeat-search exports and notification workflows.

How you can use the data

• Review a small product-mention sample. Use the existing apify story search with relevance order and a five-record cap. Retain objectId, title, author, createdAt and hnUrl before opening the discussion for context.
• Search comments as their own records. Select the comment tag for the topic rather than combining story and comment expecting two independent searches. Keep commentText, storyId and parentId together, with missing points or titles left unknown.
• Review public activity for a known author. Supply the HN username you want to research and a small cap, then keep author, item type and source links together. This is public search activity rather than private-account access or a full profile export.
• Compare recurring mentions yourself. Retain item IDs across scheduled exports, apply your own analysis and alerts, and inspect summary cutoff information before treating a missing match as an absence. Sorting collected records by points is not a search for the highest-scoring items across all HN history.

Input settings

Input	Meaning
query	Search text. Can be omitted for tag-, author- or date-oriented searches
tags	Item-type filters such as story, comment, ask_hn or show_hn
author	Public HN username, converted to an author filter
includeComments	Adds the comment tag to the search
dateFrom, dateTo	Inclusive dates in YYYY-MM-DD format
sortBy	relevance, date, or ordering the collected results by points
maxItems	Saved-result cap: default 50, form prefill 20, maximum 5,000

For a comment search, use tags: ["comment"]. Do not combine story and comment expecting the Actor to fetch both types separately: tags are passed together to the search service. Sorting by points orders the collected results. It is not a guarantee of the highest-scoring items across all HN history.

Example inputs

Use the Quick start input as a small first run. Adjust the fields described above for your own data.

Output data

Each row represents one search result. Optional fields vary between stories and comments.

Fields	What they contain
objectId, type, hnUrl	Item identity, item type and Hacker News link
title, url, author, createdAt	Available title, destination, author and timestamp
points, numComments	Available engagement counts
storyId, parentId	Related story or parent identifiers
commentText, highlight, tags	Available comment text, matched snippet and tags

A comment may have no points or its own title. Missing values are not zero engagement. numComments is a count. It does not mean every comment in the thread was collected.

Illustrative output shape only, not a captured search result:

{
  "objectId": "123",
  "type": "comment",
  "title": null,
  "url": null,
  "hnUrl": "https://news.ycombinator.com/item?id=123",
  "author": null,
  "createdAt": null,
  "points": null,
  "numComments": null,
  "storyId": null,
  "parentId": null,
  "commentText": "Example comment text",
  "highlight": null,
  "tags": ["comment"]
}

Download your results

Open your run's dataset, choose Export and select JSON, CSV or Excel. JSON keeps nested fields together. CSV and Excel are useful when you want to review the records in a spreadsheet.

See Apify's download and export guide for the available formats and field options.

Pricing

The start event is charged once after input validation. The result event is charged for each result immediately before it is saved. There are no paid error or status rows. A valid search with no matches can still incur the start event.

Use a small saved-result cap and a focused date interval initially. See the Pricing tab for current rates and plan discounts.

Tips for best results

• Choose the intended item type explicitly. Use the comment tag for comments rather than combining mutually exclusive story and comment tags.
• Keep parent and story IDs with comment records. They support source review without implying the complete thread was collected.
• Leave missing engagement fields unknown. An absent count is not zero activity.
• Use date filters for a focused sample. A cap and date interval do not guarantee every matching historical item.
• Inspect RUN_SUMMARY after short or interrupted work. Pending-search information does not guarantee automatic continuation in a later run.

Limits and caveats

Results come from public Hacker News search data. Search-service coverage, pagination, filters and the run budget can limit collection. A saved-result cap is a maximum rather than a promised number of matches, and private accounts or generated sentiment analysis are outside scope.

Rows represent matching items rather than complete discussion threads. A comment can lack points, a title or other optional fields. Comment counts do not establish that every comment was retrieved. Combined tags are sent together to the source rather than searched separately. Points sorting only orders the collected records, without guaranteeing the highest-scoring items across all history.

RUN_SUMMARY reports status and errors, with completed searches also reporting saved results, fetched pages and cutoff information. Time-bounded work may save PENDING_SEARCH. A checkpoint does not prove the search was exhausted or that a new run automatically resumes. Repeat-export history, alerts and analysis are handled in your own application.

Automation and integrations

Save your settings for next time

Once you have an input that works for you, save it as an Apify task. This keeps your settings together so you can use them again without filling in the form from scratch.

Run on a schedule

Need fresh results regularly? You can schedule the Actor or a saved task to run at a time that suits you.

Each scheduled run collects the data again. If you want to compare results or highlight changes, add that step in your own workflow.

Send results to your other tools

Use n8n, Make or Zapier to connect this scraper to the rest of your workflow. For example, you could pass its results to an AI assistant or add records to a spreadsheet. Follow the Apify integration guides for your chosen tool.

For a custom connection, an Apify webhook can notify your app when a run finishes or fails. Your app can then fetch the results or handle the failure.

API usage

Want to collect this data from your own app or automation? The examples below send input to the scraper and show how to collect its results.

The Node.js and Python examples wait for the run to finish before reading its results. The cURL example starts the run and returns its details. Wait for that run to finish, then use its defaultDatasetId to fetch the results.

For setup instructions, see Apify's JavaScript client guide, Python client guide and Actor runs API reference.

cURL

curl -X POST 'https://api.apify.com/v2/acts/fetch_cat~hacker-news-search-scraper/runs' \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  --data @- <<'JSON'
{
  "query": "apify",
  "tags": [
    "story"
  ],
  "includeComments": false,
  "sortBy": "relevance",
  "maxItems": 5
}
JSON

Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('fetch_cat/hacker-news-search-scraper').call({
  "query": "apify",
  "tags": [
    "story"
  ],
  "includeComments": false,
  "sortBy": "relevance",
  "maxItems": 5
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('fetch_cat/hacker-news-search-scraper').call(run_input={'query': 'apify', 'tags': ['story'], 'includeComments': False, 'sortBy': 'relevance', 'maxItems': 5})
print(client.dataset(run['defaultDatasetId']).list_items().items)

MCP and AI agents

You can ask your AI assistant to collect data with Hacker News Search Scraper and help you work with the results. The scraper returns the data, and your assistant can then summarize it, compare records or prepare notes.

Connect your assistant through the Apify MCP server. MCP gives your assistant a way to find this scraper, run it and read the results. Follow the official setup and sign-in guide for your app.

Easier setup with an Apify plugin

If you use a supported coding assistant, the official Apify plugin can set up the connection for you. It connects your assistant to Apify and includes instructions for common tasks. Follow the guide for your particular app.

Use Apify AI in Console

Prefer to keep everything in Apify? Open Apify AI in Console and ask it to find Fetch Cat's Hacker News Search Scraper. You can describe the data you want and ask questions about the results.

Apify AI is currently in beta. It asks for confirmation before starting an Actor, and normal run charges apply.

Choose your assistant

• Claude and Claude Code (Anthropic): Apify setup guide.
• ChatGPT and Codex (OpenAI): Apify setup guides.
• Grok Bot (xAI): Apify plugin guide.
• Gemini CLI (Google): MCP server setup guide. Use the server URL below with the HTTP configuration supported by Gemini CLI.

Connect just this scraper

To keep your setup focused on this scraper, add this server configuration to a compatible MCP client:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=fetch_cat/hacker-news-search-scraper"
    }
  }
}

Sign in to Apify when your client asks. If your client uses a Bearer token instead, follow its setup guide and keep the token in its secure configuration. Configuration formats vary by app, so use the linked guide if your client needs a different format.

Use Apify's general server

Already connected to the general Apify MCP server at https://mcp.apify.com? You can use it to find this scraper without adding another server. Ask your assistant to use Hacker News Search Scraper by Fetch Cat, or give it the identifier fetch_cat/hacker-news-search-scraper.

Your assistant can check the available settings and retrieve the results from the returned run. If the run is still working, it should follow that run rather than start it again. Both connection options use Apify's hosted service.

Example requests

• “Use Fetch Cat’s Hacker News Search Scraper to collect up to 5 story matches for apify with relevance order and comments off. Keep item IDs, source discussion links and available authors and dates for source review.”
• “Collect up to 5 comment matches for apify using only the comment item-type tag and date order. Keep comment text, parent IDs and story IDs without treating each result as a full thread.”
• “Retrieve up to 5 story matches for apify dated from 1 July through 31 July 2026, with comments off and the collected results ordered by points. Keep missing counts unknown and do not claim these are the highest-scoring items across all history.”

Normal Apify charges apply. Check the live Pricing tab before asking your assistant to run a larger batch.

Pay with x402

If your AI agent has a funded wallet, it can pay for Apify usage through x402. This experimental option lets an agent use Apify without a conventional account.

Apify AGI provides a prepaid API token with a spending cap after payment. Your agent can use that token to run this scraper through the Apify API or an authenticated MCP connection.

Some MCP setups also support paying for each request directly. That is a different payment option, so check the official guide for its availability.

Follow the official guide for wallet setup, supported payment options and current terms. Apify handles the payments. Keep wallet keys and payment tokens in your agent's secure configuration.

Other ways for agents to pay

Apify also supports MPP for prepaid access through AGI and Skyfire for eligible Actors. If your agent already uses one of these, see the agentic payment guide to choose the right setup.

FAQ

Why is my comment search empty?

Check the tags first. Use comment for comments and avoid mutually exclusive item-type tags. Broaden the query or date interval if necessary.

Can I track mentions over time?

Yes. Use an Apify schedule, retain earlier exports and compare objectId to identify new matches. Your application handles cross-run history, alerts and analysis.

What should I check after a short or interrupted run?

Inspect RUN_SUMMARY for status and error. Completed searches also report resultsSaved, pagesFetched and cutoff. A time-bounded run may save PENDING_SEARCH. It is a checkpoint, not proof that the requested search was exhausted or that a later run resumes automatically.

Does this collect an entire thread?

No. It returns matching search records. Use the item links and identifiers for a separate full-thread workflow.

Related Actors

• Reddit Scraper — Collect public Reddit discussions.
• GitHub Repositories Search Scraper — Find related software projects.
• arXiv Paper Search Scraper — Research academic papers on a topic.
• RSS Feed Reader — Collect developer and publisher feeds.
• Google News Scraper — Search news beyond Hacker News.

Support and feedback

For custom additions or integrations, email hello@fetchcat.dev.

Found an error or have a feature request? Open an issue with the run URL, a non-sensitive input example and what you expected.

Need help sharing it? See Apify’s guide to sharing a run when reporting an issue. Sharing a run can give the developer access to its input, logs and results, so use a run that contains no sensitive data. Please leave out API tokens, passwords, cookies, private documents and personal data. Use a public sample URL or a redacted example instead.

Tried this scraper? We'd love to hear how it worked for you. Leave a review to help others choose the right tool and help us improve it. Your feedback means a lot to us.

Made with care for curious people, AI agents and the possibilities in data. ♥