# Forum Pain Finder (Hacker News + Stack Exchange) (`happysproutstore/forum-pain-finder`) Actor

Validate product ideas with real user pain: search Hacker News and Stack Exchange for any topic and get only the posts that complain, wish for a tool, look for an alternative or say they would pay. Rule-based, no API key, no LLM.

- **URL**: https://apify.com/happysproutstore/forum-pain-finder.md
- **Developed by:** [Recep Vural](https://apify.com/happysproutstore) (community)
- **Categories:** Marketing, AI, Developer tools
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 signal rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Forum Pain Finder (Hacker News + Stack Exchange)

Type a product idea or category such as `invoice software` or `habit tracker`. The Actor searches **Hacker News** (stories and comments) and **Stack Exchange** sites (Software Recommendations, Web Applications, Super User, Stack Overflow and any other site you name). It keeps **only the posts where real people complain, wish a tool existed, look for an alternative, or say they would pay**.

You get one row per signal with the link, the date, the score and the exact sentence. A summary of the **top recurring pains per term** with counts and example quotes is saved too.

No API key, no login, no LLM. Both sources are free official APIs, and every label comes from transparent keyword rules (the same classifier as our App Review Pain-Point Miner), so you can see exactly why each row was picked.

### Use cases

- **Validate a product or SaaS idea before you build it.** Do people complain about existing tools in the niche, and does anyone say "I'd pay for..."?
- **Find feature gaps in competitors.** Search `<competitor> alternative`, `<category> too expensive` or `<category>` and read the wish and alternative-seeking rows.
- **Write landing pages and ads in the customer's own words.** The excerpts are verbatim user language.
- **Prioritize a roadmap.** The summary ranks complaint categories (pricing, bugs, missing features, support, privacy...) by how many posts mention them.
- **Content and SEO research.** "Is there a tool that..." questions are ready-made article and video topics.

### What counts as a signal

| `signalType` | Example phrases matched |
|---|---|
| `willing_to_pay` | "I'd pay for", "would happily pay", "willing to pay", "take my money", "I'd subscribe" |
| `wish` | "I wish there was", "is there a tool", "would be nice", "why is there no", "someone should build" |
| `alternative_seeking` | "alternative to", "replacement for", "switched from", "looking for a tool", "what do you use for" |
| `pain` | "frustrating", "too expensive", "overpriced", "nightmare", "waste of time", "clunky", "I hate", "we lost" |

Each row also gets a `complaintCategory`: `pricing_subscription`, `bugs_crashes`, `missing_feature`, `accuracy_quality`, `support`, `privacy_data`, `localization`, `ads` or `general`. The category is read from the signal sentences only, not from the whole post.

Built-in noise filters (you can switch them off):

- **Search term check.** A row is kept only if every word of the term appears in the post or its thread title. The match is loose, so `invoice` also matches `invoicing`.
- **Launch-post filter.** "Show HN" and "Launch HN" stories are skipped because they are founders describing their own product. Comments under them are kept and flagged `inLaunchThread`.
- **Self-promotion phrases** such as "would love your feedback" or "we're an alternative to" are ignored.

### Input example

```json
{
  "searchTerms": ["invoice software", "habit tracker"],
  "sources": ["hackernews", "stackexchange"],
  "hackerNewsContent": "both",
  "stackExchangeSites": ["softwarerecs", "webapps", "superuser"],
  "dateFrom": "3 years",
  "dateTo": "",
  "maxPostsPerSource": 300,
  "signalTypes": ["willing_to_pay", "wish", "alternative_seeking", "pain"]
}
```

Dates can be absolute (`2024-01-31`) or relative (`6 months`, `2 years`). `maxPostsPerSource` is the number of posts **scanned** per term on Hacker News and on each Stack Exchange site. Only posts with a signal become rows.

### Output example (one dataset row)

```json
{
  "searchTerm": "invoice software",
  "source": "hackernews",
  "site": "news.ycombinator.com",
  "postType": "comment",
  "url": "https://news.ycombinator.com/item?id=...",
  "title": "Ask HN: What do you use for invoicing?",
  "date": "2025-03-14T09:12:40Z",
  "score": null,
  "signalType": "pain",
  "signalTypes": ["pain"],
  "complaintCategory": "pricing_subscription",
  "complaintCategories": ["pricing_subscription"],
  "matchedPhrase": "too expensive",
  "matchedPhrases": ["too expensive"],
  "willingToPay": false,
  "inLaunchThread": false,
  "excerpt": "Accountants are too expensive, and I don't, yet, generate enough money to justify them."
}
```

For a comment, `title` is the title of the thread it belongs to. `score` is HN points or Stack Exchange votes. HN comments have no public score, so theirs is `null`.

### Summary (key-value store record `OUTPUT`)

Per term you get:

- `postsScanned` per source and site
- `signalTypeCounts`
- `topRecurringPains`: each complaint category with its count and share, its top phrases and up to 3 example quotes with links
- `topMatchedPhrases`
- `recurringThemes`: two-word phrases that come up in at least 2 signal posts
- `topWishes`, `willingToPayQuotes` and `alternativeSeekingQuotes`

The record also lists any errors and the remaining Stack Exchange quota.

### Pricing

Pay per event: you are charged only for **signal rows actually returned** (`post-analyzed`). Posts that are scanned but contain no signal are free, and so is the summary record.

### Speed and limits

- A typical run with one term and the default settings takes about 5-15 seconds.
- **Hacker News:** Algolia returns at most 1,000 results per query. Results come in relevance order inside your date range.
- **Stack Exchange:** only questions are searched (title and body), not answers. The anonymous quota is **300 requests per day per IP**, and Apify runs may share IPs. For heavy use, add a free key from stackapps.com in `stackExchangeKey`, which raises the quota to 10,000 requests per day. The Actor follows the API's `backoff` field and stops cleanly when the quota runs low. Stack Exchange has much less activity than a few years ago, so short date ranges return few questions. The default range is 5 years.
- The rules are tuned for English. They are keyword signals, not perfect classification: sarcasm, quotes of other people and off-topic asides can be mislabelled. Always read the excerpt and open the link before making a decision.
- Stack Exchange content is licensed CC BY-SA. Rows keep the source link, and excerpts are short quotes. Keep attribution if you republish them.

### FAQ

**Why do I get few rows?** Most forum posts contain no complaint or wish. Raise `maxPostsPerSource`, widen `dateFrom`, add more Stack Exchange sites (e.g. `stackoverflow`, `apple`, `android`, `pm`, `freelancing`), or turn off `requireTermMatch`.

**Is Reddit included?** No. This Actor uses only free official APIs that allow this use without login.

# Actor input Schema

## `searchTerms` (type: `array`):

Topics or product categories to search, e.g. "invoice software", "habit tracker", "notion alternative". Each term is searched separately and gets its own summary.

## `sources` (type: `array`):

Where to search. Both use free official APIs (Algolia HN Search API and Stack Exchange API 2.3).

## `hackerNewsContent` (type: `string`):

Which Hacker News items to search. Comments hold most of the complaints and wishes.

## `stackExchangeSites` (type: `array`):

Site API names, e.g. softwarerecs, webapps, superuser, stackoverflow, apple, android, pm, workplace, freelancing, money. Each site costs one or more API requests per term (anonymous limit: 300 requests per day per IP).

## `dateFrom` (type: `string`):

Oldest post date. Absolute (2024-01-31) or relative ("6 months", "2 years", "30 days"). Empty means no lower limit. Stack Exchange activity has dropped in recent years, so short ranges return few Stack Exchange posts.

## `dateTo` (type: `string`):

Newest post date, absolute (2025-12-31) or relative ("1 month"). Empty means today.

## `maxPostsPerSource` (type: `integer`):

How many posts to scan per term for Hacker News and for each Stack Exchange site (most relevant first). Only posts with a signal become rows and are charged.

## `signalTypes` (type: `array`):

Only posts with at least one of these signals are returned.

## `requireTermMatch` (type: `boolean`):

Keep only posts whose text or thread title contains every word of the search term (loose match: "invoice" also matches "invoicing"). Turn off for broader, noisier results.

## `excludeLaunchPosts` (type: `boolean`):

Skip Hacker News stories titled "Show HN" or "Launch HN": they are founders describing their own product, not user demand. Comments under them are still analyzed (flagged inLaunchThread).

## `maxSignalRowsPerTerm` (type: `integer`):

Cap on output rows per term (0 = no cap). Willing-to-pay rows are kept first, then by score.

## `minScore` (type: `integer`):

Skip posts below this score (HN points / Stack Exchange votes). HN comments have no public score and are never skipped. Empty = no filter.

## `stackExchangeKey` (type: `string`):

Optional free key from stackapps.com. Raises the Stack Exchange quota from 300 to 10,000 requests per day. Not needed for normal runs.

## Actor input object example

```json
{
  "searchTerms": [
    "habit tracker"
  ],
  "sources": [
    "hackernews",
    "stackexchange"
  ],
  "hackerNewsContent": "both",
  "stackExchangeSites": [
    "softwarerecs",
    "webapps",
    "superuser"
  ],
  "dateFrom": "5 years",
  "maxPostsPerSource": 300,
  "signalTypes": [
    "willing_to_pay",
    "wish",
    "alternative_seeking",
    "pain"
  ],
  "requireTermMatch": true,
  "excludeLaunchPosts": true,
  "maxSignalRowsPerTerm": 0
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchTerms": [
        "habit tracker"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("happysproutstore/forum-pain-finder").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "searchTerms": ["habit tracker"] }

# Run the Actor and wait for it to finish
run = client.actor("happysproutstore/forum-pain-finder").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchTerms": [
    "habit tracker"
  ]
}' |
apify call happysproutstore/forum-pain-finder --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,happysproutstore/forum-pain-finder"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gzsCthQXdE5b0Fz8E/builds/Q2EhfNXk5r1MsBnGg/openapi.json