Forum Pain Finder (Hacker News + Stack Exchange)

Type a product idea or category such as invoice software or habit tracker. The Actor searches Hacker News (stories and comments) and Stack Exchange sites (Software Recommendations, Web Applications, Super User, Stack Overflow and any other site you name). It keeps only the posts where real people complain, wish a tool existed, look for an alternative, or say they would pay.

You get one row per signal with the link, the date, the score and the exact sentence. A summary of the top recurring pains per term with counts and example quotes is saved too.

No API key, no login, no LLM. Both sources are free official APIs, and every label comes from transparent keyword rules (the same classifier as our App Review Pain-Point Miner), so you can see exactly why each row was picked.

Use cases

• Validate a product or SaaS idea before you build it. Do people complain about existing tools in the niche, and does anyone say "I'd pay for..."?
• Find feature gaps in competitors. Search <competitor> alternative, <category> too expensive or <category> and read the wish and alternative-seeking rows.
• Write landing pages and ads in the customer's own words. The excerpts are verbatim user language.
• Prioritize a roadmap. The summary ranks complaint categories (pricing, bugs, missing features, support, privacy...) by how many posts mention them.
• Content and SEO research. "Is there a tool that..." questions are ready-made article and video topics.

What counts as a signal

signalType	Example phrases matched
willing_to_pay	"I'd pay for", "would happily pay", "willing to pay", "take my money", "I'd subscribe"
wish	"I wish there was", "is there a tool", "would be nice", "why is there no", "someone should build"
alternative_seeking	"alternative to", "replacement for", "switched from", "looking for a tool", "what do you use for"
pain	"frustrating", "too expensive", "overpriced", "nightmare", "waste of time", "clunky", "I hate", "we lost"

Each row also gets a complaintCategory: pricing_subscription, bugs_crashes, missing_feature, accuracy_quality, support, privacy_data, localization, ads or general. The category is read from the signal sentences only, not from the whole post.

Built-in noise filters (you can switch them off):

• Search term check. A row is kept only if every word of the term appears in the post or its thread title. The match is loose, so invoice also matches invoicing.
• Launch-post filter. "Show HN" and "Launch HN" stories are skipped because they are founders describing their own product. Comments under them are kept and flagged inLaunchThread.
• Self-promotion phrases such as "would love your feedback" or "we're an alternative to" are ignored.

Input example

{
  "searchTerms": ["invoice software", "habit tracker"],
  "sources": ["hackernews", "stackexchange"],
  "hackerNewsContent": "both",
  "stackExchangeSites": ["softwarerecs", "webapps", "superuser"],
  "dateFrom": "3 years",
  "dateTo": "",
  "maxPostsPerSource": 300,
  "signalTypes": ["willing_to_pay", "wish", "alternative_seeking", "pain"]
}

Dates can be absolute (2024-01-31) or relative (6 months, 2 years). maxPostsPerSource is the number of posts scanned per term on Hacker News and on each Stack Exchange site. Only posts with a signal become rows.

Output example (one dataset row)

{
  "searchTerm": "invoice software",
  "source": "hackernews",
  "site": "news.ycombinator.com",
  "postType": "comment",
  "url": "https://news.ycombinator.com/item?id=...",
  "title": "Ask HN: What do you use for invoicing?",
  "date": "2025-03-14T09:12:40Z",
  "score": null,
  "signalType": "pain",
  "signalTypes": ["pain"],
  "complaintCategory": "pricing_subscription",
  "complaintCategories": ["pricing_subscription"],
  "matchedPhrase": "too expensive",
  "matchedPhrases": ["too expensive"],
  "willingToPay": false,
  "inLaunchThread": false,
  "excerpt": "Accountants are too expensive, and I don't, yet, generate enough money to justify them."
}

For a comment, title is the title of the thread it belongs to. score is HN points or Stack Exchange votes. HN comments have no public score, so theirs is null.

Summary (key-value store record OUTPUT)

Per term you get:

• postsScanned per source and site
• signalTypeCounts
• topRecurringPains: each complaint category with its count and share, its top phrases and up to 3 example quotes with links
• topMatchedPhrases
• recurringThemes: two-word phrases that come up in at least 2 signal posts
• topWishes, willingToPayQuotes and alternativeSeekingQuotes

The record also lists any errors and the remaining Stack Exchange quota.

Pricing

Pay per event: you are charged only for signal rows actually returned (post-analyzed). Posts that are scanned but contain no signal are free, and so is the summary record.

Speed and limits

• A typical run with one term and the default settings takes about 5-15 seconds.
• Hacker News: Algolia returns at most 1,000 results per query. Results come in relevance order inside your date range.
• Stack Exchange: only questions are searched (title and body), not answers. The anonymous quota is 300 requests per day per IP, and Apify runs may share IPs. For heavy use, add a free key from stackapps.com in stackExchangeKey, which raises the quota to 10,000 requests per day. The Actor follows the API's backoff field and stops cleanly when the quota runs low. Stack Exchange has much less activity than a few years ago, so short date ranges return few questions. The default range is 5 years.
• The rules are tuned for English. They are keyword signals, not perfect classification: sarcasm, quotes of other people and off-topic asides can be mislabelled. Always read the excerpt and open the link before making a decision.
• Stack Exchange content is licensed CC BY-SA. Rows keep the source link, and excerpts are short quotes. Keep attribution if you republish them.

FAQ

Why do I get few rows? Most forum posts contain no complaint or wish. Raise maxPostsPerSource, widen dateFrom, add more Stack Exchange sites (e.g. stackoverflow, apple, android, pm, freelancing), or turn off requireTermMatch.

Is Reddit included? No. This Actor uses only free official APIs that allow this use without login.