Technical Q&A Search — Stack Exchange Question Miner

Search Stack Overflow and 27 sister Stack Exchange sites by keyword and/or tag, and get back one structured JSON row per question: title, plain-text body excerpt, tags, score, view count, answer count, accepted-answer flag, author reputation, URL — optionally with the highest-voted answer.

Built for developer-tooling research, content and SEO pipelines, support-deflection analysis, competitive "what breaks for users" studies, RAG corpus building, and any agent that needs real technical Q&A rather than a marketing page.

Why this Actor

Quora serves a captcha interstitial to datacenter IPs (probed directly, no proxy route fixes it) — that is why the previous Q&A Actor was retired. The Stack Exchange API is the open alternative: it answers 200 with real question items datacenter-direct, no API key, no Cloudflare, and it covers the same buyer intent with far better structure (tags, scores, accepted answers, view counts, author reputation).

Feasibility was proven from Apify datacenter IPs before this Actor was written (dc-probe run U0iaJmkfyXFZZh2bS):

Probe target	Result
api.stackexchange.com/2.3/search/advanced (+ filter=withbody)	200, real questions
api.stackexchange.com/2.3/questions/{ids}/answers	200, real answers
api.stackexchange.com/2.3/sites	200, 100 sites listed
softwarerecs.stackexchange.com (search)	200, real questions
quora.com (the retired source)	captcha interstitial

What it does

1. Runs every query against every selected site with search/advanced, newest state of the public API (filter=withbody returns question HTML in the same call — no second request).
2. Applies client-side quality filters the API cannot express: minScore, minAnswers, acceptedOnly.
3. Converts each question body from HTML to readable plain text and truncates it to a word-boundary excerpt.
4. Optionally fetches the highest-voted answer for every emitted question (one extra API call per query, not per question).
5. Reports per-query/per-site fetch-vs-emit counts, the remaining anonymous quota, and any API error in the run's SUMMARY key-value record.

Input

{
  "queries": ["python asyncio", "kubernetes ingress"],
  "sites": ["stackoverflow", "serverfault"],
  "tagged": [],
  "sort": "relevance",
  "minScore": 5,
  "minAnswers": 1,
  "acceptedOnly": false,
  "maxResultsPerQuery": 25,
  "includeBody": true,
  "bodyExcerptChars": 1000,
  "includeTopAnswer": false
}

Option	Type	Description
queries	string[]	Keyword searches (default ["python asyncio"]).
sites	string[]	Stack Exchange sites by api_site_parameter (default ["stackoverflow"]). Any valid parameter works, not just the curated 28.
tagged	string[]	Tag filter, ANDed by the API, e.g. ["python","asyncio"] (default []).
sort	string	relevance (default), votes, creation, activity.
minScore	integer	Drop questions scoring below this (default 0).
minAnswers	integer	Drop questions with fewer answers (default 0).
acceptedOnly	boolean	Only questions with an accepted answer (default false).
maxResultsPerQuery	integer	Max questions per query per site, also the API page size (default 25, max 100).
includeBody	boolean	Include the plain-text question body (default true).
bodyExcerptChars	integer	Excerpt length, cut at a word boundary (default 1000).
includeTopAnswer	boolean	Fetch the highest-voted answer per emitted question (default false).

Output — one item per question

Field	Description
questionId	Stack Exchange question id
site, siteName	api_site_parameter and its display name
title	Question title
url	Canonical question URL
bodyExcerpt	Question text, HTML stripped, truncated at a word boundary
bodyLength	Full plain-text length before truncation
tags	Tag list
score	Upvotes minus downvotes
viewCount	Lifetime views
answerCount	Number of answers
isAnswered	API's "has an upvoted-or-accepted answer" flag
hasAcceptedAnswer, acceptedAnswerId	Whether an answer was accepted, and its id
creationDate, lastActivityDate	UTC ISO-8601 timestamps
owner	{displayName, reputation, userId, profileUrl}
contentLicense	Content licence reported by the API (attribution requirement)
query, sort	Which input produced this row, and under which ordering
topAnswer	{answerId, score, isAccepted, bodyExcerpt, creationDate, ownerDisplayName, ownerReputation, url} or null
scrapedAt	Extraction timestamp (UTC ISO-8601)

Worked example

Input:

{ "queries": ["python asyncio"], "sites": ["stackoverflow"], "minScore": 5, "includeTopAnswer": true }

Output (trimmed):

{
  "questionId": 27435284,
  "site": "stackoverflow",
  "siteName": "Stack Overflow",
  "title": "multiprocessing vs multithreading vs asyncio",
  "url": "https://stackoverflow.com/questions/27435284/multiprocessing-vs-multithreading-vs-asyncio",
  "bodyExcerpt": "A pod in my cluster is stuck on \"ContainerCreating\" …",
  "tags": ["python", "asyncio", "multiprocessing"],
  "score": 399,
  "viewCount": 1234567,
  "answerCount": 13,
  "isAnswered": true,
  "hasAcceptedAnswer": true,
  "acceptedAnswerId": 27436735,
  "creationDate": "2014-11-20T17:00:41+00:00",
  "owner": {"displayName": "jsbueno", "reputation": 12345, "userId": 7},
  "query": "python asyncio",
  "topAnswer": {"answerId": 48484593, "score": 120, "isAccepted": false}
}

1,000 questions cost about $3.00 in event fees.

Run it from your code or on a schedule

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("darknezz/technical-qa-search").call(
    run_input={"queries": ["python asyncio", "kubernetes ingress"],
               "sites": ["stackoverflow"], "minScore": 5},
    max_total_charge_usd=5.00,
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["score"], item["title"], item["url"])

REST equivalent:

curl -X POST "https://api.apify.com/v2/acts/darknezz~technical-qa-search/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"queries":["python asyncio","kubernetes ingress"],"sites":["stackoverflow"],"minScore":5}'

Schedule a weekly sweep of your product's keywords, or trigger a run per support ticket to surface prior art. max_total_charge_usd (Console → Run options → Maximum cost per run) caps spend: 10 queries × 1 site × 25 results ≈ 250 questions ≈ $0.75.

Pricing

Pay per event: $0.003 per question (question-extracted primary event), charged once per emitted question — fetching the top answer adds no per-item charge, only one API call per query. 1,000 questions ≈ $3.00; 10,000 ≈ $30.00.

FAQ

Do I need a Stack Exchange API key? No. Anonymous requests work; the shared quota is about 300 calls/day per IP, and each query costs one call (plus one if includeTopAnswer is on). The run's SUMMARY record reports quotaRemaining and warns below 30.

What happens when the quota runs out? The API replies with a backoff/throttle error; the Actor logs it, records it in SUMMARY.errors, and finishes the run with whatever it already collected instead of crashing.

Why do I get fewer results than maxResultsPerQuery? Filters (minScore, minAnswers, acceptedOnly) run after fetch, and the query/page is a sample of matches rather than the whole corpus.

Can I search multiple pages? Not in this version: one page per query per site (up to 100 results) keeps runs inside the store's quality window. Use several more specific queries instead — that is also how the API is meant to be used.

Which sites can I pick? Anything from GET /2.3/sites (about 180). The 28 in the dropdown are the ones that matter for technical work; softwarerecs is included for "what tool should I use" questions.

Is the body text HTML? No — it is converted to plain text (code blocks fenced) so it drops straight into an embedding pipeline, a spreadsheet or an LLM prompt.

Can I reuse the content commercially? Stack Exchange content is CC BY-SA (each row carries contentLicense); attribution is your responsibility. The Actor extracts metadata and excerpts — not full answer threads.

Is this a Quora scraper? No. Quora blocks datacenter traffic with an interactive captcha, so this Actor deliberately uses the open, keyless Stack Exchange API instead.

Limitations

• One page per query per site (max 100 results), by design; fan out with more queries.
• Excerpts, not full threads. The question body is truncated to bodyExcerptChars; only the single top answer is fetched, and only when includeTopAnswer is on.
• Deleted, closed or migrated questions may appear with a link that redirects — the API reports what is live at request time.
• Anonymous quota is per-IP and shared with everything else egressing from the same Apify datacenter IP; the Actor surfaces it rather than hiding it.
• No answer-level deep crawl. If you need every answer with comments, that is a different, heavier Actor — ask for it.