Stack Exchange Q&A Scraper

Questions, answers and tags from Stack Overflow and the ~200 other Stack Exchange sites, through the official public API v2.3. No login, no browser — plain HTTP. An app key is optional and free; the actor works without one.

Three surfaces, one actor, selected with mode:

mode	what you get
search	full-text and tag search over questions (/search/advanced)
questions	questions by tag, or by explicit question ID
tags	the site's tag list with usage counts

Turn on includeAnswers to pull every answer for the questions returned, and includeBody to get the post bodies (both HTML and stripped text).

What this actor gets right that a naive client does not

1. The walk stops at page 25 — and the API says "more" right up to it. Without an app key, page 26 is refused. Page 25 still returns a full 100 items with has_more: true, so a while has_more: loop walks straight into an error instead of ending. Measured on q="pandas dataframe": reachable 2,500 rows against a total of 90,477 — 2.76%. Every summary reports pageCeilingHit, hasMoreAtStop, upstreamTotal and reachableFraction, so a truncated result is never handed over as a complete one.

2. Every API error is HTTP 400, whatever it actually is. A missing method (error_id 404), a refused deep page (error_id 403) and a bad parameter (error_id 400) are indistinguishable by status code. This actor classifies from the body, so a 400 is never retried as though it were transient and the page ceiling is reported as a ceiling rather than a failure.

3. There is a second wall, and it is not the quota. Beyond the daily allowance Stack Exchange runs a per-IP edge throttle that answers HTTP 429 with an HTML page and Retry-After: 247 — hit during this actor's own development while quota_remaining was still 217 of 300. A 2/4/8-second retry ladder cannot outlast four minutes. 429 gets its own handling: Retry-After is read, short waits are sat out on the shared rate limiter, and a long one ends the job with a message that says plainly the daily quota is not the problem.

4. total is not in the default filter — and on /sites it is a lie. The default response carries only items, has_more, quota_max and quota_remaining; payload["total"] is absent and reads as zero. This actor spends one extra request per query to fetch it deliberately. On /sites the same field reports 0 while page 2 still returns 100 sites, so it is never trusted there.

5. /tags?sort=popular returns each page alphabetised. The page holds the right set — the most-used tags — but in name order, so items[0] is android (1.41M), not javascript (2.52M). Rows are re-sorted by count, and each one keeps upstreamPageRank, the position the API actually sent it in.

6. An empty result is not an error. A minScore nothing meets answers HTTP 200 with items: []. That comes back as a summary row with resultsReturned: 0, not an ERROR — and never as silence.

7. Answers come back 100 per request, not 100 per question. One /questions/{ids}/answers call returns a global top-100 by score across the whole batch of IDs. Measured on 50 questions declaring 1,469 answers: one page gave 100 of them — 6.8% — between 1 and 4 per question. This actor pages through instead of stopping at one, and reports answersTruncated and questionsWithoutAnswerRows when even that is not enough.

8. body is absent unless you ask for it. The default filter omits post bodies entirely. Rows carry bodyRequested so a null body reads as a filter choice rather than a missing post.

Input

{
  "mode": "search",                  // search | questions | tags
  "site": "stackoverflow",
  "queries": ["pandas dataframe"],
  "tagged": ["python", "pandas"],    // ANDed by the API
  "sort": "votes",                   // relevance|votes|activity|creation (search)
  "order": "desc",
  "acceptedOnly": false,
  "minScore": 5,
  "includeBody": false,
  "includeAnswers": false,
  "maxResultsPerQuery": 100,         // 0 = unlimited, but see the ceiling
  "pageSize": 100,
  "startPage": 1,                    // resume or shard a long walk
  "apiKey": ""                       // optional, free, raises the limits
}

Output

One SEARCH_SUMMARY row per query, then the data rows.

recordType	when
SEARCH_SUMMARY	always, one per query
QUESTION	search and questions modes
ANSWER	when includeAnswers is on
TAG	tags mode
ERROR	invalid input or an upstream failure — every input yields at least one row

Every row carries _input, _source, _scrapedAt and recordType. Unix epochs come with ISO copies beside them (creationDate / creationDateIso).

The summary reports resultsReturned, answersReturned, requestsMade, pagesFetched, upstreamTotal, pageCeilingHit, hasMoreAtStop, reachableFraction, filterUsed, quotaRemaining / quotaMax, backoffsHonoured, throttlesWaitedOut and — in tags mode — upstreamPageWasResorted.

Limits worth knowing before you run it

• 300 requests per day per IP without an app key; quotaRemaining is on every summary. A free key from stackapps.com raises it to 10,000 and lifts the page-25 ceiling. It is not a login and grants no private access.
• Page 25 is the wall without a key: at pageSize: 100 that is 2,500 questions per query, whatever maxResultsPerQuery says.
• Pace yourself. minRequestInterval defaults to 1 second and maxConcurrency to 2 for the edge throttle described above.
• The API is public data under CC BY-SA; each row carries its contentLicense.

Notes

• No WAF. All six TLS profiles tried answered 200 cold, no warmup, no proxy. Responses are always gzipped. A proxy is available but off by default — note the daily quota is per IP, so a rotating proxy spreads it.
• api.stackexchange.com serves no robots.txt (the path answers HTTP 400), so RFC 9309's "unavailable" case applies.