# Stack Exchange Q\&A Scraper (`scrapyx/stackexchange-qa-scraper`) Actor

Searches questions, answers and tags across Stack Overflow and the ~200 other Stack Exchange sites. Reports what the daily quota and the page-25 ceiling actually let through, re-sorts the tag pages the API returns alphabetised, and reads errors from the body since every one arrives as HTTP 400.

- **URL**: https://apify.com/scrapyx/stackexchange-qa-scraper.md
- **Developed by:** [Ibnu Adzim](https://apify.com/scrapyx) (community)
- **Categories:** Developer tools, AI, Education
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.10 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Stack Exchange Q\&A Scraper

Questions, answers and tags from **Stack Overflow** and the ~200 other Stack
Exchange sites, through the official public API v2.3. No login, no browser —
plain HTTP. An app key is optional and free; the actor works without one.

Three surfaces, one actor, selected with `mode`:

| mode | what you get |
|---|---|
| `search` | full-text and tag search over questions (`/search/advanced`) |
| `questions` | questions by tag, or by explicit question ID |
| `tags` | the site's tag list with usage counts |

Turn on `includeAnswers` to pull every answer for the questions returned, and
`includeBody` to get the post bodies (both HTML and stripped text).

### What this actor gets right that a naive client does not

**1. The walk stops at page 25 — and the API says "more" right up to it.**
Without an app key, page 26 is refused. Page 25 still returns a full 100 items
with `has_more: true`, so a `while has_more:` loop walks straight into an
error instead of ending. Measured on `q="pandas dataframe"`: reachable
**2,500 rows against a `total` of 90,477 — 2.76%**. Every summary reports
`pageCeilingHit`, `hasMoreAtStop`, `upstreamTotal` and `reachableFraction`, so
a truncated result is never handed over as a complete one.

**2. Every API error is HTTP 400, whatever it actually is.** A missing method
(`error_id` 404), a refused deep page (`error_id` 403) and a bad parameter
(`error_id` 400) are indistinguishable by status code. This actor classifies
from the body, so a 400 is never retried as though it were transient and the
page ceiling is reported as a ceiling rather than a failure.

**3. There is a second wall, and it is not the quota.** Beyond the daily
allowance Stack Exchange runs a per-IP edge throttle that answers **HTTP 429
with an HTML page and `Retry-After: 247`** — hit during this actor's own
development while `quota_remaining` was still 217 of 300. A 2/4/8-second retry
ladder cannot outlast four minutes. 429 gets its own handling: Retry-After is
read, short waits are sat out on the shared rate limiter, and a long one ends
the job with a message that says plainly the daily quota is not the problem.

**4. `total` is not in the default filter — and on `/sites` it is a lie.** The
default response carries only `items`, `has_more`, `quota_max` and
`quota_remaining`; `payload["total"]` is absent and reads as zero. This actor
spends one extra request per query to fetch it deliberately. On `/sites` the
same field reports **0** while page 2 still returns 100 sites, so it is never
trusted there.

**5. `/tags?sort=popular` returns each page alphabetised.** The page holds the
right *set* — the most-used tags — but in name order, so `items[0]` is
`android` (1.41M), not `javascript` (2.52M). Rows are re-sorted by count, and
each one keeps `upstreamPageRank`, the position the API actually sent it in.

**6. An empty result is not an error.** A `minScore` nothing meets answers
HTTP 200 with `items: []`. That comes back as a summary row with
`resultsReturned: 0`, not an `ERROR` — and never as silence.

**7. Answers come back 100 per *request*, not 100 per question.** One
`/questions/{ids}/answers` call returns a global top-100 by score across the
whole batch of IDs. Measured on 50 questions declaring 1,469 answers: one page
gave 100 of them — 6.8% — between 1 and 4 per question. This actor pages
through instead of stopping at one, and reports `answersTruncated` and
`questionsWithoutAnswerRows` when even that is not enough.

**8. `body` is absent unless you ask for it.** The default filter omits post
bodies entirely. Rows carry `bodyRequested` so a null body reads as a filter
choice rather than a missing post.

### Input

```jsonc
{
  "mode": "search",                  // search | questions | tags
  "site": "stackoverflow",
  "queries": ["pandas dataframe"],
  "tagged": ["python", "pandas"],    // ANDed by the API
  "sort": "votes",                   // relevance|votes|activity|creation (search)
  "order": "desc",
  "acceptedOnly": false,
  "minScore": 5,
  "includeBody": false,
  "includeAnswers": false,
  "maxResultsPerQuery": 100,         // 0 = unlimited, but see the ceiling
  "pageSize": 100,
  "startPage": 1,                    // resume or shard a long walk
  "apiKey": ""                       // optional, free, raises the limits
}
```

### Output

One `SEARCH_SUMMARY` row per query, then the data rows.

| recordType | when |
|---|---|
| `SEARCH_SUMMARY` | always, one per query |
| `QUESTION` | search and questions modes |
| `ANSWER` | when `includeAnswers` is on |
| `TAG` | tags mode |
| `ERROR` | invalid input or an upstream failure — every input yields at least one row |

Every row carries `_input`, `_source`, `_scrapedAt` and `recordType`. Unix
epochs come with ISO copies beside them (`creationDate` / `creationDateIso`).

The summary reports `resultsReturned`, `answersReturned`, `requestsMade`,
`pagesFetched`, `upstreamTotal`, `pageCeilingHit`, `hasMoreAtStop`,
`reachableFraction`, `filterUsed`, `quotaRemaining` / `quotaMax`,
`backoffsHonoured`, `throttlesWaitedOut` and — in tags mode —
`upstreamPageWasResorted`.

### Limits worth knowing before you run it

- **300 requests per day per IP** without an app key; `quotaRemaining` is on
  every summary. A free key from stackapps.com raises it to 10,000 and lifts
  the page-25 ceiling. It is not a login and grants no private access.
- **Page 25 is the wall** without a key: at `pageSize: 100` that is 2,500
  questions per query, whatever `maxResultsPerQuery` says.
- **Pace yourself.** `minRequestInterval` defaults to 1 second and
  `maxConcurrency` to 2 for the edge throttle described above.
- The API is **public data under CC BY-SA**; each row carries its
  `contentLicense`.

### Notes

- **No WAF.** All six TLS profiles tried answered 200 cold, no warmup, no
  proxy. Responses are always gzipped. A proxy is available but off by
  default — note the daily quota is per IP, so a rotating proxy spreads it.
- `api.stackexchange.com` serves no robots.txt (the path answers HTTP 400),
  so RFC 9309's "unavailable" case applies.

# Actor input Schema

## `mode` (type: `string`):

search = full-text and tag search over questions. questions = questions by tag or by explicit question ID. tags = browse the site's tag list with usage counts.

## `site` (type: `string`):

Which site to query, as its API parameter: stackoverflow, serverfault, superuser, askubuntu, math, unix, and ~200 more. An unknown name is refused with 'No site found for name'.

## `queries` (type: `array`):

Free-text phrases for mode='search'. Each phrase runs as its own query and gets its own summary row. Combine with Tags to require both.

## `tagged` (type: `array`):

Restrict to questions carrying ALL of these tags (the API ANDs them). Works in search and questions modes; measured on Stack Overflow, tagged=pandas gives 288,853 questions and pandas+python gives 249,194.

## `questionIds` (type: `array`):

Fetch specific questions by numeric ID (mode='questions'). Sent in batches of 100 per request.

## `sort` (type: `string`):

search: relevance, votes, activity, creation. questions: votes, activity, creation, hot, week, month. tags: popular, activity, name. Anything else is refused with HTTP 400. NOTE: for tags the API returns each page alphabetised regardless — this actor re-sorts by count and keeps upstream's original position in `upstreamPageRank`.

## `order` (type: `string`):

desc (default) or asc.

## `acceptedOnly` (type: `boolean`):

Search mode only. Note this is stricter than `isAnswered`, which merely means the question has an upvoted answer.

## `minScore` (type: `integer`):

Drop questions scoring below this. A threshold nothing meets returns HTTP 200 with an empty list, not an error — the summary's resultsReturned is what tells you.

## `includeBody` (type: `boolean`):

The API's default filter omits `body` entirely, so bodies are null unless you turn this on. Rows also carry `bodyText`, the HTML stripped to readable text.

## `includeAnswers` (type: `boolean`):

Fetch the answers for every question returned, 100 question IDs per request. Adds ANSWER rows carrying `isAccepted` and the answer score.

## `maxResultsPerQuery` (type: `integer`):

Stop after this many questions per query. Set 0 for unlimited — but without an app key the API refuses page 26, so an unauthenticated run can never exceed 2,500 rows per query however high you set this. The summary reports `pageCeilingHit` and `reachableFraction`.

## `pageSize` (type: `integer`):

Between 1 and 100. Larger pages mean fewer requests against the daily quota, so 100 is the default.

## `startPage` (type: `integer`):

Begin the walk at this page instead of page 1 — useful for resuming or sharding a long run. Without an app key the API refuses any page above 25, so values above that are rejected up front.

## `apiKey` (type: `string`):

Optional and free — register an app at stackapps.com. It is not a login and grants no private access; it raises the daily quota from 300 to 10,000 requests and lifts the page-25 ceiling. Leave empty to run anonymously.

## `maxConcurrency` (type: `integer`):

Requests in flight at once across all queries. Kept low by default because the daily quota is shared across everything this IP does.

## `minRequestInterval` (type: `integer`):

Politeness pacing shared across all workers, in seconds. Beyond the daily quota Stack Exchange runs a separate per-IP edge throttle that answers HTTP 429 with an HTML page and a Retry-After measured in minutes (247 seconds when it was hit during development). Leave at 0 to use the built-in 1-second default; raise it if you see 429s.

## `proxyConfiguration` (type: `object`):

Optional and OFF by default. There is no WAF (6 of 6 TLS profiles answered 200 cold). Note the daily quota is counted per IP, so a rotating proxy spreads it across addresses.

## Actor input object example

```json
{
  "mode": "search",
  "site": "serverfault",
  "queries": [
    "memory leak",
    "async await"
  ],
  "tagged": [
    "python",
    "pandas"
  ],
  "questionIds": [
    "11227809",
    "509211"
  ],
  "sort": "votes",
  "order": "desc",
  "acceptedOnly": false,
  "minScore": 5,
  "includeBody": false,
  "includeAnswers": false,
  "maxResultsPerQuery": 100,
  "pageSize": 100,
  "startPage": 1,
  "maxConcurrency": 2,
  "minRequestInterval": 0,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `items` (type: `string`):

One row per scraped record. See the dataset's default view for field definitions.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "pandas dataframe"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapyx/stackexchange-qa-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "queries": ["pandas dataframe"] }

# Run the Actor and wait for it to finish
run = client.actor("scrapyx/stackexchange-qa-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "pandas dataframe"
  ]
}' |
apify call scrapyx/stackexchange-qa-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapyx/stackexchange-qa-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/CqNbrMeSUOW4HLu0F/builds/JiRnKFrMPxUTd6J11/openapi.json