# Stack Overflow Scraper (`real-estate-api/stack-overflow-scraper`) Actor

Stack Overflow Scraper is a Python tool designed to extract Q\&A data, accepted answers, vote counts, code snippets, and user profiles from Stack Overflow. It exports structured developer knowledge into clean CSV or JSON formats for research and AI model training.

- **URL**: https://apify.com/real-estate-api/stack-overflow-scraper.md
- **Developed by:** [real-estate-api](https://apify.com/real-estate-api) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.99 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Stack Overflow Scraper: Export Questions, Answers, Tags and Users

<p align="center">
  <img src="https://imgur.com/yOwBoZT.png" alt="Stack Overflow Scraper" width="800" />
</p>

Stack Overflow Scraper collects developer-community data through the official Stack Exchange API 2.3. Retrieve questions, answers, tags, users, and public Stack Exchange sites, then export structured datasets in JSON, CSV, or Excel format.

Search by keyword, tag, title, answer status, or date range. Collect question bodies, scores, view counts, accepted-answer information, and user details without logging in.

When redistributing post content, preserve the required author attribution, source link, and applicable CC BY-SA license information.

### Features

- **Question collection:** Retrieve questions by tag, sorting method, and date range.
- **Advanced search:** Search by keywords, title, tags, accepted-answer status, and minimum answer count.
- **Answer extraction:** Collect answers for specified question IDs or URLs.
- **Tag statistics:** Retrieve tag names and associated question counts.
- **User information:** Collect user profiles, reputation, and badge counts.
- **Site discovery:** List public sites in the Stack Exchange network.
- **Body formats:** Store post bodies in Markdown, plain text, or HTML.
- **Pagination:** Follow API pagination metadata to collect additional results.
- **Rate-limit handling:** Respect API backoff instructions and monitor remaining quota.
- **Error recovery:** Handle invalid parameters, throttling, timeouts, and individual failed requests.
- **Duplicate removal:** Avoid duplicate question, answer, user, tag, and site IDs.
- **Structured exports:** Prepare collected records for analysis and downstream applications.

### Use Cases

- **Developer-tool research:** Analyze common technical problems using question tags, scores, and view counts.
- **AI and RAG datasets:** Prepare question-and-answer collections while preserving attribution and license information.
- **Content research and SEO:** Identify recurring developer questions and popular technical topics.
- **Developer relations:** Discover active contributors and top answerers in specific technology areas.
- **Technology trend analysis:** Compare tag counts and community activity across Stack Overflow, Super User, Server Fault, and other supported sites.

### Available Modes

| Mode | API Resource | Description |
|---|---|---|
| `questions` | `/questions` | Retrieve questions by tag, sorting method, and date range. |
| `search` | `/search/advanced` | Search questions by keyword, title, tags, accepted-answer status, and minimum answer count. |
| `answers` | `/questions/{ids}/answers` | Retrieve answers for specified question IDs or URLs, using batches of up to 100 IDs. |
| `tags` | `/tags` | Retrieve tag names and question counts. |
| `users` | `/users` or `/users/{ids}` | Search users by name or retrieve specific user profiles. |
| `sites` | `/sites` | Retrieve public Stack Exchange sites. |

### Input Parameters

| Parameter | Type | Default | Description |
|---|---|---|---|
| `mode` | String | `search` | Collection mode: `questions`, `search`, `answers`, `tags`, `users`, or `sites`. |
| `site` | String | `stackoverflow` | Stack Exchange site key, such as `stackoverflow`, `superuser`, `serverfault`, `askubuntu`, `math`, or `datascience`. |
| `query` | String | `web scraping` | Keyword used for question searches. |
| `tags` | String Array | `[]` | Tags to filter by. Every supplied tag must match. |
| `titleContains` | String | Empty | Filter search results by title text. |
| `acceptedOnly` | Boolean | `false` | Return only questions with an accepted answer in search mode. |
| `minAnswers` | Integer | Empty | Minimum number of answers in search mode. |
| `sort` | String | `activity` | Sorting method. Options depend on the selected mode; examples include `relevance` for search, `popular` for tags, and `reputation` for users. |
| `fromDate` | String | Empty | Earliest question date. |
| `toDate` | String | Empty | Latest question date. |
| `questionIds` | String Array | `[]` | Question IDs or URLs for answers mode. |
| `userIds` | String Array | `[]` | User IDs or profile URLs for users mode. |
| `nameContains` | String | Empty | Partial tag or user name for supported modes. |
| `includeBody` | Boolean | `true` | Include post bodies in the output. |
| `includeAnswers` | Boolean | `false` | Include nested answers with each question. Answers are fetched in batches of up to 100 IDs. |
| `bodyFormat` | String | `markdown` | Post body format: `markdown`, `text`, or `html`. |
| `maxItems` | Integer | `200` | Maximum number of rows to save. Maximum: 10,000. |
| `timeoutSecs` | Integer | `300` | Overall execution time limit in seconds. Maximum: 3,600. |
| `requestTimeoutSecs` | Integer | `30` | Timeout for an individual API request, in seconds. |
| `apiKey` | String | Empty | Optional Stack Apps API key. Can increase the daily request quota. |

### Example Input

```json id="f7q2nk"
{
  "mode": "search",
  "site": "stackoverflow",
  "query": "web scraping",
  "tags": ["python"],
  "minAnswers": 1,
  "includeBody": true,
  "bodyFormat": "markdown",
  "maxItems": 50
}
```

### Output

Each collected item is saved as a structured dataset record. Output fields depend on the selected mode and the information return.

#### Example Question Record

```json id="v4j8rp"
{
  "site": "stackoverflow",
  "questionId": 80008831,
  "questionTitle": "Merge two Celery tasks",
  "title": "Merge two Celery tasks",
  "url": "STACK_OVERFLOW_QUESTION_URL",
  "tags": ["python", "web-scraping", "celery"],
  "score": -5,
  "viewCount": 139,
  "answerCount": 0,
  "isAnswered": false,
  "createdAt": "2026-10-08T12:00:00Z",
  "ownerName": "Sandi",
  "ownerReputation": 188,
  "license": "CC BY-SA 4.0"
}
```

#### Output Fields

| Field | Type | Description |
|---|---|---|
| `questionTitle` | String | Decoded question title. Also available as `title` for compatibility with existing exports. |
| `title` | String | Question title duplicated for compatibility. |
| `url` | String | Link to a question, answer, or tag. |
| `body` | String | Post body in the selected format. May be null when body collection is disabled. |
| `answers` | Array | Nested answers when answer collection is enabled. |
| `ownerName` | String | Display name of the post owner. May be empty for deleted accounts. |
| `license` | String | License information returned by the API. |
| `badges` | Object | Gold, silver, and bronze badge counts in user records. |
| `apiSiteParameter` | String | Site identifier returned by the sites collection. |
| `error` | String | Error code or description, such as `QUOTA_LOW`, `TIMEOUT`, `INVALID_INPUT`, `bad_parameter`, or `throttle_violation`. |

Additional fields vary by mode. Question records can include scores, view counts, tags, answer counts, accepted-answer information, creation dates, and owner details.

A key-value store record named `STATS` contains quota and request statistics, including `quotaMax`, `quotaRemaining`, and `requests`.

**Duplicate handling:** Duplicate question, answer, user, tag, and site IDs are removed. A question with no answers is treated as a successful empty result rather than an error.

### How It Works

1. Reads the selected mode, site, filters, and collection limits.
2. Sends requests to the relevant Stack Exchange API 2.3 resource.
3. Processes result pages containing up to 100 items each.
4. Follows the API's `has_more` pagination information when additional results are available.
5. Waits for the specified backoff period before making the next call to the same method.
6. Retries supported throttling responses, including HTTP 429, HTTP 502, or API error ID 502, once after waiting.
7. Records invalid parameters and other request failures without terminating the entire run.
8. Stops when the requested item limit, runtime limit, or available API quota prevents further collection.

When fewer than five API requests remain, the scraper records a `QUOTA_LOW` warning rather than exhausting the remaining quota.

### Limits and API Quotas

- Each API response returns up to 100 items.
- The maximum collection limit is 10,000 rows per run.
- API backoff instructions are respected before subsequent requests to the same method.
- Request timeouts and overall execution limits can be configured.

Actual quotas and limits may vary according to the API's current policies and usage conditions.

### Licensing and Attribution

Stack Exchange post content is provided under the applicable Creative Commons Attribution-ShareAlike license, as indicated by the API and the relevant post.

When republishing question or answer bodies, preserve the required attribution information, including the author's name, a link to the original post, and the applicable license. Review the current Stack Exchange licensing guidance before redistributing collected content.

### FAQ

**Can I collect data from Super User or Server Fault?**

Yes. Set `site` to the relevant API site parameter, such as `superuser` or `serverfault`. Site-specific links use the corresponding site's domain.

**Are question bodies returned as HTML?**

The default format is Markdown. Set `bodyFormat` to `text` or `html` to request another supported format. Question titles are decoded before being saved.

**Can I retrieve answers for a specific question?**

Yes. Select `answers` mode and provide the question ID or full question URL through `questionIds`. A question with no answers returns no answer records.

**Can I collect only questions with accepted answers?**

Yes. In search mode, set `acceptedOnly` to `true`.

**What happens when one site or ID fails?**

The scraper records an appropriate error and continues processing other inputs when possible.

### Related Tools

Related data-collection workflows may include Hacker News, GitHub, npm, and PyPI data collection. These can complement Stack Exchange datasets for developer-community research and technical trend analysis.

### Trademark Notice

Stack Overflow, Stack Exchange, and their related names, logos, and trademarks belong to their respective owners. Their use in this documentation is solely to identify the data source and does not imply affiliation, sponsorship, endorsement, or partnership.

### Disclaimer

This is an independent data-collection tool and is not affiliated with, endorsed by, sponsored by, or officially associated with Stack Overflow or Stack Exchange. It accesses publicly available information through the official API. Users are responsible for complying with the API terms, usage limits, attribution obligations, and applicable licensing requirements.

### Is Web Scraping Legal?

This tool uses the public Stack Exchange without logging in or bypassing access controls. Whether collecting, storing, or redistributing data is permitted depends on the information involved, applicable laws, API terms, licensing conditions, and intended use.

- **Public information:** Public availability does not automatically grant unrestricted permission to reuse every type of content.
- **Personal data:** Privacy laws may apply when collected information identifies individuals or includes other personal information.
- **Copyright and licensing:** Post bodies and other creative content may be protected by copyright or governed by specific license terms. Preserve required attribution and follow the applicable license.
- **Terms of use:** Respect API usage rules, rate limits, and applicable source policies.
- **Legal advice:** This section provides general information, not legal advice. Consult a qualified legal professional if you need guidance about a specific use case.

# Actor input Schema

## `mode` (type: `string`):

Questions by tag, advanced question search, answers for question ids, tags, users, or the list of Stack Exchange sites.

## `site` (type: `string`):

api_site_parameter, e.g. stackoverflow, superuser, serverfault, askubuntu, math, datascience. Ignored for the sites list.

## `query` (type: `string`):

Keyword query for search mode. Sent as q to /search/advanced.

## `tags` (type: `array`):

Tags that must all match. Used as tagged on questions and search. One tag per line.

## `titleContains` (type: `string`):

Search mode only. Keep questions whose title contains this text.

## `acceptedOnly` (type: `boolean`):

Search mode only. When checked, only questions with an accepted answer are returned.

## `minAnswers` (type: `integer`):

Search mode only. Minimum number of answers a question must have.

## `sort` (type: `string`):

relevance: search only; popular: tags; reputation: users. A sort the endpoint does not accept is replaced with that mode's usual sort.

## `fromDate` (type: `string`):

Oldest creation date for questions and search, inclusive. Sent as a Unix timestamp.

## `toDate` (type: `string`):

Newest creation date for questions and search, inclusive. Sent as a Unix timestamp.

## `questionIds` (type: `array`):

Answers mode. Numeric question ids or stackoverflow.com/questions/ID URLs. Up to 100 ids are sent per request.

## `userIds` (type: `array`):

Users mode. Numeric user ids or /users/ID profile URLs. When this list is set, name search is skipped.

## `nameContains` (type: `string`):

Partial name for tags (inname) or users (inname). Leave empty to list by the selected sort.

## `includeBody` (type: `boolean`):

Request filter=withbody and store the question or answer body. Turn off to keep rows smaller.

## `includeAnswers` (type: `boolean`):

Questions and search only. Batches up to 100 question ids per /questions/{ids}/answers call and nests the answers on each question.

## `bodyFormat` (type: `string`):

How to store the body: Markdown, plain text, or the original HTML.

## `maxItems` (type: `integer`):

Free cap 10,000. Without an API key Stack Exchange allows about 300 requests/day per IP (100 items each).

## `timeoutSecs` (type: `integer`):

Stop the run after this many seconds and write a timeout error row. Default 300. Maximum 3600.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for one Stack Exchange response. Default 30 seconds.

## `apiKey` (type: `string`):

Optional Stack Apps key. It is sent only on the request and is never written to the dataset or the repo. A key raises the daily quota from about 300 requests per IP to 10,000.

## `proxyConfiguration` (type: `object`):

Optional proxy for Stack Exchange requests. Leave this off unless an IP has used its daily quota. The API allows calls with no proxy.

## Actor input object example

```json
{
  "mode": "search",
  "site": "stackoverflow",
  "query": "web scraping",
  "tags": [
    "web-scraping"
  ],
  "titleContains": "playwright",
  "acceptedOnly": false,
  "sort": "activity",
  "questionIds": [
    "11227809",
    "https://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-processing-an-unsorted-array"
  ],
  "userIds": [
    "22656",
    "https://stackoverflow.com/users/22656/jon-skeet"
  ],
  "nameContains": "scrap",
  "includeBody": true,
  "includeAnswers": false,
  "bodyFormat": "markdown",
  "maxItems": 200,
  "timeoutSecs": 300,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `questions` (type: `string`):

Question title, site, score, views, answer count, and URL.

## `answers` (type: `string`):

Author, score, accepted flag, question id, and answer URL.

## `users` (type: `string`):

Display name, reputation, location, and profile URL.

## `tags` (type: `string`):

Tag name, site, question count, and tag URL.

## `errors` (type: `string`):

Quota, timeout, bad parameter, and input errors.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "query": "web scraping",
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("real-estate-api/stack-overflow-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "query": "web scraping",
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("real-estate-api/stack-overflow-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "query": "web scraping",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call real-estate-api/stack-overflow-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,real-estate-api/stack-overflow-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/JG2RlulhsAG0lcova/builds/xNhB6QGREFZvubREn/openapi.json