# Stack Exchange Q\&A Scraper (`crawlerbros/stack-exchange-qa-scraper`) Actor Scrape questions, answers, and site listings from Stack Overflow and 170+ Stack Exchange communities via the official Stack Exchange API v2.3. No login, no cookies, no proxy needed. - **URL**: https://apify.com/crawlerbros/stack-exchange-qa-scraper.md - **Developed by:** [Crawler Bros](https://apify.com/crawlerbros) (community) - **Categories:** Developer tools, Automation, Integrations - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $3.00 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Stack Exchange Q&A Scraper Scrape questions, answers, and site listings from **Stack Overflow** and **170+ Stack Exchange communities** (Server Fault, Super User, Math, Cross Validated, Ask Ubuntu, Code Review, Software Engineering, AI, Data Science, Security, DBA, GIS, and more) via the **official Stack Exchange API v2.3**. No login, no cookies, no proxy required. ### What this actor does - **Four fetch modes** — search questions by keyword, browse by tag, fetch answers for specific question IDs, or list all Stack Exchange network sites - Returns full **question metadata** — title, body excerpt, tags, score, view count, answer count, accepted answer ID, author, and direct URLs - **Answers mode** — fetch all answers for a list of question IDs with body, score, author, and accepted flag - **Sites mode** — list all 170+ Stack Exchange network sites with name, audience, type, icon, and launch date - Honors Stack Exchange's **quota and backoff headers** automatically - Optional **API key** for 10,000 requests/day (vs. 300/day anonymous) - **No proxy, no cookies** — Stack Exchange is publicly accessible ### Output #### Questions (mode=searchQuestions or getQuestionsByTag) | Field | Type | Description | |---|---|---| | `questionId` | integer | Stack Exchange question ID | | `title` | string | Question title | | `body` | string | Plain-text excerpt of the question body (up to 500 chars) | | `tags` | array | Question tags (e.g. `["python", "asyncio"]`) | | `score` | integer | Up-votes minus down-votes | | `viewCount` | integer | Total page views | | `answerCount` | integer | Number of posted answers | | `acceptedAnswerId` | integer | ID of the accepted answer (if any) | | `isAnswered` | boolean | Whether the question has an accepted answer | | `author` | string | Question author's display name | | `authorReputation` | integer | Author's Stack Exchange reputation | | `createdAt` | string | ISO-8601 creation timestamp (UTC) | | `lastActivityAt` | string | ISO-8601 last activity timestamp (UTC) | | `questionUrl` | string | Direct URL to the question | | `site` | string | Stack Exchange site slug (e.g. `stackoverflow`) | | `scrapedAt` | string | ISO-8601 scrape timestamp (UTC) | #### Answers (mode=getAnswers) | Field | Type | Description | |---|---|---| | `answerId` | integer | Stack Exchange answer ID | | `questionId` | integer | Parent question ID | | `body` | string | Plain-text excerpt of the answer body (up to 500 chars) | | `score` | integer | Answer score | | `isAccepted` | boolean | Whether this answer was accepted | | `author` | string | Answer author's display name | | `authorReputation` | integer | Author's reputation | | `createdAt` | string | ISO-8601 creation timestamp (UTC) | | `scrapedAt` | string | ISO-8601 scrape timestamp (UTC) | #### Sites (mode=listSites) | Field | Type | Description | |---|---|---| | `siteUrl` | string | Base URL of the community (e.g. `https://stackoverflow.com`) | | `siteName` | string | Community name (e.g. `Stack Overflow`) | | `audience` | string | Target audience description | | `siteType` | string | Site type (`main_site`, `meta_site`, etc.) | | `iconUrl` | string | Community icon URL | | `launchDate` | string | ISO-8601 launch date (UTC) | | `scrapedAt` | string | ISO-8601 scrape timestamp (UTC) | Empty fields are always omitted — no nulls in output. ### Input | Field | Type | Default | Description | |---|---|---|---| | `mode` | string | `searchQuestions` | `searchQuestions` / `getQuestionsByTag` / `getAnswers` / `listSites` | | `query` | string | – | Search keyword (required for `mode=searchQuestions`) | | `tags` | array | `[]` | Tags to filter (required for `mode=getQuestionsByTag`) | | `questionIds` | array | `[]` | Question IDs to fetch answers for (required for `mode=getAnswers`) | | `site` | string | `stackoverflow` | Stack Exchange site slug | | `sortBy` | string | `votes` | `votes` / `activity` / `creation` / `relevance` | | `apiKey` | string | – | Optional API key (10k/day vs 300/day anonymous) | | `maxItems` | integer | `25` | Maximum records to return (1–200) | ### Example inputs #### Search for Python questions on Stack Overflow ```json { "mode": "searchQuestions", "query": "python", "site": "stackoverflow", "sortBy": "votes", "maxItems": 25 } ```` #### Browse JavaScript questions by tag ```json { "mode": "getQuestionsByTag", "tags": ["javascript"], "site": "stackoverflow", "sortBy": "votes", "maxItems": 50 } ``` #### Fetch answers for specific questions ```json { "mode": "getAnswers", "questionIds": ["11227809", "231767"], "site": "stackoverflow", "maxItems": 50 } ``` #### List all Stack Exchange network sites ```json { "mode": "listSites", "maxItems": 200 } ``` ### Use cases - **Developer relations** — monitor your library/SDK tag for new unanswered questions - **Technical content marketing** — find gaps in documentation by analyzing high-view low-score questions - **Q\&A datasets for ML/RAG** — export curated answers for fine-tuning or retrieval-augmented generation - **Recruiting** — identify domain experts by reputation and tag activity - **Community analytics** — analyze trends in what developers struggle with - **Competitive intelligence** — track questions about competitor products - **Documentation prioritization** — high-view questions reveal documentation gaps ### FAQ **Does it require a login or cookies?** No. The Stack Exchange API is fully public and returns real data without authentication. **Is a proxy needed?** No. Stack Exchange accepts requests from any IP address. **What is the API quota?** 300 requests/day without an API key; 10,000/day with a free key. Register at https://stackapps.com/apps/oauth/register and pass it via `apiKey`. **Which sites are supported?** All 170+ Stack Exchange communities. Pass any valid site slug to the `site` field (e.g. `stackoverflow`, `serverfault`, `math`, `cooking`, `worldbuilding`). Use `mode=listSites` to enumerate all available sites. **Why are some fields missing?** Fields are omitted when the API returns no data for them (e.g. `acceptedAnswerId` only appears when a question has an accepted answer, `authorReputation` only appears when the author profile is available). **What does `body` contain?** The `body` field is a plain-text excerpt (up to 500 characters) extracted from the question or answer's HTML body. HTML tags are stripped, common entities are decoded, and the text is trimmed. **How does `sortBy=relevance` work?** `relevance` is only valid for `mode=searchQuestions` — it ranks results by how closely they match the search query. For other modes, `votes` is used. **What happens if I exceed the quota?** The actor watches the `quota_remaining` and `backoff` headers and sleeps gracefully when asked. Once the daily quota is exhausted, you will get fewer results — pass an `apiKey` to raise the cap to 10,000/day. **Can I scrape multiple tags at once?** Yes — pass multiple values in the `tags` array. Stack Exchange treats them as an AND filter (questions must have all specified tags). **Is the data real-time?** Yes. Stack Exchange's API surfaces new questions within seconds of posting. # Actor input Schema ## `mode` (type: `string`): What to fetch. `searchQuestions`: search Q\&A by keyword. `getQuestionsByTag`: browse questions by tag. `getAnswers`: fetch answers for specific question IDs. `listSites`: list all Stack Exchange network sites. ## `query` (type: `string`): Keyword to search for in question titles. Required for mode=searchQuestions. ## `tags` (type: `array`): Tags to filter by. Required for mode=getQuestionsByTag (e.g. `javascript`, `python`, `react`). ## `questionIds` (type: `array`): Numeric Stack Exchange question IDs to fetch answers for. Required for mode=getAnswers. ## `site` (type: `string`): Which Stack Exchange community to query. ## `sortBy` (type: `string`): How to order results. `votes` = highest score first; `activity` = recently active; `creation` = newest; `relevance` = search relevance (searchQuestions only). ## `apiKey` (type: `string`): Optional API key from https://stackapps.com/apps/oauth/register. Without key: 300 requests/day. With free key: 10,000/day. ## `maxItems` (type: `integer`): Maximum number of records to return (1–200). ## Actor input object example ```json { "mode": "searchQuestions", "query": "python tutorial", "tags": [], "questionIds": [], "site": "stackoverflow", "sortBy": "votes", "maxItems": 20 } ``` # Actor output Schema ## `items` (type: `string`): Dataset containing all scraped records. Questions include: questionId, title, body, tags, score, viewCount, answerCount, acceptedAnswerId, isAnswered, author, authorReputation, createdAt, lastActivityAt, questionUrl, site, scrapedAt. Answers include: answerId, questionId, body, score, isAccepted, author, authorReputation, createdAt, scrapedAt. Sites include: siteUrl, siteName, audience, siteType, iconUrl, launchDate, scrapedAt. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "mode": "searchQuestions", "query": "python tutorial", "tags": [], "questionIds": [], "site": "stackoverflow", "sortBy": "votes", "maxItems": 20 }; // Run the Actor and wait for it to finish const run = await client.actor("crawlerbros/stack-exchange-qa-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "mode": "searchQuestions", "query": "python tutorial", "tags": [], "questionIds": [], "site": "stackoverflow", "sortBy": "votes", "maxItems": 20, } # Run the Actor and wait for it to finish run = client.actor("crawlerbros/stack-exchange-qa-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "mode": "searchQuestions", "query": "python tutorial", "tags": [], "questionIds": [], "site": "stackoverflow", "sortBy": "votes", "maxItems": 20 }' | apify call crawlerbros/stack-exchange-qa-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=crawlerbros/stack-exchange-qa-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Stack Exchange Q&A Scraper", "description": "Scrape questions, answers, and site listings from Stack Overflow and 170+ Stack Exchange communities via the official Stack Exchange API v2.3. No login, no cookies, no proxy needed.", "version": "1.0", "x-build-id": "KC3LWf5oDuUAOFFkI" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/crawlerbros~stack-exchange-qa-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-crawlerbros-stack-exchange-qa-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/crawlerbros~stack-exchange-qa-scraper/runs": { "post": { "operationId": "runs-sync-crawlerbros-stack-exchange-qa-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/crawlerbros~stack-exchange-qa-scraper/run-sync": { "post": { "operationId": "run-sync-crawlerbros-stack-exchange-qa-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "mode" ], "properties": { "mode": { "title": "Mode", "enum": [ "searchQuestions", "getQuestionsByTag", "getAnswers", "listSites" ], "type": "string", "description": "What to fetch. `searchQuestions`: search Q&A by keyword. `getQuestionsByTag`: browse questions by tag. `getAnswers`: fetch answers for specific question IDs. `listSites`: list all Stack Exchange network sites.", "default": "searchQuestions" }, "query": { "title": "Search query (mode=searchQuestions)", "type": "string", "description": "Keyword to search for in question titles. Required for mode=searchQuestions." }, "tags": { "title": "Tags (mode=getQuestionsByTag)", "type": "array", "description": "Tags to filter by. Required for mode=getQuestionsByTag (e.g. `javascript`, `python`, `react`).", "default": [], "items": { "type": "string" } }, "questionIds": { "title": "Question IDs (mode=getAnswers)", "type": "array", "description": "Numeric Stack Exchange question IDs to fetch answers for. Required for mode=getAnswers.", "default": [], "items": { "type": "string" } }, "site": { "title": "Stack Exchange site", "enum": [ "stackoverflow", "serverfault", "superuser", "askubuntu", "math", "physics", "chemistry", "biology", "datascience", "softwareengineering", "security", "dba", "ai", "stats", "electronics" ], "type": "string", "description": "Which Stack Exchange community to query.", "default": "stackoverflow" }, "sortBy": { "title": "Sort by", "enum": [ "votes", "activity", "creation", "relevance" ], "type": "string", "description": "How to order results. `votes` = highest score first; `activity` = recently active; `creation` = newest; `relevance` = search relevance (searchQuestions only).", "default": "votes" }, "apiKey": { "title": "Stack Exchange API key (optional)", "type": "string", "description": "Optional API key from https://stackapps.com/apps/oauth/register. Without key: 300 requests/day. With free key: 10,000/day." }, "maxItems": { "title": "Max items", "minimum": 1, "maximum": 200, "type": "integer", "description": "Maximum number of records to return (1–200).", "default": 50 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```