# Stack Exchange Scraper (`crawlerbros/stack-exchange-scraper`) Actor Scrape questions, answers, users, and tags from Stack Overflow and 170+ Stack Exchange communities. HTTP-only via the public Stack Exchange API. No login, no proxy. - **URL**: https://apify.com/crawlerbros/stack-exchange-scraper.md - **Developed by:** [Crawler Bros](https://apify.com/crawlerbros) (community) - **Categories:** Developer tools, Automation, Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 20 bookmarks - **User rating**: 5.00 out of 5 stars ## Pricing from $1.00 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Stack Exchange Scraper Scrape questions, answers, users, and tags from Stack Overflow + 170+ Stack Exchange communities (Server Fault, Super User, Math, Cross Validated, Ask Ubuntu, Code Review, Software Engineering, AI, Data Science, Security, DBA, GIS, and more). HTTP-only via the official Stack Exchange API v2.3. No login, no cookies, no proxy. ### What this actor does - Five fetch modes: top questions, full-text search, by tag, by user, unanswered - Returns full HTML + Markdown bodies, scores, view counts, answer counts, tags, owner profile, accepted answer ID, and direct URLs - Optionally fetches all answers per question (sorted by votes) - Filters by score, answer count, "answered only", date range, and tag intersection - Honors Stack Exchange's quota and `backoff` headers; passes your free API key for 10k/day quota (vs. 300/day anonymous) - Sites covered: 170+ Stack Exchange communities (programming, math, science, language, hobbies, professional, life) ### Output per question - `questionId`, `title`, `link`, `score`, `viewCount`, `answerCount` - `body` (HTML) + `bodyMarkdown` (cleaned Markdown) — when `includeBody=true` - `tags[]`, `isAnswered`, `acceptedAnswerId` - `owner` — `{userId, displayName, reputation, profileUrl, profileImage}` - `createdAt`, `lastActivityAt`, `lastEditAt` - `answers[]` — when `includeAnswers=true`, each with `answerId`, `body`, `bodyMarkdown`, `score`, `isAccepted`, `owner`, `createdAt` - `recordType: "question"`, `site`, `scrapedAt` Mode `user` emits user records: `userId`, `displayName`, `profileUrl`, `profileImage`, `reputation`, `goldBadges`, `silverBadges`, `bronzeBadges`, `location`, `websiteUrl`, `aboutMe` (Markdown), `creationDate`, `lastAccessDate`, `recordType: "user"`. Empty fields are omitted (no nulls). ### Input | Field | Type | Default | Description | |---|---|---|---| | `site` | string | `stackoverflow` | Which Stack Exchange site (90+ enum) | | `mode` | string | `topQuestions` | `topQuestions` / `search` / `tag` / `user` / `unanswered` | | `searchQuery` | string | – | Required for `mode=search` | | `tagAnyOf` | array | `[]` | Tags to filter (required for `mode=tag`) | | `userIds` | array | `[]` | User IDs (required for `mode=user`) | | `sortBy` | string | `votes` | `activity` / `votes` / `creation` / `hot` / `week` / `month` | | `dateRangeFrom` | string | – | ISO date — drop questions older than this | | `dateRangeTo` | string | – | ISO date — drop questions newer than this | | `minScore` | int | – | Drop questions below this score | | `minAnswers` | int | – | Drop questions with fewer answers than this | | `isAnsweredOnly` | bool | `false` | Only emit questions with an accepted answer | | `includeAnswers` | bool | `false` | Also fetch answers per question (~1 extra call each) | | `includeBody` | bool | `true` | Include full HTML + Markdown body | | `maxItems` | int | `50` | Hard cap on emitted records (1–5000) | | `apiKey` | string | – | Optional free API key (10k/day vs 300/day anonymous) | #### Example: top Python questions on Stack Overflow ```json { "site": "stackoverflow", "mode": "topQuestions", "tagAnyOf": ["python"], "sortBy": "votes", "maxItems": 50 } ```` #### Example: search for "async/await" ```json { "site": "stackoverflow", "mode": "search", "searchQuery": "async await", "tagAnyOf": ["python"], "minScore": 10, "includeAnswers": true, "maxItems": 25 } ``` #### Example: unanswered questions for Devrel monitoring ```json { "site": "stackoverflow", "mode": "unanswered", "tagAnyOf": ["langchain"], "maxItems": 100 } ``` #### Example: a user's profile + their top questions ```json { "site": "stackoverflow", "mode": "user", "userIds": ["9285"], "maxItems": 1 } ``` #### Example: latest hot questions on Server Fault ```json { "site": "serverfault", "mode": "topQuestions", "sortBy": "hot", "dateRangeFrom": "2025-01-01", "maxItems": 100 } ``` ### Use cases - **Developer relations** — find unanswered questions about your library/tool/SDK to engage with the community - **Technical content marketing** — gap analysis on what topics get asked but aren't well-covered yet - **Recruiting** — find domain experts by tag + reputation (mode=user) - **Q\&A datasets for ML/RAG** — bulk-export curated answers for fine-tuning or retrieval indexes - **Community management** — monitor your tag for new questions - **Programming research** — analyze patterns in what developers struggle with - **Competitive intelligence** — track questions about competitor products - **Documentation prioritization** — high-view low-score questions reveal docs gaps ### FAQ **Does it require a login or cookies?** No. The Stack Exchange API is fully public. **Is a proxy needed?** No. Stack Exchange accepts requests from any IP. **What is the API quota?** 300 requests/day without an API key; 10,000/day with a free key. Register a key at https://stackapps.com/apps/oauth/register and pass it via `apiKey`. **Which sites are supported?** 170+ Stack Exchange communities. The `site` enum lists the most popular 90; you can also pass any other valid Stack Exchange site key (e.g. `cooking`, `worldbuilding`, `quant`). **Why are some fields empty?** The actor omits empty fields rather than emit nulls. For example, `aboutMe` is only present if a user filled it in. **Can I get the answer text?** Yes — set `includeAnswers: true`. Each question record then includes an `answers[]` array with body, score, owner, and accepted flag for every answer. **What's the difference between `score` and `viewCount`?** `score` is up-votes minus down-votes; `viewCount` is total page views. High-view low-score questions often signal missing canonical answers. **How do `tagAnyOf` filters interact across modes?** In `mode=tag` they're required and ANDed by the API. In `mode=topQuestions` and `mode=search` they're an optional filter. In `mode=user` they're ignored (user mode returns the user, not their questions). **Why HTML + Markdown bodies?** HTML is what Stack Exchange returns natively (preserves code blocks, tables, links). Markdown is a cleaned version optimized for LLM ingestion / display in dashboards. **How fresh is the data?** Real-time. Stack Exchange's API surfaces new questions within seconds of posting. **Can I scrape comments?** Comments are not part of v1. Use `mode=topQuestions` + `includeAnswers=true` to get the full Q\&A thread; comment support may be added in a future version. **What happens if I exceed the quota?** The actor watches the `quota_remaining` and `backoff` headers and waits gracefully. If the daily quota is exhausted, you'll get an empty result and a status message — pass an `apiKey` to raise the cap to 10k/day. # Actor input Schema ## `site` (type: `string`): Which Stack Exchange site to query. ## `mode` (type: `string`): What to fetch. `topQuestions` returns highest-scoring questions; `search` runs a full-text search; `tag` returns all questions tagged X; `user` returns a user's posts; `unanswered` returns questions with no accepted answer. ## `searchQuery` (type: `string`): Free-text query passed to Stack Exchange's title search. ## `tagAnyOf` (type: `array`): Tags to filter by. For mode=`tag`: required. For mode=`search` / `topQuestions`: optional refinement. ## `userIds` (type: `array`): Numeric Stack Exchange user IDs (e.g. `9285` for Jon Skeet on SO). ## `minScore` (type: `integer`): Drop questions with fewer up-votes than this. ## `minAnswers` (type: `integer`): Drop questions with fewer answers than this. ## `isAnsweredOnly` (type: `boolean`): Only emit questions that have an accepted answer. ## `dateRangeFrom` (type: `string`): Drop questions created before this date. ## `dateRangeTo` (type: `string`): Drop questions created after this date. ## `sortBy` (type: `string`): Stack Exchange's sort order. ## `includeAnswers` (type: `boolean`): Fetch the answers for each question. Adds ~1 extra API call per question (uses your quota faster). ## `includeBody` (type: `boolean`): Include the full HTML body and Markdown body of each question. ## `maxItems` (type: `integer`): Hard cap on emitted records. ## `apiKey` (type: `string`): Optional API key for the Stack Exchange API. Without it, you get 300 requests/day. With a free key (register at https://stackapps.com/apps/oauth/register), 10,000/day. ## Actor input object example ```json { "site": "stackoverflow", "mode": "topQuestions", "tagAnyOf": [], "userIds": [], "isAnsweredOnly": false, "sortBy": "votes", "includeAnswers": false, "includeBody": true, "maxItems": 50 } ``` # Actor output Schema ## `items` (type: `string`): Dataset containing all scraped Stack Exchange items (questions / answers / users / tags depending on mode). # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "site": "stackoverflow", "mode": "topQuestions", "tagAnyOf": [], "userIds": [], "isAnsweredOnly": false, "sortBy": "votes", "includeAnswers": false, "includeBody": true, "maxItems": 50 }; // Run the Actor and wait for it to finish const run = await client.actor("crawlerbros/stack-exchange-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "site": "stackoverflow", "mode": "topQuestions", "tagAnyOf": [], "userIds": [], "isAnsweredOnly": False, "sortBy": "votes", "includeAnswers": False, "includeBody": True, "maxItems": 50, } # Run the Actor and wait for it to finish run = client.actor("crawlerbros/stack-exchange-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "site": "stackoverflow", "mode": "topQuestions", "tagAnyOf": [], "userIds": [], "isAnsweredOnly": false, "sortBy": "votes", "includeAnswers": false, "includeBody": true, "maxItems": 50 }' | apify call crawlerbros/stack-exchange-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=crawlerbros/stack-exchange-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Stack Exchange Scraper", "description": "Scrape questions, answers, users, and tags from Stack Overflow and 170+ Stack Exchange communities. HTTP-only via the public Stack Exchange API. No login, no proxy.", "version": "1.0", "x-build-id": "VLtkFaPWxrJZDbZ3j" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/crawlerbros~stack-exchange-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-crawlerbros-stack-exchange-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/crawlerbros~stack-exchange-scraper/runs": { "post": { "operationId": "runs-sync-crawlerbros-stack-exchange-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/crawlerbros~stack-exchange-scraper/run-sync": { "post": { "operationId": "run-sync-crawlerbros-stack-exchange-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "site", "mode" ], "properties": { "site": { "title": "Stack Exchange site", "enum": [ "stackoverflow", "serverfault", "superuser", "askubuntu", "stats", "math", "mathoverflow", "tex", "english", "german", "spanish", "french", "physics", "chemistry", "biology", "astronomy", "ai", "datascience", "softwareengineering", "codereview", "security", "crypto", "networkengineering", "devops", "dba", "gis", "salesforce", "sharepoint", "wordpress", "drupal", "magento", "android", "apple", "ux", "webapps", "webmasters", "gaming", "rpg", "boardgames", "puzzling", "cooking", "diy", "homeimprovement", "gardening", "outdoors", "movies", "scifi", "anime", "music", "writers", "philosophy", "history", "skeptics", "law", "money", "personalfinance", "ell", "expressionengine", "ham", "blender", "graphicdesign", "photo", "video", "sound", "robotics", "iot", "electronics", "engineering", "academia", "matheducators", "tridion", "bitcoin", "ethereum", "monero", "elementaryos", "raspberrypi", "arduino", "unix", "emacs", "vi", "fitness", "interpersonal", "parenting", "pets", "judaism", "christianity", "buddhism", "islam", "hinduism" ], "type": "string", "description": "Which Stack Exchange site to query.", "default": "stackoverflow" }, "mode": { "title": "Mode", "enum": [ "topQuestions", "search", "tag", "user", "unanswered" ], "type": "string", "description": "What to fetch. `topQuestions` returns highest-scoring questions; `search` runs a full-text search; `tag` returns all questions tagged X; `user` returns a user's posts; `unanswered` returns questions with no accepted answer.", "default": "topQuestions" }, "searchQuery": { "title": "Search query (mode=search)", "type": "string", "description": "Free-text query passed to Stack Exchange's title search." }, "tagAnyOf": { "title": "Tags", "type": "array", "description": "Tags to filter by. For mode=`tag`: required. For mode=`search` / `topQuestions`: optional refinement.", "default": [], "items": { "type": "string" } }, "userIds": { "title": "User IDs (mode=user)", "type": "array", "description": "Numeric Stack Exchange user IDs (e.g. `9285` for Jon Skeet on SO).", "default": [], "items": { "type": "string" } }, "minScore": { "title": "Min question score", "minimum": 0, "maximum": 100000, "type": "integer", "description": "Drop questions with fewer up-votes than this." }, "minAnswers": { "title": "Min answer count", "minimum": 0, "maximum": 1000, "type": "integer", "description": "Drop questions with fewer answers than this." }, "isAnsweredOnly": { "title": "Answered only", "type": "boolean", "description": "Only emit questions that have an accepted answer.", "default": false }, "dateRangeFrom": { "title": "Date range from (YYYY-MM-DD)", "type": "string", "description": "Drop questions created before this date." }, "dateRangeTo": { "title": "Date range to (YYYY-MM-DD)", "type": "string", "description": "Drop questions created after this date." }, "sortBy": { "title": "Sort by", "enum": [ "activity", "votes", "creation", "hot", "week", "month" ], "type": "string", "description": "Stack Exchange's sort order.", "default": "votes" }, "includeAnswers": { "title": "Include answers", "type": "boolean", "description": "Fetch the answers for each question. Adds ~1 extra API call per question (uses your quota faster).", "default": false }, "includeBody": { "title": "Include question body", "type": "boolean", "description": "Include the full HTML body and Markdown body of each question.", "default": true }, "maxItems": { "title": "Max items", "minimum": 1, "maximum": 5000, "type": "integer", "description": "Hard cap on emitted records.", "default": 50 }, "apiKey": { "title": "Stack Exchange API key (optional)", "type": "string", "description": "Optional API key for the Stack Exchange API. Without it, you get 300 requests/day. With a free key (register at https://stackapps.com/apps/oauth/register), 10,000/day." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```