# Stack Overflow Scraper: Questions, Answers, Users & Tags (`perconey/stackoverflow-scraper`) Actor Scrape any Stack Exchange site (stackoverflow, superuser, askubuntu, math.stackexchange and 170+ more) via the official Stack Exchange API. Questions, answers with full body, user profiles with reputation and badges, top tags, search. No auth, no proxies, no cookies. Pay only per result item. - **URL**: https://apify.com/perconey/stackoverflow-scraper.md - **Developed by:** [Perconey](https://apify.com/perconey) (community) - **Categories:** Developer tools, Lead generation, Other - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing $1.00 / 1,000 result items This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ### What does Stack Overflow Scraper do? **Stack Overflow Scraper** pulls structured data from any Stack Exchange site through the **official Stack Exchange API v2.3**. Get top questions, full search results, every answer to a single question, user profiles with reputation and badges, and trending tags. The actor calls the documented public API directly, so no browser, no proxies, no cookies, no anti-bot fight, ever. One actor covers **170+ Stack Exchange sites** under a single `site` parameter: stackoverflow, superuser, askubuntu, math.stackexchange, codereview.stackexchange, gaming.stackexchange, parenting.stackexchange, and many more. Try it instantly: pick **getQuestions**, leave site as `stackoverflow`, click Start. You get the top 100 highest-voted Stack Overflow questions of all time with full metadata, in under 10 seconds, for about $0.10. ### Why use Stack Overflow Scraper? - **Recruiters and sourcers**: Stack Overflow Talent shut down in 2024. Roll your own developer pipeline. Use `searchUsers` and `getUserAnswers` to find high-reputation engineers in a specific tag, then export contact via their linked profiles. - **DevRel and product teams**: Monitor questions tagged with your product (e.g. `tensorflow`, `langchain`, `kubernetes`) using `getQuestionsByTag`. Set up an Apify schedule to alert you on new high-vote questions about your SDK. - **Content marketers**: Use `searchQuestions` with `sort=hot` or `sort=week` to find trending questions worth writing about. Use `getTopTags` to discover where developer attention is shifting. - **Q&A dataset builders**: With `getQuestionDetail` + `includeAnswers: true` + `includeBody: true`, you get clean markdown Q&A pairs perfect for fine-tuning or RAG. Far cheaper than scraping Stack Overflow's own data dump (license tightened in 2024). - **Competitive intelligence**: How active is the community around a competitor's stack? Run `getQuestionsByTag` for their products and ours over the same date window. ### How to use Stack Overflow Scraper 1. Open the **Input** tab. 2. Pick an **action** from the dropdown. `getQuestions` is the simplest starting point. 3. Set **site** (default `stackoverflow`). To scrape a different Stack Exchange site, type its short name without `.com` (e.g. `superuser`, `askubuntu`). 4. For search/profile/tag/detail actions, fill **queries** (one entry per line). For top-questions and top-tags actions, leave queries empty. 5. Set **maxItems** to cap the run. Default 100. 6. (Optional) Paste a free **Stack Apps API key** to lift the quota from 300 to 10,000 requests per day. Register at https://stackapps.com/apps/oauth/register. 7. Click **Start**. Results stream to the dataset and you can preview them on the **Output** tab. ### Input Field | Required | Description --- | --- | --- `action` | yes | Which API call to make. See the dropdown for the eight options. `site` | yes | Stack Exchange site short name. Default `stackoverflow`. `queries` | sometimes | Required for search/detail/profile actions. Free text for searchQuestions/searchUsers; a tag for getQuestionsByTag; an id or full URL for getQuestionDetail, getUserProfile, getUserAnswers. `maxItems` | no | Max items per query. Default 100. `sort` | no | API sort key (e.g. `votes`, `activity`, `hot`, `week`, `reputation`). `order` | no | `desc` (default) or `asc`. `tagged` | no | For getQuestions / searchQuestions: limit to questions with this tag. `since` / `until` | no | ISO date filters for question-listing actions. `includeBody` | no | If true, also fetch full question/answer body as markdown. `includeAnswers` | no | For getQuestionDetail: also fetch all answers. Default true. `apiKey` | no | Stack Apps app key to lift the daily quota. ### Output Every dataset item carries a `_type` field (`question`, `answer`, `user`, `tag`, or `error`) plus `_action` and `_site` for filtering when one run mixes actions. Field names match the [Stack Exchange API types](https://api.stackexchange.com/docs/types), with Unix timestamps converted to ISO 8601. ```json { "_type": "question", "_action": "getQuestions", "_site": "stackoverflow", "question_id": 11227809, "title": "Why is processing a sorted array faster than processing an unsorted array?", "link": "https://stackoverflow.com/questions/11227809/...", "tags": ["java", "c++", "performance", "cpu-architecture", "branch-prediction"], "score": 27000, "view_count": 1900000, "answer_count": 27, "comment_count": 4, "is_answered": true, "accepted_answer_id": 11227902, "creation_date": "2012-06-27T13:51:36.000Z", "owner": { "user_id": 1539405, "display_name": "GManNickG", "reputation": 510000, "link": "https://stackoverflow.com/users/1539405/gmannickg" } } ```` You can download the dataset in JSON, CSV, XML, Excel, RSS or HTML format from the Output tab or the Apify API. ### Data fields Type | Key fields \--- | --- `question` | question\_id, title, link, tags, score, view\_count, answer\_count, is\_answered, accepted\_answer\_id, creation\_date, owner, body (optional) `answer` | answer\_id, question\_id, is\_accepted, score, link, creation\_date, owner, body (optional) `user` | user\_id, display\_name, reputation, link, location, website\_url, about\_me, badge\_counts, question\_count, answer\_count `tag` | name, count (total questions tagged), is\_required, has\_synonyms ### Pricing **Pay-per-result: $0.001 per item.** One question = one event. One answer = one event. One user profile = one event. No flat monthly fee, no rental, no charge for the time the actor runs (just Apify's default compute, ~$0.0002 per typical run at 512 MB). Cost examples: - Top 100 questions on a tag: **$0.10** - 500-user shortlist for a recruiting campaign (searchUsers + getUserAnswers, ~6 answers per user): **$3.50** - 1,000 Q\&A pairs for a fine-tuning dataset (getQuestionDetail with includeAnswers, ~5 answers per question): **$6.00** ### Tips - **Get a Stack Apps key** if you plan to run more than ~30 small runs per day from the same IP. It lifts the API quota from 300 to 10,000 requests/day. Free to register, no review process. - **Use `includeBody: false`** unless you actually need the markdown body. The response is ~10x smaller and faster. - **`getQuestionDetail` is the cheapest way to get Q\&A pairs** because one API call returns the question, and a second returns all answers paged. With `maxItems: 50` you cap at 50 answers per question. - **Cross-site research**: schedule the same input with different `site` values to compare communities (e.g. `react` tag on stackoverflow vs. softwareengineering.stackexchange). - **Date windows**: combine `tagged: tensorflow` with `since: 2025-01-01` to see what users asked since the GPT-5 release. ### FAQ, disclaimers, support **Is this legal?** The actor calls Stack Exchange's official public API (api.stackexchange.com) with documented endpoints. Public read access is explicitly permitted by Stack Exchange's API Terms of Service. We send a clear User-Agent string identifying the actor. Stack Exchange content is licensed CC BY-SA - attribution required when republishing. **Why the 300-request anonymous limit?** That's Stack Exchange's policy, not Apify's. Register a free app at https://stackapps.com/apps/oauth/register for 10,000/day per IP. **Will I get rate-limited?** The actor reads the API's `backoff` hint and sleeps automatically. We also retry on 429/502/503/504 with exponential backoff. Quota and backoff are logged for transparency. **Does it cover Reddit / Quora / GitHub Discussions?** No. This actor is only for Stack Exchange's 170+ sites. Each platform deserves its own actor for clean data shapes. **Bug or feature request?** Open an Issue on the actor's Issues tab. I usually respond within a day. **Need a custom scraper for another Q\&A platform?** Bluesky? Substack? Mastodon? See my other actors at https://apify.com/perconey, or open an Issue. # Actor input Schema ## `action` (type: `string`): Pick the type of data. getQuestions and getTopTags work with no queries. searchQuestions, getQuestionsByTag, getQuestionDetail, getUserProfile, getUserAnswers and searchUsers need at least one entry in queries. ## `site` (type: `string`): Which SE site to query. Defaults to stackoverflow. Full list at https://api.stackexchange.com/docs/sites. Common alternatives: superuser, askubuntu, math.stackexchange, gaming.stackexchange, parenting.stackexchange, codereview.stackexchange. ## `queries` (type: `array`): Depends on the action: searchQuestions/searchUsers - free text. getQuestionsByTag - a single tag like 'python' or 'react'. getQuestionDetail - question id (e.g. 11227809) or full URL. getUserProfile/getUserAnswers - user id (e.g. 22656 for Jon Skeet) or full profile URL. getQuestions and getTopTags can be left empty. ## `maxItems` (type: `integer`): Stop after this many items per query. Use 0 for unlimited (subject to API quota). ## `sort` (type: `string`): How to sort results. Valid values depend on the action - see https://api.stackexchange.com/docs. Common: 'votes' (default for getQuestions), 'activity', 'creation', 'hot', 'week', 'month', 'reputation' (default for searchUsers). ## `order` (type: `string`): asc or desc. Default desc. ## `tagged` (type: `string`): For getQuestions and searchQuestions: restrict to questions with this tag (e.g. 'python', 'react'). Multiple tags can be semi-colon separated. ## `since` (type: `string`): Only items on or after this date. Format 2024-01-01 or 2024-01-01T00:00:00Z. Applies to question-listing actions. ## `until` (type: `string`): Only items before this date. ## `includeBody` (type: `boolean`): If true, also returns the full question/answer body (markdown). Doubles response payload but lets you skip a follow-up call. ## `includeAnswers` (type: `boolean`): For getQuestionDetail: also fetch all answers (with body). Each answer counts as one more result-item. ## `apiKey` (type: `string`): Anonymous quota is 300 requests/day per IP. Register a free app at https://stackapps.com/apps/oauth/register to get a key and unlock 10,000/day per IP. ## Actor input object example ```json { "action": "getQuestions", "site": "stackoverflow", "queries": [], "maxItems": 100, "order": "desc", "includeBody": false, "includeAnswers": true } ``` # Actor output Schema ## `dataset` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "site": "stackoverflow", "queries": [] }; // Run the Actor and wait for it to finish const run = await client.actor("perconey/stackoverflow-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "site": "stackoverflow", "queries": [], } # Run the Actor and wait for it to finish run = client.actor("perconey/stackoverflow-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "site": "stackoverflow", "queries": [] }' | apify call perconey/stackoverflow-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=perconey/stackoverflow-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Stack Overflow Scraper: Questions, Answers, Users & Tags", "description": "Scrape any Stack Exchange site (stackoverflow, superuser, askubuntu, math.stackexchange and 170+ more) via the official Stack Exchange API. Questions, answers with full body, user profiles with reputation and badges, top tags, search. No auth, no proxies, no cookies. Pay only per result item.", "version": "0.1", "x-build-id": "6Lx6OcIlMOb9jSDiO" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/perconey~stackoverflow-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-perconey-stackoverflow-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/perconey~stackoverflow-scraper/runs": { "post": { "operationId": "runs-sync-perconey-stackoverflow-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/perconey~stackoverflow-scraper/run-sync": { "post": { "operationId": "run-sync-perconey-stackoverflow-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "action", "site" ], "properties": { "action": { "title": "What do you want to scrape?", "enum": [ "getQuestions", "searchQuestions", "getQuestionsByTag", "getQuestionDetail", "getUserProfile", "getUserAnswers", "searchUsers", "getTopTags" ], "type": "string", "description": "Pick the type of data. getQuestions and getTopTags work with no queries. searchQuestions, getQuestionsByTag, getQuestionDetail, getUserProfile, getUserAnswers and searchUsers need at least one entry in queries.", "default": "getQuestions" }, "site": { "title": "Stack Exchange site", "type": "string", "description": "Which SE site to query. Defaults to stackoverflow. Full list at https://api.stackexchange.com/docs/sites. Common alternatives: superuser, askubuntu, math.stackexchange, gaming.stackexchange, parenting.stackexchange, codereview.stackexchange.", "default": "stackoverflow" }, "queries": { "title": "Queries", "type": "array", "description": "Depends on the action: searchQuestions/searchUsers - free text. getQuestionsByTag - a single tag like 'python' or 'react'. getQuestionDetail - question id (e.g. 11227809) or full URL. getUserProfile/getUserAnswers - user id (e.g. 22656 for Jon Skeet) or full profile URL. getQuestions and getTopTags can be left empty.", "items": { "type": "string" } }, "maxItems": { "title": "Max items per query", "minimum": 0, "maximum": 100000, "type": "integer", "description": "Stop after this many items per query. Use 0 for unlimited (subject to API quota).", "default": 100 }, "sort": { "title": "Sort order", "type": "string", "description": "How to sort results. Valid values depend on the action - see https://api.stackexchange.com/docs. Common: 'votes' (default for getQuestions), 'activity', 'creation', 'hot', 'week', 'month', 'reputation' (default for searchUsers)." }, "order": { "title": "Order direction", "enum": [ "desc", "asc" ], "type": "string", "description": "asc or desc. Default desc.", "default": "desc" }, "tagged": { "title": "Tagged with", "type": "string", "description": "For getQuestions and searchQuestions: restrict to questions with this tag (e.g. 'python', 'react'). Multiple tags can be semi-colon separated." }, "since": { "title": "Since (ISO date)", "type": "string", "description": "Only items on or after this date. Format 2024-01-01 or 2024-01-01T00:00:00Z. Applies to question-listing actions." }, "until": { "title": "Until (ISO date)", "type": "string", "description": "Only items before this date." }, "includeBody": { "title": "Include full body markdown", "type": "boolean", "description": "If true, also returns the full question/answer body (markdown). Doubles response payload but lets you skip a follow-up call.", "default": false }, "includeAnswers": { "title": "Include answers (getQuestionDetail only)", "type": "boolean", "description": "For getQuestionDetail: also fetch all answers (with body). Each answer counts as one more result-item.", "default": true }, "apiKey": { "title": "Stack Apps API key (optional)", "type": "string", "description": "Anonymous quota is 300 requests/day per IP. Register a free app at https://stackapps.com/apps/oauth/register to get a key and unlock 10,000/day per IP." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```