# Stack Overflow Scraper: Questions, Answers & Tags (`scrapemint/stackoverflow-scraper`) Actor Search Stack Overflow (or any Stack Exchange site) by keyword or tag and get one row per question: title, body, tags, score, views, answer count, accepted answer and author, with top answers optionally attached. For dev tool marketing, research and support teams. No API key needed. - **URL**: https://apify.com/scrapemint/stackoverflow-scraper.md - **Developed by:** [Ken M](https://apify.com/scrapemint) (community) - **Categories:** Developer tools, Business - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Stack Overflow Scraper: Questions, Answers & Tags Search Stack Overflow by **keyword** or sweep a **tag**, and get one clean row per question: title, body text, tags, score, view count, answer count, accepted answer, author and reputation — with the **top answers optionally attached**. Works on any Stack Exchange site (Super User, Server Fault, Ask Ubuntu and 170+ more) with one input switch. Built for **developer tool marketing (find the questions your product answers), AI and research datasets, support teams and technical content writers**. No login, no browser, and no API key needed to start. ### What you get for each question - **title**, **bodyText** (clean text, no HTML) and **url** - **tags**, **score**, **views**, **answerCount**, **isAnswered**, **acceptedAnswerId** - **author**, **authorReputation** and profile link - **createdAt** and **lastActivityAt**: freshness at a glance - **topAnswers** (optional): up to 5 best answers with score, accepted flag, text and author ### Example output ```json { "input": "playwright timeout", "mode": "search", "questionId": 70714523, "title": "Playwright: Timeout 30000ms exceeded waiting for selector", "tags": ["playwright", "node.js"], "score": 42, "views": 118234, "answerCount": 6, "isAnswered": true, "author": "jsmith", "createdAt": "2022-01-14T09:31:00.000Z", "url": "https://stackoverflow.com/questions/70714523/..." } ```` ### Searches or tags? - **Searches**: free text, ranked by relevance ("memory leak nodejs", "pandas merge slow"). - **Tags**: a tag's top questions by votes ("web-scraping", "playwright"). Great for mapping a topic. - **newerThanDays** filters to recent questions only, for trend and demand research. - **site** switches to any Stack Exchange community: `superuser`, `serverfault`, `askubuntu`, `softwareengineering`... ### Pricing **$0.003 per question row**, answers included at no extra charge. Searches and tags that match nothing are **free**, and the first 2 rows of every run are free. Mapping 1,000 questions in your product's niche costs $3. ### About the shared API quota The public Stack Exchange API allows 300 requests/day per IP without a key, and one request fetches up to 100 questions, so normal runs use a handful of requests. On busy shared infrastructure the quota can run out; the actor then stops cleanly, keeps everything fetched so far, and never charges for what it could not get. Adding your own free key from [stackapps.com](https://stackapps.com) in the `apiKey` field raises the quota to 10,000/day — your key stays secret and is only sent to Stack Exchange. ### How to run it via API ```bash curl -X POST "https://api.apify.com/v2/acts/scrapemint~stackoverflow-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"queries": ["playwright timeout"], "tags": ["web-scraping"], "maxPerQuery": 25, "includeAnswers": true}' ``` ### Frequently asked questions **Where does the data come from?** The official public Stack Exchange API. Content is user contributed under CC BY-SA; rows link back to the source question as attribution. **Can I get full answer bodies?** Yes — set `includeAnswers` and each question row carries its top 5 answers by votes with clean text bodies. **Does it cover comments or user profiles?** Not in this version. Question and answer data covers the main use cases; tell us if you need more. **Why did my huge run stop early?** The shared keyless quota ran out. The run saves everything fetched and the log says exactly what happened; add your own free API key to raise the limit to 10,000/day. ### More tools from Scrapemint - [GitHub Repo Stats](https://apify.com/scrapemint/github-repo-stats): stars, forks and metadata for repos in bulk. - [Google Trends Scraper](https://apify.com/scrapemint/google-trends-scraper): search interest over time for any topic. - [Website Content Scraper](https://apify.com/scrapemint/website-content-scraper): clean text from any page. # Actor input Schema ## `queries` (type: `array`): Free text searches, one per line ("memory leak nodejs", "pandas merge slow"). Each returns the most relevant questions. ## `tags` (type: `array`): Tags to sweep, one per line ("playwright", "apify"). Each returns that tag's top questions by votes. ## `maxPerQuery` (type: `integer`): How many questions to return for each search or tag (max 100). ## `includeAnswers` (type: `boolean`): Attach the top answers (by votes, up to 5) to each question row. Costs one extra API call per 100 questions. ## `site` (type: `string`): Which Stack Exchange site to search. Default stackoverflow; also works with superuser, serverfault, askubuntu, softwareengineering and any other site slug. ## `newerThanDays` (type: `integer`): Skip questions created more than this many days ago. 0 = no age limit. ## `apiKey` (type: `string`): Optional: your own Stack Exchange app key. Raises the shared daily quota (300 requests/day per IP) to 10,000/day. Get one free at stackapps.com. ## `maxRows` (type: `integer`): Stop after this many rows in total. ## Actor input object example ```json { "queries": [ "web scraping blocked", "playwright timeout" ], "tags": [ "web-scraping" ], "maxPerQuery": 15, "includeAnswers": false, "site": "stackoverflow", "newerThanDays": 0, "maxRows": 1000 } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "queries": [ "web scraping blocked", "playwright timeout" ], "tags": [ "web-scraping" ] }; // Run the Actor and wait for it to finish const run = await client.actor("scrapemint/stackoverflow-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "queries": [ "web scraping blocked", "playwright timeout", ], "tags": ["web-scraping"], } # Run the Actor and wait for it to finish run = client.actor("scrapemint/stackoverflow-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "queries": [ "web scraping blocked", "playwright timeout" ], "tags": [ "web-scraping" ] }' | apify call scrapemint/stackoverflow-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=scrapemint/stackoverflow-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Stack Overflow Scraper: Questions, Answers & Tags", "description": "Search Stack Overflow (or any Stack Exchange site) by keyword or tag and get one row per question: title, body, tags, score, views, answer count, accepted answer and author, with top answers optionally attached. For dev tool marketing, research and support teams. No API key needed.", "version": "0.1", "x-build-id": "G1cvoesj9KdCzopu4" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/scrapemint~stackoverflow-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-scrapemint-stackoverflow-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/scrapemint~stackoverflow-scraper/runs": { "post": { "operationId": "runs-sync-scrapemint-stackoverflow-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/scrapemint~stackoverflow-scraper/run-sync": { "post": { "operationId": "run-sync-scrapemint-stackoverflow-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "queries": { "title": "Search queries", "type": "array", "description": "Free text searches, one per line (\"memory leak nodejs\", \"pandas merge slow\"). Each returns the most relevant questions.", "items": { "type": "string" } }, "tags": { "title": "Tags", "type": "array", "description": "Tags to sweep, one per line (\"playwright\", \"apify\"). Each returns that tag's top questions by votes.", "items": { "type": "string" } }, "maxPerQuery": { "title": "Questions per query or tag", "minimum": 1, "maximum": 100, "type": "integer", "description": "How many questions to return for each search or tag (max 100).", "default": 15 }, "includeAnswers": { "title": "Include top answers", "type": "boolean", "description": "Attach the top answers (by votes, up to 5) to each question row. Costs one extra API call per 100 questions.", "default": false }, "site": { "title": "Stack Exchange site", "type": "string", "description": "Which Stack Exchange site to search. Default stackoverflow; also works with superuser, serverfault, askubuntu, softwareengineering and any other site slug.", "default": "stackoverflow" }, "newerThanDays": { "title": "Only questions newer than (days)", "minimum": 0, "maximum": 7300, "type": "integer", "description": "Skip questions created more than this many days ago. 0 = no age limit.", "default": 0 }, "apiKey": { "title": "Stack Exchange API key (optional)", "type": "string", "description": "Optional: your own Stack Exchange app key. Raises the shared daily quota (300 requests/day per IP) to 10,000/day. Get one free at stackapps.com." }, "maxRows": { "title": "Max rows", "minimum": 1, "maximum": 50000, "type": "integer", "description": "Stop after this many rows in total.", "default": 1000 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```