# Stack Overflow Scraper (`ef12/stack-overflow-scraper`) Actor Scrape Stack Overflow questions and top answers by tags, search query, and sort order using the Stack Exchange API. Get titles, scores, view counts, answers, and question bodies. - **URL**: https://apify.com/ef12/stack-overflow-scraper.md - **Developed by:** [Daniel Wilson](https://apify.com/ef12) (community) - **Categories:** Developer tools - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per event This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Stack Overflow Scraper **Search and extract Stack Overflow questions by tags, keywords, and sort order — with optional top answers — via the official Stack Exchange API.** This Actor calls the [Stack Exchange API](https://api.stackexchange.com/docs) to pull questions from Stack Overflow without scraping HTML. It supports filtering by tags, free-text search in titles, multiple sort orders, minimum score thresholds, and can include the highest-voted answer for each question. Results are pushed to the Apify dataset as clean, structured JSON ready for analysis, export, or integration. --- ### What does Stack Overflow Scraper do? Stack Overflow Scraper lets you programmatically search and extract **Stack Overflow questions and their top answers** using the official Stack Exchange API at `api.stackexchange.com/2.3`. You give it **tags** (e.g., `python`, `asyncio`), an optional **text query**, a **sort order** (votes, creation date, activity, or relevance), and it returns clean JSON with titles, URLs, scores, view counts, answer counts, tags, dates, the full question body, and optionally the top-voted answer body. It is a **pure API consumer** — no HTML scraping, no browser overhead, no DOM parsing. This makes it fast, reliable, and resilient to website structure changes. All HTML from question and answer bodies is stripped using BeautifulSoup so you get plain text ready for downstream processing. Because it runs on the [Apify platform](https://apify.com), you get all the standard platform benefits out of the box: **scheduled runs** via webhooks and cron, **API access** to results, **integrations** with Zapier, Make, Google Sheets, and Slack, **proxy rotation** (though the API typically doesn't need it), **monitoring** and **retry logic**, and **persistent datasets** you can export as JSON, CSV, Excel, or HTML. ### Why use Stack Overflow Scraper? - **Research and analysis** — Collect questions about a specific technology, framework, or problem domain. Analyze trends in question volume, score, and answer rates over time. - **Content creation** — Find high-voted questions on a topic to inspire blog posts, tutorials, or documentation. The top answer often contains the community-approved solution worth explaining further. - **Community management** — Monitor your open-source project's Stack Overflow tag. Track unanswered questions, low-score answers that need improvement, or trending topics in your ecosystem. - **Data science** — Build datasets for NLP models (question classification, answer quality prediction, duplicate detection), recommendation systems, or community health metrics. - **Job search prep** — Pull the most-voted questions for a stack you're interviewing for, and study the accepted solutions before an interview. - **Documentation gap analysis** — Find questions with high scores but no accepted answer — these signal areas where official documentation or tutorials are missing. ### How to use Stack Overflow Scraper 1. **Open the Actor** in Apify Console (or run locally with `apify run`). 2. **Configure the input fields** in the Input tab: - **Tags**: Enter one or more Stack Overflow tags (e.g., `python`, `asyncio`). Questions must have **all** of the specified tags. - **Search Query** (optional): Enter a free-text query to filter by question title. - **Sort Order**: Choose how to rank results — by **Votes** (default), **Creation Date**, **Activity**, or **Relevance**. - **Direction**: Ascending or descending (default: descending). - **Max Results**: How many questions to return (1–100, default 25). - **Include Top Answer**: Toggle to include the highest-voted answer body for each question. - **Minimum Score**: Filter out questions below a certain score (default 0). 3. **Run the Actor**. It will call the Stack Exchange API, process the results, and push them to the dataset. 4. **Download the results** from the Dataset tab in JSON, HTML, CSV, Excel, or XML format. Or access them programmatically via the [Apify API](https://docs.apify.com/api/v2). ### Input | Field | Type | Default | Description | | --- | --- | --- | --- | | `tags` | `string[]` | `["python"]` | Stack Overflow tags to filter by | | `query` | `string` | `""` | Free-text search in question titles | | `sort` | `string` | `"votes"` | Sort order: `votes`, `creation`, `activity`, `relevance` | | `order` | `string` | `"desc"` | Direction: `asc` or `desc` | | `max_results` | `integer` | `25` | Max questions to return (1–100) | | `include_answers` | `boolean` | `false` | Include top-voted answer body | | `min_score` | `integer` | `0` | Minimum question score | ### Output Each result is a JSON object pushed to the default dataset: ```json { "title": "How does asyncio actually work?", "url": "https://stackoverflow.com/questions/49005651/how-does-asyncio-actually-work", "score": 2124, "view_count": 678032, "answer_count": 8, "is_answered": true, "tags": ["python", "asyncio", "python-3.x"], "creation_date": "2018-02-27T09:27:18", "body": "This question is a duplicate of ... how does asyncio work ...", "top_answer": "This is the highest-voted answer text..." } ```` | Field | Type | Description | | --- | --- | --- | | `title` | `string` | Question title | | `url` | `string` | Stack Overflow URL | | `score` | `integer` | Upvote count | | `view_count` | `integer` | Number of views | | `answer_count` | `integer` | Number of answers | | `is_answered` | `boolean` | Has an accepted answer | | `tags` | `string[]` | Question tags | | `creation_date` | `string` | ISO-8601 date | | `body` | `string` | Plain text question body | | `top_answer` | `string` | (optional) Plain text top answer | ### Pricing / Cost Estimation This Actor uses **pay-per-event (PPE) pricing**: - **10 results are free** per run — no charge for the first 10 questions pushed to the dataset. - **After 10 results**, each additional result costs **$0.003** (0.3 cents). - **Actor start event**: $0.00005 per run. For a typical search of 25 questions (with answers), expect roughly **$0.00005 (start) + 15 × $0.003 = $0.045** per run. A full 100-question search costs about **$0.00005 + 90 × $0.003 = $0.27**. These costs apply to **Apify platform runs only**. Local `apify run` testing is free. ### Advanced Options - **Combine tags and query**: Use `tags` for broad filtering and `query` for title matching. For example, `tags: ["python"]` with `query: "asyncio event loop"` returns Python questions about asyncio event loops. - **Use `relevance` sort** when you provide a query — it ranks by text relevance rather than just votes. - **Filter by minimum score**: Set `min_score: 5` to skip low-quality questions and focus on well-received content. - **Keep `max_results` under 100**: The Stack Exchange API returns at most 100 results per call, and the Actor caps at 100. - **Performance**: Each run makes 1–2 API calls (one for questions, optionally one for answers). No crawling, no browser — results return in seconds. ### FAQ, Disclaimers, and Support **Is this legal?** The Actor uses the official [Stack Exchange API](https://api.stackexchange.com/docs), which is a public, rate-limited API provided by Stack Overflow. Usage is subject to the [Stack Exchange API Terms of Service](https://stackoverflow.com/legal/api-terms-of-use). This Actor respects the API's `backoff` mechanism and rate limits. You are responsible for complying with the applicable terms of service for your use case. **Rate limits**: The Stack Exchange API allows up to 10,000 requests per day per IP (or 300 per day without an API key). The Actor does not require an API key — it hits the public endpoint — but adding one via environment variables will increase your rate limit if needed. **Output format**: You can download the dataset in various formats such as JSON, HTML, CSV, or Excel from the Apify Console Dataset tab or via the API. **Known limitations**: The Actor does not support pagination (only the first page of results is fetched). It does not fetch all answers — only the top-voted one per question (when `include_answers` is enabled). It does not handle API errors gracefully beyond error logging and exit. **Need help?** Open an issue on the [GitHub repository](https://github.com/apify/apify-fleet) or visit the [Apify documentation](https://docs.apify.com). For custom scraping solutions, contact Apify's development team. *** *Built with the Apify Python SDK and Stack Exchange API.* # Actor input Schema ## `query` (type: `string`): Search query for matching question titles (e.g. 'python async'). Optional — use tags instead for a broader search. ## `tags` (type: `array`): Stack Overflow tags to filter by (e.g. \['python', 'asyncio']). Questions must have all specified tags. ## `sort` (type: `string`): How to sort the questions ## `order` (type: `string`): Sort direction ## `max_results` (type: `integer`): Maximum number of questions to return (max 100) ## `include_answers` (type: `boolean`): Include the highest-voted answer for each question ## `min_score` (type: `integer`): Minimum question score to include in results (0 = all) ## Actor input object example ```json { "query": "", "tags": [ "python" ], "sort": "votes", "order": "desc", "max_results": 25, "include_answers": false, "min_score": 0 } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("ef12/stack-overflow-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("ef12/stack-overflow-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call ef12/stack-overflow-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=ef12/stack-overflow-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Stack Overflow Scraper", "description": "Scrape Stack Overflow questions and top answers by tags, search query, and sort order using the Stack Exchange API. Get titles, scores, view counts, answers, and question bodies.", "version": "0.0", "x-build-id": "D3C8XSRKXsRpHm8YE" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/ef12~stack-overflow-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-ef12-stack-overflow-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/ef12~stack-overflow-scraper/runs": { "post": { "operationId": "runs-sync-ef12-stack-overflow-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/ef12~stack-overflow-scraper/run-sync": { "post": { "operationId": "run-sync-ef12-stack-overflow-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "query": { "title": "Search Query", "type": "string", "description": "Search query for matching question titles (e.g. 'python async'). Optional — use tags instead for a broader search.", "default": "" }, "tags": { "title": "Tags", "type": "array", "description": "Stack Overflow tags to filter by (e.g. ['python', 'asyncio']). Questions must have all specified tags.", "default": [ "python" ], "items": { "type": "string" } }, "sort": { "title": "Sort Order", "enum": [ "votes", "creation", "activity", "relevance" ], "type": "string", "description": "How to sort the questions", "default": "votes" }, "order": { "title": "Direction", "enum": [ "asc", "desc" ], "type": "string", "description": "Sort direction", "default": "desc" }, "max_results": { "title": "Max Results", "minimum": 1, "maximum": 100, "type": "integer", "description": "Maximum number of questions to return (max 100)", "default": 25 }, "include_answers": { "title": "Include Top Answer", "type": "boolean", "description": "Include the highest-voted answer for each question", "default": false }, "min_score": { "title": "Minimum Score", "type": "integer", "description": "Minimum question score to include in results (0 = all)", "default": 0 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```