# Stack Overflow Scraper — Questions, Answers & Tags for RAG (`ahampton83/stackoverflow-scraper`) Actor Fetch Stack Overflow questions, answers, and comments as clean structured data with markdown formatting. Search by keyword, explore tags, or get full question threads. MCP-enabled for Claude, Cursor, and AI agents. - **URL**: https://apify.com/ahampton83/stackoverflow-scraper.md - **Developed by:** [Aaron Hampton](https://apify.com/ahampton83) (community) - **Categories:** AI, Developer tools, MCP servers - **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Stack Overflow Scraper — Questions, Answers & Tags for RAG Fetch Stack Overflow questions, answers, and comments as clean structured data with markdown-formatted bodies. Search by keyword, explore top questions for a tag, or get full question threads with accepted answers. Perfect for RAG pipelines, developer research, AI agents, and technical Q&A datasets. ### Features - **Keyword search** — search all Stack Overflow questions by keyword and tags - **Full question threads** — get questions with bodies, answers, and comments in clean markdown - **Tag exploration** — fetch top-voted questions for any tag (python, javascript, react, etc.) - **User profiles** — fetch user reputation, badges, and top questions - **Clean markdown** — all HTML bodies converted to readable markdown (code blocks, lists, links, images, tables, blockquotes) - **Answer ordering** — sorted by score with accepted answers first - **Dual-mode** — run as a normal Apify Actor OR connect as an MCP server for Claude, Cursor, and other AI agents - **No browser required** — uses the free Stack Exchange API, no proxies or auth needed ### Use Cases - **RAG pipelines** — feed Stack Overflow Q&A into vector databases for code-aware AI assistants - **Developer research** — find canonical answers to programming questions across technologies - **AI agents** — let Claude/Cursor search Stack Overflow via MCP tools - **Documentation** — collect top answers for a tag to build curated knowledge bases - **Trend analysis** — track popular topics and technologies by tag and score ### Input (Normal Actor Mode) | Field | Type | Description | |-------|------|-------------| | `mode` | enum | `search`, `question`, `tag`, or `user` | | `query` | string | Search keyword (mode=search) | | `questionId` | integer | Stack Overflow question ID (mode=question) | | `tag` | string | Tag name like "python" (mode=tag) | | `userId` | integer | Stack Overflow user ID (mode=user) | | `tags` | array | Filter search by tags (mode=search) | | `acceptedOnly` | boolean | Only questions with accepted answers (mode=search) | | `maxQuestions` | integer | Max questions to return (default 10) | | `includeAnswers` | boolean | Fetch answers for each question (default true) | | `includeComments` | boolean | Fetch comments on questions (default false) | | `maxAnswersPerQuestion` | integer | Max answers per question (default 5) | | `sortBy` | enum | `relevance`, `votes`, or `newest` (default relevance) | ### Output ```json { "id": 393272, "title": "How to reverse a list in Python", "url": "https://stackoverflow.com/questions/393272/...", "score": 2534, "viewCount": 1200000, "answerCount": 15, "tags": ["python", "list", "reverse"], "isAnswered": true, "acceptedAnswerId": 393280, "createdAt": "2010-10-13T10:00:00.000Z", "bodyMarkdown": "I want to reverse a list:\n```python\nmy_list = [1, 2, 3]\n```", "answers": [ { "id": 393280, "score": 1800, "isAccepted": true, "createdAt": "2010-10-13T10:05:00.000Z", "bodyMarkdown": "Use slicing:\n```python\nreversed_list = my_list[::-1]\n```" } ] } ```` ### MCP Tools When connected as an MCP server, the following tools are available: | Tool | Description | |------|-------------| | `search_questions` | Search Stack Overflow by keyword. Returns titles, scores, tags, and optionally answers. | | `get_question` | Fetch a full question with markdown body, answers (sorted accepted-first), and optional comments. | | `get_top_answers` | Get top-voted questions for a tag with their best answers. Great for discovering canonical Q\&A. | ### Pricing Pay-per-event. First results are free; subsequent events billed at: | Event | Price | |-------|-------| | Question fetched | $0.002 | | Answer fetched | $0.001 | | MCP tool call | $0.005 | | Actor start | $0.00005 | Volume discounts available at higher tiers. ### Data Source This Actor uses the [Stack Exchange API](https://api.stackexchange.com/) — a free, public API with generous rate limits (300 requests/2min without a key, 10,000/2min with a key). No authentication or proxies required. # Actor input Schema ## `mode` (type: `string`): search = keyword search, question = full question with answers, tag = top questions for a tag, user = user profile + top questions ## `query` (type: `string`): Keyword search query (required for mode=search). Searches SO question titles via the Stack Exchange API. ## `questionId` (type: `integer`): Stack Overflow question ID (required for mode=question). E.g., 393272 for 'How to reverse a list in Python'. ## `tag` (type: `string`): Stack Overflow tag name (required for mode=tag). E.g., 'python', 'javascript', 'react'. ## `userId` (type: `integer`): Stack Overflow user ID (required for mode=user). E.g., 22656 for Jon Skeet. ## `tags` (type: `array`): Filter search results to questions with these tags (mode=search). ## `acceptedOnly` (type: `boolean`): Filter to questions that have an accepted answer (mode=search). ## `maxQuestions` (type: `integer`): Maximum number of questions to return. ## `includeAnswers` (type: `boolean`): Fetch answers for each question. ## `includeComments` (type: `boolean`): Fetch comments for each question. ## `maxAnswersPerQuestion` (type: `integer`): Maximum answers to fetch per question. ## `sortBy` (type: `string`): Sort order for search results. ## `outputFormat` (type: `string`): markdown = include markdown-formatted bodies, json = raw structured data only. ## Actor input object example ```json { "mode": "search", "query": "", "questionId": 0, "tag": "", "userId": 0, "tags": [], "acceptedOnly": false, "maxQuestions": 10, "includeAnswers": true, "includeComments": false, "maxAnswersPerQuestion": 5, "sortBy": "relevance", "outputFormat": "markdown" } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("ahampton83/stackoverflow-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("ahampton83/stackoverflow-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call ahampton83/stackoverflow-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=ahampton83/stackoverflow-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Stack Overflow Scraper — Questions, Answers & Tags for RAG", "description": "Fetch Stack Overflow questions, answers, and comments as clean structured data with markdown formatting. Search by keyword, explore tags, or get full question threads. MCP-enabled for Claude, Cursor, and AI agents.", "version": "0.1", "x-build-id": "pkplQRP3JIbj2lu2U" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/ahampton83~stackoverflow-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-ahampton83-stackoverflow-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/ahampton83~stackoverflow-scraper/runs": { "post": { "operationId": "runs-sync-ahampton83-stackoverflow-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/ahampton83~stackoverflow-scraper/run-sync": { "post": { "operationId": "run-sync-ahampton83-stackoverflow-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "mode" ], "properties": { "mode": { "title": "Mode", "enum": [ "search", "question", "tag", "user" ], "type": "string", "description": "search = keyword search, question = full question with answers, tag = top questions for a tag, user = user profile + top questions", "default": "search" }, "query": { "title": "Search query", "type": "string", "description": "Keyword search query (required for mode=search). Searches SO question titles via the Stack Exchange API.", "default": "" }, "questionId": { "title": "Question ID", "type": "integer", "description": "Stack Overflow question ID (required for mode=question). E.g., 393272 for 'How to reverse a list in Python'.", "default": 0 }, "tag": { "title": "Tag", "type": "string", "description": "Stack Overflow tag name (required for mode=tag). E.g., 'python', 'javascript', 'react'.", "default": "" }, "userId": { "title": "User ID", "type": "integer", "description": "Stack Overflow user ID (required for mode=user). E.g., 22656 for Jon Skeet.", "default": 0 }, "tags": { "title": "Filter by tags", "type": "array", "description": "Filter search results to questions with these tags (mode=search).", "default": [], "items": { "type": "string" } }, "acceptedOnly": { "title": "Only questions with accepted answers", "type": "boolean", "description": "Filter to questions that have an accepted answer (mode=search).", "default": false }, "maxQuestions": { "title": "Max questions", "minimum": 1, "maximum": 100, "type": "integer", "description": "Maximum number of questions to return.", "default": 10 }, "includeAnswers": { "title": "Include answers", "type": "boolean", "description": "Fetch answers for each question.", "default": true }, "includeComments": { "title": "Include comments", "type": "boolean", "description": "Fetch comments for each question.", "default": false }, "maxAnswersPerQuestion": { "title": "Max answers per question", "minimum": 0, "maximum": 50, "type": "integer", "description": "Maximum answers to fetch per question.", "default": 5 }, "sortBy": { "title": "Sort by", "enum": [ "relevance", "votes", "newest" ], "type": "string", "description": "Sort order for search results.", "default": "relevance" }, "outputFormat": { "title": "Output format", "enum": [ "markdown", "json" ], "type": "string", "description": "markdown = include markdown-formatted bodies, json = raw structured data only.", "default": "markdown" } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```