Stack Overflow Scraper: Export Questions, Answers, Tags and Users

Stack Overflow Scraper collects developer-community data through the official Stack Exchange API 2.3. Retrieve questions, answers, tags, users, and public Stack Exchange sites, then export structured datasets in JSON, CSV, or Excel format.

Search by keyword, tag, title, answer status, or date range. Collect question bodies, scores, view counts, accepted-answer information, and user details without logging in.

When redistributing post content, preserve the required author attribution, source link, and applicable CC BY-SA license information.

Features

• Question collection: Retrieve questions by tag, sorting method, and date range.
• Advanced search: Search by keywords, title, tags, accepted-answer status, and minimum answer count.
• Answer extraction: Collect answers for specified question IDs or URLs.
• Tag statistics: Retrieve tag names and associated question counts.
• User information: Collect user profiles, reputation, and badge counts.
• Site discovery: List public sites in the Stack Exchange network.
• Body formats: Store post bodies in Markdown, plain text, or HTML.
• Pagination: Follow API pagination metadata to collect additional results.
• Rate-limit handling: Respect API backoff instructions and monitor remaining quota.
• Error recovery: Handle invalid parameters, throttling, timeouts, and individual failed requests.
• Duplicate removal: Avoid duplicate question, answer, user, tag, and site IDs.
• Structured exports: Prepare collected records for analysis and downstream applications.

Use Cases

• Developer-tool research: Analyze common technical problems using question tags, scores, and view counts.
• AI and RAG datasets: Prepare question-and-answer collections while preserving attribution and license information.
• Content research and SEO: Identify recurring developer questions and popular technical topics.
• Developer relations: Discover active contributors and top answerers in specific technology areas.
• Technology trend analysis: Compare tag counts and community activity across Stack Overflow, Super User, Server Fault, and other supported sites.

Available Modes

Mode	API Resource	Description
questions	/questions	Retrieve questions by tag, sorting method, and date range.
search	/search/advanced	Search questions by keyword, title, tags, accepted-answer status, and minimum answer count.
answers	/questions/{ids}/answers	Retrieve answers for specified question IDs or URLs, using batches of up to 100 IDs.
tags	/tags	Retrieve tag names and question counts.
users	/users or /users/{ids}	Search users by name or retrieve specific user profiles.
sites	/sites	Retrieve public Stack Exchange sites.

Input Parameters

Parameter	Type	Default	Description
mode	String	search	Collection mode: questions, search, answers, tags, users, or sites.
site	String	stackoverflow	Stack Exchange site key, such as stackoverflow, superuser, serverfault, askubuntu, math, or datascience.
query	String	web scraping	Keyword used for question searches.
tags	String Array	[]	Tags to filter by. Every supplied tag must match.
titleContains	String	Empty	Filter search results by title text.
acceptedOnly	Boolean	false	Return only questions with an accepted answer in search mode.
minAnswers	Integer	Empty	Minimum number of answers in search mode.
sort	String	activity	Sorting method. Options depend on the selected mode; examples include relevance for search, popular for tags, and reputation for users.
fromDate	String	Empty	Earliest question date.
toDate	String	Empty	Latest question date.
questionIds	String Array	[]	Question IDs or URLs for answers mode.
userIds	String Array	[]	User IDs or profile URLs for users mode.
nameContains	String	Empty	Partial tag or user name for supported modes.
includeBody	Boolean	true	Include post bodies in the output.
includeAnswers	Boolean	false	Include nested answers with each question. Answers are fetched in batches of up to 100 IDs.
bodyFormat	String	markdown	Post body format: markdown, text, or html.
maxItems	Integer	200	Maximum number of rows to save. Maximum: 10,000.
timeoutSecs	Integer	300	Overall execution time limit in seconds. Maximum: 3,600.
requestTimeoutSecs	Integer	30	Timeout for an individual API request, in seconds.
apiKey	String	Empty	Optional Stack Apps API key. Can increase the daily request quota.

Example Input

{
  "mode": "search",
  "site": "stackoverflow",
  "query": "web scraping",
  "tags": ["python"],
  "minAnswers": 1,
  "includeBody": true,
  "bodyFormat": "markdown",
  "maxItems": 50
}

Output

Each collected item is saved as a structured dataset record. Output fields depend on the selected mode and the information return.

Example Question Record

{
  "site": "stackoverflow",
  "questionId": 80008831,
  "questionTitle": "Merge two Celery tasks",
  "title": "Merge two Celery tasks",
  "url": "STACK_OVERFLOW_QUESTION_URL",
  "tags": ["python", "web-scraping", "celery"],
  "score": -5,
  "viewCount": 139,
  "answerCount": 0,
  "isAnswered": false,
  "createdAt": "2026-10-08T12:00:00Z",
  "ownerName": "Sandi",
  "ownerReputation": 188,
  "license": "CC BY-SA 4.0"
}

Output Fields

Field	Type	Description
questionTitle	String	Decoded question title. Also available as title for compatibility with existing exports.
title	String	Question title duplicated for compatibility.
url	String	Link to a question, answer, or tag.
body	String	Post body in the selected format. May be null when body collection is disabled.
answers	Array	Nested answers when answer collection is enabled.
ownerName	String	Display name of the post owner. May be empty for deleted accounts.
license	String	License information returned by the API.
badges	Object	Gold, silver, and bronze badge counts in user records.
apiSiteParameter	String	Site identifier returned by the sites collection.
error	String	Error code or description, such as QUOTA_LOW, TIMEOUT, INVALID_INPUT, bad_parameter, or throttle_violation.

Additional fields vary by mode. Question records can include scores, view counts, tags, answer counts, accepted-answer information, creation dates, and owner details.

A key-value store record named STATS contains quota and request statistics, including quotaMax, quotaRemaining, and requests.

Duplicate handling: Duplicate question, answer, user, tag, and site IDs are removed. A question with no answers is treated as a successful empty result rather than an error.

How It Works

1. Reads the selected mode, site, filters, and collection limits.
2. Sends requests to the relevant Stack Exchange API 2.3 resource.
3. Processes result pages containing up to 100 items each.
4. Follows the API's has_more pagination information when additional results are available.
5. Waits for the specified backoff period before making the next call to the same method.
6. Retries supported throttling responses, including HTTP 429, HTTP 502, or API error ID 502, once after waiting.
7. Records invalid parameters and other request failures without terminating the entire run.
8. Stops when the requested item limit, runtime limit, or available API quota prevents further collection.

When fewer than five API requests remain, the scraper records a QUOTA_LOW warning rather than exhausting the remaining quota.

Limits and API Quotas

• Each API response returns up to 100 items.
• The maximum collection limit is 10,000 rows per run.
• API backoff instructions are respected before subsequent requests to the same method.
• Request timeouts and overall execution limits can be configured.

Actual quotas and limits may vary according to the API's current policies and usage conditions.

Licensing and Attribution

Stack Exchange post content is provided under the applicable Creative Commons Attribution-ShareAlike license, as indicated by the API and the relevant post.

When republishing question or answer bodies, preserve the required attribution information, including the author's name, a link to the original post, and the applicable license. Review the current Stack Exchange licensing guidance before redistributing collected content.

FAQ

Can I collect data from Super User or Server Fault?

Yes. Set site to the relevant API site parameter, such as superuser or serverfault. Site-specific links use the corresponding site's domain.

Are question bodies returned as HTML?

The default format is Markdown. Set bodyFormat to text or html to request another supported format. Question titles are decoded before being saved.

Can I retrieve answers for a specific question?

Yes. Select answers mode and provide the question ID or full question URL through questionIds. A question with no answers returns no answer records.

Can I collect only questions with accepted answers?

Yes. In search mode, set acceptedOnly to true.

What happens when one site or ID fails?

The scraper records an appropriate error and continues processing other inputs when possible.

Related Tools

Related data-collection workflows may include Hacker News, GitHub, npm, and PyPI data collection. These can complement Stack Exchange datasets for developer-community research and technical trend analysis.

Trademark Notice

Stack Overflow, Stack Exchange, and their related names, logos, and trademarks belong to their respective owners. Their use in this documentation is solely to identify the data source and does not imply affiliation, sponsorship, endorsement, or partnership.

Disclaimer

This is an independent data-collection tool and is not affiliated with, endorsed by, sponsored by, or officially associated with Stack Overflow or Stack Exchange. It accesses publicly available information through the official API. Users are responsible for complying with the API terms, usage limits, attribution obligations, and applicable licensing requirements.

Is Web Scraping Legal?

This tool uses the public Stack Exchange without logging in or bypassing access controls. Whether collecting, storing, or redistributing data is permitted depends on the information involved, applicable laws, API terms, licensing conditions, and intended use.

• Public information: Public availability does not automatically grant unrestricted permission to reuse every type of content.
• Personal data: Privacy laws may apply when collected information identifies individuals or includes other personal information.
• Copyright and licensing: Post bodies and other creative content may be protected by copyright or governed by specific license terms. Preserve required attribution and follow the applicable license.
• Terms of use: Respect API usage rules, rate limits, and applicable source policies.
• Legal advice: This section provides general information, not legal advice. Consult a qualified legal professional if you need guidance about a specific use case.