Stack Overflow Scraper — Questions, Answers, Users & Tags

Scrape Stack Overflow at scale using the official Stack Exchange API v2.3. Extract full question threads with answers + comments, search by keyword or tag, pull user profiles with reputation & badges, and browse the tag ecosystem.

No authentication needed for the typical quota (300 requests per IP per day — more than enough for most jobs). Clean JSON output, ready for analytics, RAG, or trend-tracking pipelines.

---

Why this scraper

Stack Overflow is still the world's largest technical Q&A corpus — 24M+ questions, 35M+ answers, battle-tested solutions to every common programming problem. The downside: the API is clunky, the filter system is obscure, and the site has no bulk export. This actor hides all of that behind a single clean JSON interface.

• ✅ 6 modes — search, questions-by-tag, question-detail (with answers), answers, user-profile, tags
• ✅ Full Q&A threads — score, accepted-answer flag, markdown body, comments
• ✅ Tag-filtered feeds with any tag combo (python;asyncio or react;hooks)
• ✅ User profiles with top posts, rep history, badges
• ✅ Sort options — votes / relevance / creation / activity / hot / week / month
• ✅ Rate-limit aware — API backoff header respected automatically

---

Use cases

1. Build a RAG corpus for a coding assistant

Pull the top 1000 questions in python + asyncio, fetch each with answers, index into your vector DB.

{ "mode": "questions", "tagged": "python;asyncio", "sort": "votes", "maxResults": 1000 }

2. Competitive research — what errors do users hit with your library?

Search for your library name + common error terms. The question-per-view and up-vote counts rank real pain.

{ "mode": "search", "query": "your-library-name error", "sort": "votes" }

3. Content strategy — find high-traffic questions without a great answer

Look at questions with many views but low accepted-answer scores — prime opportunity for a blog post that ranks.

{ "mode": "questions", "tagged": "typescript", "sort": "popular", "maxResults": 500 }

4. Expert-finder — top contributors in a niche

Search questions by tag, aggregate answer authors by reputation, extract specialists.

{ "mode": "questions", "tagged": "rust", "sort": "votes", "maxResults": 200 }

Then iterate top answerers with mode: "user_profile" + userId.

---

Input schema

Field	Type	Description
mode	enum	search / questions / question_detail / answers / user_profile / tags
query	string	Keyword search (for search mode)
tagged	string	Tag filter; use ; for multi-tag (python;pandas)
questionId	int	Question ID (for question_detail / answers)
userId	int	User ID (for user_profile)
sort	enum	relevance / votes / creation / activity / hot / week / month / popular / name
maxResults	int	Result cap

Output fields

Questions: question_id, title, link, tags[], score, answer_count, view_count, is_answered, creation_date, owner{}, and on question_detail also body, answers[] with full comment threads.

Answers: answer_id, body, score, is_accepted, owner{}, creation_date, comments[].

User profiles: user_id, display_name, reputation, badge_counts{}, top_questions[], top_answers[], about_me.

Tags: name, count, has_synonyms, is_moderator_only.

---

Pricing

Stack Exchange API allows 300 requests/day per IP without auth — enough to pull thousands of questions. The actor is optimized to batch API calls (up to 100 question IDs per request where supported).

Typical runs:

• Search, 100 results: ~5 seconds, ~$0.001
• 100 full question details with answers: ~15 seconds, ~$0.003
• User profile + top posts: ~3 seconds, ~$0.0005

---

Integrations

• Scheduler: Apify cron for daily/hourly exports
• Destinations: S3 / GCS / BigQuery / Sheets / Airtable / Webhook
• Automation: Zapier, Make, n8n
• Code access: JS/Python SDK + REST API + Apify CLI

# REST
curl -X POST "https://api.apify.com/v2/acts/EkV1XtaiS0jz6WvJL/runs?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"questions","tagged":"python","sort":"votes","maxResults":100}'

---

FAQ

Do I need a Stack Exchange API key? No — the default 300/day per-IP quota works for most jobs. For higher throughput, you can add an API key as a future input field.

Can I scrape answers that were deleted / on hold? No — the API only returns publicly visible content. Deleted content is not accessible.

Does this include Stack Exchange sites other than Stack Overflow? Currently Stack Overflow only. Other sites (Server Fault, Math, etc.) use the same API and could be added on request.

How accurate is the hot sort? It mirrors the SO home-page "hot" algorithm (recent + upvoted). Good for trending-question dashboards.

Will it handle rate-limit headers? Yes — the actor reads backoff in the API response and sleeps accordingly before the next request.

---

Keywords

stackoverflow scraper, stack overflow scraper, stack exchange api, stackoverflow questions, stackoverflow answers, SO scraper, developer Q&A scraper, programming questions scraper, stackoverflow tags, stackoverflow user profile, stackoverflow export, coding Q&A dataset

Companion actors (same author)

• DEV.to Article Scraper — technical blog posts
• Hacker News Scraper — stories, comments, search
• GitHub Trending Scraper — trending repos
• Lobsters Scraper — curated tech community

Changelog

• v0.1 — Initial release. 6 modes (search, questions, question_detail, answers, user_profile, tags), 9 sort options, API-level rate-limit backoff.