Stack Overflow Scraper

Search and extract Stack Overflow questions by tags, keywords, and sort order — with optional top answers — via the official Stack Exchange API.

This Actor calls the Stack Exchange API to pull questions from Stack Overflow without scraping HTML. It supports filtering by tags, free-text search in titles, multiple sort orders, minimum score thresholds, and can include the highest-voted answer for each question. Results are pushed to the Apify dataset as clean, structured JSON ready for analysis, export, or integration.

---

What does Stack Overflow Scraper do?

Stack Overflow Scraper lets you programmatically search and extract Stack Overflow questions and their top answers using the official Stack Exchange API at api.stackexchange.com/2.3. You give it tags (e.g., python, asyncio), an optional text query, a sort order (votes, creation date, activity, or relevance), and it returns clean JSON with titles, URLs, scores, view counts, answer counts, tags, dates, the full question body, and optionally the top-voted answer body.

It is a pure API consumer — no HTML scraping, no browser overhead, no DOM parsing. This makes it fast, reliable, and resilient to website structure changes. All HTML from question and answer bodies is stripped using BeautifulSoup so you get plain text ready for downstream processing.

Because it runs on the Apify platform, you get all the standard platform benefits out of the box: scheduled runs via webhooks and cron, API access to results, integrations with Zapier, Make, Google Sheets, and Slack, proxy rotation (though the API typically doesn't need it), monitoring and retry logic, and persistent datasets you can export as JSON, CSV, Excel, or HTML.

Why use Stack Overflow Scraper?

• Research and analysis — Collect questions about a specific technology, framework, or problem domain. Analyze trends in question volume, score, and answer rates over time.
• Content creation — Find high-voted questions on a topic to inspire blog posts, tutorials, or documentation. The top answer often contains the community-approved solution worth explaining further.
• Community management — Monitor your open-source project's Stack Overflow tag. Track unanswered questions, low-score answers that need improvement, or trending topics in your ecosystem.
• Data science — Build datasets for NLP models (question classification, answer quality prediction, duplicate detection), recommendation systems, or community health metrics.
• Job search prep — Pull the most-voted questions for a stack you're interviewing for, and study the accepted solutions before an interview.
• Documentation gap analysis — Find questions with high scores but no accepted answer — these signal areas where official documentation or tutorials are missing.

How to use Stack Overflow Scraper

1. Open the Actor in Apify Console (or run locally with apify run).
2. Configure the input fields in the Input tab:
   • Tags: Enter one or more Stack Overflow tags (e.g., python, asyncio). Questions must have all of the specified tags.
   • Search Query (optional): Enter a free-text query to filter by question title.
   • Sort Order: Choose how to rank results — by Votes (default), Creation Date, Activity, or Relevance.
   • Direction: Ascending or descending (default: descending).
   • Max Results: How many questions to return (1–100, default 25).
   • Include Top Answer: Toggle to include the highest-voted answer body for each question.
   • Minimum Score: Filter out questions below a certain score (default 0).
3. Run the Actor. It will call the Stack Exchange API, process the results, and push them to the dataset.
4. Download the results from the Dataset tab in JSON, HTML, CSV, Excel, or XML format. Or access them programmatically via the Apify API.

Input

Field	Type	Default	Description
tags	string[]	["python"]	Stack Overflow tags to filter by
query	string	""	Free-text search in question titles
sort	string	"votes"	Sort order: votes, creation, activity, relevance
order	string	"desc"	Direction: asc or desc
max_results	integer	25	Max questions to return (1–100)
include_answers	boolean	false	Include top-voted answer body
min_score	integer	0	Minimum question score

Output

Each result is a JSON object pushed to the default dataset:

{
  "title": "How does asyncio actually work?",
  "url": "https://stackoverflow.com/questions/49005651/how-does-asyncio-actually-work",
  "score": 2124,
  "view_count": 678032,
  "answer_count": 8,
  "is_answered": true,
  "tags": ["python", "asyncio", "python-3.x"],
  "creation_date": "2018-02-27T09:27:18",
  "body": "This question is a duplicate of ... how does asyncio work ...",
  "top_answer": "This is the highest-voted answer text..."
}

Field	Type	Description
title	string	Question title
url	string	Stack Overflow URL
score	integer	Upvote count
view_count	integer	Number of views
answer_count	integer	Number of answers
is_answered	boolean	Has an accepted answer
tags	string[]	Question tags
creation_date	string	ISO-8601 date
body	string	Plain text question body
top_answer	string	(optional) Plain text top answer

Pricing / Cost Estimation

This Actor uses pay-per-event (PPE) pricing:

• 10 results are free per run — no charge for the first 10 questions pushed to the dataset.
• After 10 results, each additional result costs $0.003 (0.3 cents).
• Actor start event: $0.00005 per run.

For a typical search of 25 questions (with answers), expect roughly $0.00005 (start) + 15 × $0.003 = $0.045 per run. A full 100-question search costs about $0.00005 + 90 × $0.003 = $0.27.

These costs apply to Apify platform runs only. Local apify run testing is free.

Advanced Options

• Combine tags and query: Use tags for broad filtering and query for title matching. For example, tags: ["python"] with query: "asyncio event loop" returns Python questions about asyncio event loops.
• Use relevance sort when you provide a query — it ranks by text relevance rather than just votes.
• Filter by minimum score: Set min_score: 5 to skip low-quality questions and focus on well-received content.
• Keep max_results under 100: The Stack Exchange API returns at most 100 results per call, and the Actor caps at 100.
• Performance: Each run makes 1–2 API calls (one for questions, optionally one for answers). No crawling, no browser — results return in seconds.

FAQ, Disclaimers, and Support

Is this legal? The Actor uses the official Stack Exchange API, which is a public, rate-limited API provided by Stack Overflow. Usage is subject to the Stack Exchange API Terms of Service. This Actor respects the API's backoff mechanism and rate limits. You are responsible for complying with the applicable terms of service for your use case.

Rate limits: The Stack Exchange API allows up to 10,000 requests per day per IP (or 300 per day without an API key). The Actor does not require an API key — it hits the public endpoint — but adding one via environment variables will increase your rate limit if needed.

Output format: You can download the dataset in various formats such as JSON, HTML, CSV, or Excel from the Apify Console Dataset tab or via the API.

Known limitations: The Actor does not support pagination (only the first page of results is fetched). It does not fetch all answers — only the top-voted one per question (when include_answers is enabled). It does not handle API errors gracefully beyond error logging and exit.

Need help? Open an issue on the GitHub repository or visit the Apify documentation. For custom scraping solutions, contact Apify's development team.

---

Built with the Apify Python SDK and Stack Exchange API.