Stack Overflow Scraper

Scrape Stack Overflow questions, answers, and tags via Stack Exchange API. Search by keyword or tag, get accepted answers, vote counts, and view statistics.

Use cases

• Find the most common problems developers hit with your library
• Mine accepted answers to build a support knowledge base
• Track question volume for a tag as an adoption signal
• Feed a developer-support chatbot with real Q&A
• Prioritise docs work by what people actually ask

Input

Parameter	Type	Required	Default	Description
searchQuery	string	Yes	"how to use async await javascript"	Text to search for in questions (e.g., 'how to use async await javascript')
tag	string	No	—	Optional tag to filter results (e.g., 'javascript', 'python'). Leave empty for no tag filter.
sort	string	No	"relevance"	How to sort the results Allowed: relevance, votes, creation, activity.
minAnswers	integer	No	1	Filter questions with at least this many answers
includeAnswers	boolean	No	true	Fetch answers for each question
maxResults	integer	Yes	30	Maximum number of questions to scrape (1-100)
proxyConfiguration	object (proxy)	No	{"useApifyProxy":true}	Proxy settings (Apify Proxy recommended for reliability)

Example input

{
  "searchQuery": "how to use async await javascript",
  "sort": "relevance",
  "minAnswers": 1,
  "includeAnswers": true,
  "maxResults": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

Output

The exact fields depend on the mode you run. This is real output from an actual run of this Actor:

{
  "questionId": 51328292,
  "title": "How to use Async and Await with AWS SDK Javascript",
  "body": "I am working with the AWS SDK using the KMS libary. I would like to use async and await instead of callbacks.\nimport AWS, { KMS } from \"aws-sdk\";\nthis…",
  "link": "https://stackoverflow.com/questions/51328292/how-to-use-async-and-await-with-aws-sdk-javascript",
  "score": 81,
  "answerCount": 3,
  "viewCount": 37088,
  "tags": [
    "node.js",
    "aws-sdk"
  ],
  "owner": {
    "displayName": "Kay",
    "reputation": 19956,
    "link": "https://stackoverflow.com/users/1548432/kay"
  },
  "isAnswered": true,
  "createdAt": "2018-07-13T15:18:07.000Z",
  "lastActivityAt": "2021-08-07T03:47:26.000Z",
  "answers": [
    {
      "answerId": 51328537,
      "body": "If you are using aws-sdk with version > 2.x, you can tranform a aws.Request to a promise with chain .promise() function.\nFor your case:\n  try {\n    le…",
      "score": 130,
      "isAccepted": true,
      "owner": {
        "displayName": "hoangdv",
        "reputation": 16303,
        "link": "https://stackoverflow.com/users/5196394/hoangdv"
      }
    },
    {
      "answerId": 51328494,
      "body": "await requires a Promise.  generateDataKey() returns a AWS.Request, not a Promise.  AWS.Request are EventEmitters (more or less) but have a promise me…",
      "score": 18,
      "isAccepted": false,
      "owner": {
        "displayName": "zero298",
        "reputation": 27202,
        "link": "https://stackoverflow.com/users/691711/zero298"
      }
    }
  ]
}

Field	Type
questionId	number
title	string
body	string
link	string
score	number
answerCount	number
viewCount	number
tags	array
owner	object
isAnswered	boolean
createdAt	string
lastActivityAt	string
answers	array of object — each with answerId, body, score, isAccepted, owner

Results are exportable from Apify Console or the API as JSON, CSV, Excel, or XML.

How to run it

In Apify Console — open the Actor, fill in the input form, click Start, then download the results from the Dataset tab.

With the JavaScript client

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });

const run = await client.actor('cloud9_ai/stackoverflow-scraper').call({
  "searchQuery": "how to use async await javascript",
  "sort": "relevance",
  "minAnswers": 1,
  "includeAnswers": true,
  "maxResults": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

With the Python client

from apify_client import ApifyClient

client = ApifyClient('YOUR_APIFY_TOKEN')

run = client.actor('cloud9_ai/stackoverflow-scraper').call(run_input={
  "searchQuery": "how to use async await javascript",
  "sort": "relevance",
  "minAnswers": 1,
  "includeAnswers": True,
  "maxResults": 5,
  "proxyConfiguration": {
    "useApifyProxy": True
  }
})

for item in client.dataset(run['defaultDatasetId']).iterate_items():
    print(item)

With the API — POST https://api.apify.com/v2/acts/cloud9_ai~stackoverflow-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN with the input JSON as the body.

Notes and limits

• No API key, account, or login is needed — just the input above.
• maxResults caps how much a single run collects, which is also what caps the run's cost.
• Requests to the source are paced to stay inside its rate limits.
• Supports Apify Proxy via proxyConfiguration; the default is fine for most runs.
• Only publicly available data is collected. How you use the output is your responsibility, including the source's terms of use and any applicable data-protection law.

Support

Found a bug, or need a field that isn't in the output? Open an issue on the Issues tab of this Actor in Apify Console. Issues there are read and answered.

License

Apache-2.0