Stack Exchange Questions & Answers Scraper

Extract Stack Overflow and Stack Exchange questions, answers, comments, tags, authors, scores, and view counts with the public Stack Exchange API.

Use this actor when you need reliable public Q&A data for developer community intelligence, support knowledge mining, technical content research, FAQ generation, or retrieval-augmented generation datasets.

What does Stack Exchange Questions & Answers Scraper do?

Stack Exchange Questions & Answers Scraper searches Stack Overflow and other Stack Exchange sites through the official public API.

It returns one dataset row per question.

Each row can include the question body, tags, owner details, score, view count, answer count, accepted-answer status, timestamps, and optional nested answers and comments.

The actor supports keyword searches, tag searches, date filters, direct question URLs, and numeric question IDs.

Because it uses the public API, it does not need browser automation, login cookies, or residential proxies.

Who is it for?

🧑‍💻 Developer relations teams can monitor questions about their SDKs, APIs, tools, and integrations.

🛟 Support teams can mine recurring public problems and convert them into docs, FAQs, and knowledge-base articles.

📚 Content teams can research developer pain points before writing tutorials or comparison pages.

🔎 Search and RAG teams can build public Q&A datasets with stable question URLs and answer text.

📈 Product teams can track score, view count, and answer-count signals around technical topics.

Why use this actor?

It is focused on structured Q&A records, not raw HTML pages.

It works across configurable Stack Exchange sites such as stackoverflow, serverfault, superuser, askubuntu, math, stats, and many others.

It supports direct URL enrichment when you already have a list of questions.

It can collect answers and comments in the same question row for downstream analysis.

It honors Stack Exchange API backoff instructions and retries transient failures.

What Stack Exchange sites are supported?

Use the Stack Exchange API site parameter.

Common values include:

• stackoverflow
• serverfault
• superuser
• askubuntu
• math
• stats
• electronics
• security
• softwareengineering

The default is stackoverflow.

Data you can extract

Field	Description
site	Stack Exchange API site value used for the run
questionId	Numeric Stack Exchange question ID
questionUrl	Canonical question URL
title	Question title
body	Question body HTML returned by the API
tags	Stack Exchange tags
ownerDisplayName	Public owner display name
ownerUserId	Public Stack Exchange user ID
ownerReputation	Owner reputation at fetch time
ownerProfileUrl	Owner profile URL
score	Question score
viewCount	Public view count
answerCount	Number of answers
commentCount	Number of comments
isAnswered	Whether the question is answered
acceptedAnswerId	Accepted answer ID when available
creationDate	Question creation date
lastActivityDate	Last activity date
lastEditDate	Last edit date
closedDate	Closed date when available
closedReason	Closed reason when available
answers	Optional nested answer objects
comments	Optional nested question comments
fetchedAt	Actor fetch timestamp

How much does it cost to scrape Stack Exchange questions?

The actor uses pay-per-event pricing.

There is a small start charge for each run and a per-question charge for each saved dataset item.

Current pricing is a $0.005 start event plus tiered per-question pricing. The BRONZE per-question price is $0.00003452, with lower per-question prices on higher usage tiers.

Keep maxQuestions small for exploratory runs.

Increase it when you are ready to build a larger dataset.

Input options

site

The Stack Exchange API site parameter.

Default: stackoverflow.

query

Free-text query for question search.

Example: apify web scraping.

tags

One or more tags to match.

Example: web-scraping, javascript, or python.

questionUrls

Specific question URLs to enrich.

The actor extracts numeric question IDs from /questions/{id}/... URLs.

questionIds

Specific numeric question IDs from the selected site.

fromDate and toDate

Optional creation-date filters.

Use YYYY-MM-DD or a full ISO timestamp.

sort and order

Choose Stack Exchange API sorting.

Supported sorts: activity, votes, creation, relevance.

Supported orders: desc, asc.

maxQuestions

Maximum number of question records to save.

Each saved question is one dataset item.

includeAnswers

Fetch nested answers for every saved question.

This is enabled by default.

includeComments

Fetch nested question comments.

This is disabled by default because it increases API calls.

apiKey

Optional Stack Apps API key for higher quotas.

Anonymous API access works for small and medium runs.

Example input: keyword and tag search

{
  "site": "stackoverflow",
  "query": "apify web scraping",
  "tags": ["web-scraping"],
  "maxQuestions": 25,
  "includeAnswers": true,
  "includeComments": false,
  "sort": "activity",
  "order": "desc"
}

Example input: direct question URLs

{
  "site": "stackoverflow",
  "questionUrls": [
    "https://stackoverflow.com/questions/11227809/why-is-processing-a-sorted-array-faster-than-processing-an-unsorted-array"
  ],
  "includeAnswers": true,
  "includeComments": true,
  "maxQuestions": 10
}

Example input: another Stack Exchange site

{
  "site": "serverfault",
  "query": "nginx reverse proxy timeout",
  "maxQuestions": 50,
  "includeAnswers": true
}

Example output

{
  "site": "stackoverflow",
  "questionId": 123456,
  "questionUrl": "https://stackoverflow.com/questions/123456/example",
  "title": "Example Stack Overflow question",
  "tags": ["javascript", "web-scraping"],
  "ownerDisplayName": "Example user",
  "score": 12,
  "viewCount": 3456,
  "answerCount": 3,
  "isAnswered": true,
  "answers": [
    {
      "answer_id": 987654,
      "question_id": 123456,
      "is_accepted": true,
      "score": 8,
      "body": "<p>Example answer body</p>"
    }
  ],
  "fetchedAt": "2026-07-10T00:00:00.000Z"
}

How to run

1. Choose a Stack Exchange site.

2. Enter a query, tags, direct question URLs, or question IDs.

3. Set maxQuestions.

4. Decide whether to include answers and comments.

5. Start the actor.

6. Export the dataset as JSON, CSV, Excel, XML, or HTML.

Tips for better results

Use tags when you need high precision.

Use a broad query when you are discovering unknown terminology.

Use sort: "votes" when you want evergreen high-signal questions.

Use sort: "activity" when you want recent discussion.

Disable comments for broad runs unless comments are required.

Add an API key for larger recurring workflows.

Integrations

Connect this actor to a vector database to build technical support retrieval datasets.

Send output to Google Sheets for community monitoring.

Load JSON exports into a data warehouse for trend analysis.

Trigger scheduled runs to monitor new questions about your product or SDK.

Use webhooks to notify developer relations teams when new matching questions appear.

API usage with Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/stack-exchange-questions-answers-scraper').call({
  site: 'stackoverflow',
  query: 'apify web scraping',
  tags: ['web-scraping'],
  maxQuestions: 25,
  includeAnswers: true
});
console.log(run.defaultDatasetId);

API usage with Python

from apify_client import ApifyClient
import os

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/stack-exchange-questions-answers-scraper').call(run_input={
    'site': 'stackoverflow',
    'query': 'apify web scraping',
    'tags': ['web-scraping'],
    'maxQuestions': 25,
    'includeAnswers': True,
})
print(run['defaultDatasetId'])

API usage with cURL

curl -X POST "https://api.apify.com/v2/acts/automation-lab~stack-exchange-questions-answers-scraper/runs?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"site":"stackoverflow","query":"apify web scraping","maxQuestions":25,"includeAnswers":true}'

MCP usage

Use the Apify MCP server with this actor enabled.

Claude Code example:

$claude mcp add apify https://mcp.apify.com/?tools=automation-lab/stack-exchange-questions-answers-scraper

Claude Desktop configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=automation-lab/stack-exchange-questions-answers-scraper"
    }
  }
}

Example prompts

Use these prompts after adding https://mcp.apify.com/?tools=automation-lab/stack-exchange-questions-answers-scraper to your MCP client:

• "Run automation-lab/stack-exchange-questions-answers-scraper for recent Stack Overflow questions about Apify web scraping and summarize recurring issues."
• "Use the Stack Exchange Questions & Answers Scraper MCP tool to extract top-voted Server Fault questions about nginx proxy timeouts."
• "Call the scraper for questions tagged web-scraping, include answers, then build a JSON dataset of accepted-answer patterns."
• "Find Stack Overflow questions about Playwright timeouts from the last year and list the most common fixes."

Rate limits and quotas

The Stack Exchange API applies quotas and may return a backoff instruction.

The actor waits when the API requests backoff.

Anonymous access is suitable for small and medium runs.

For larger recurring runs, add a Stack Apps API key.

Dataset design

The actor stores one row per question.

Answers and comments are nested in the question row when enabled.

This keeps exports simple for knowledge-mining workflows.

It also avoids forcing users to join separate datasets before sending records to a search index or LLM pipeline.

Common workflows

Developer community intelligence

Monitor new questions mentioning your product, SDK, or API.

Track which tags and error phrases appear most often.

Prioritize docs updates based on view counts and scores.

Support knowledge mining

Collect public questions and accepted answers around common errors.

Use output as seed material for internal support macros.

Identify outdated answers that may need official clarification.

Technical content research

Find high-vote questions before writing tutorials.

Review accepted answers and unanswered threads.

Use tags and date filters to understand topic evolution.

RAG and search datasets

Export question and answer bodies as JSON.

Clean HTML downstream if your embedding pipeline requires plain text.

Store question IDs and URLs for citation.

Troubleshooting

The run returns no items

Check that your site value is valid.

Try a broader query or remove strict tags.

For direct URLs, make sure they belong to the selected site.

The API reports quota or throttling

Reduce maxQuestions for the current run.

Disable comments to reduce request volume.

Add a Stack Apps API key for higher quotas.

Dates do not work

Use YYYY-MM-DD or a full ISO timestamp.

The actor filters by question creation date.

Answers are missing

Make sure includeAnswers is enabled.

Some questions have no answers or deleted answers that the public API cannot return.

Limitations

The actor returns public data exposed by the Stack Exchange API.

It cannot access deleted posts, private moderator data, or account-only content.

The question body and answer body are returned as HTML from the API.

Very large runs are subject to Stack Exchange API quotas.

Legality and ethics

This actor uses the public Stack Exchange API.

Respect Stack Exchange API terms and attribution requirements.

Do not use the data to republish private or sensitive information.

Review your use case for privacy, copyright, and database-rights obligations.

Related scrapers

Explore related Automation Lab actors for developer, content, and web data workflows:

• https://apify.com/automation-lab/github-marketplace-scraper
• https://apify.com/automation-lab/npm-package-intelligence-scraper
• https://apify.com/automation-lab/pubmed-search-scraper
• https://apify.com/automation-lab/webpage-text-extractor

FAQ

Can I scrape Stack Overflow only?

Yes. Use the default site value, stackoverflow.

Can I scrape other Stack Exchange communities?

Yes. Set site to the API site value for that community.

Do I need a Stack Exchange account?

No. Anonymous public API access works for normal small runs.

Do I need proxies?

No. The actor calls the public API directly.

Can I include comments?

Yes. Enable includeComments, but remember that it increases API calls.

Can I monitor questions daily?

Yes. Schedule the actor with a query, tags, and date strategy that matches your monitoring workflow.

Can I use the data for RAG?

Yes. The output includes question URLs, titles, body HTML, answer bodies, and metadata suitable for downstream cleaning and indexing.

Changelog

See the actor changelog for release notes.