Stack Exchange Questions and Answers Scraper

Search selected Stack Exchange sites and export normalized questions and answers with bodies, tags, scores, authors, acceptance status, dates, and source links.

This Actor is designed for recurring developer research datasets, Q&A trend analysis, knowledge-base preparation, and source-backed topic monitoring.

It uses the official public Stack Exchange API and does not require a Stack Exchange login.

What does Stack Exchange Questions and Answers Scraper do?

The Actor turns Stack Exchange search results into integration-ready dataset rows.

You can:

• search one or several Stack Exchange network sites;
• combine a text query with tags, author, score, and date filters;
• fetch known Stack Overflow or Stack Exchange question URLs;
• include complete answer bodies and acceptance state;
• preserve both source HTML and readable plain text;
• schedule the same input to build a repeatable research dataset;
• export results as JSON, JSONL, CSV, XML, Excel, or RSS through Apify.

Question and answer rows use one stable schema, with recordType identifying the entity.

Who is this Stack Exchange questions and answers export for?

Developer-relations teams can identify recurring questions around an API, SDK, or framework.

Product researchers can compare highly voted questions, accepted answers, and topic activity.

Data teams can create normalized Q&A datasets without maintaining API pagination and backoff logic.

Technical writers can find source-linked questions that reveal documentation gaps.

AI and search teams can collect public text with IDs and canonical provenance for evaluation or retrieval workflows.

Why use this Actor?

The output keeps questions and answers separate while retaining their relationship.

Every answer includes its parent question ID and question title.

Every row includes a canonical source link.

All search and URL inputs use the same filters.

The Actor honors Stack Exchange API backoff instructions and applies bounded retries to temporary failures.

It stops at your requested question limit and does not use browser or proxy traffic.

Supported Stack Exchange sites

Use the site's API identifier in sites.

Common values include:

• stackoverflow
• math
• serverfault
• superuser
• askubuntu
• salesforce
• worldbuilding
• softwareengineering

For sites hosted at <name>.stackexchange.com, the identifier is usually <name>.

Stack Overflow, Server Fault, Super User, and Ask Ubuntu use their familiar identifiers shown above.

What data can be exported?

Field	Meaning
recordType	question or answer
site	Stack Exchange API site identifier
questionId	Source question ID
answerId	Answer ID, or null for questions
parentQuestionId	Parent question ID on answer rows
title	Question title
bodyHtml	Original API body HTML
bodyText	Readable plain-text body
tags	Parent question tags
score	Question or answer score
viewCount	Question views
answerCount	Number of question answers
isAnswered	Stack Exchange answered state
isAccepted	Accepted state for an answer
acceptedAnswerId	Accepted answer ID when available
authorId	Public numeric author ID
authorName	Public display name
authorUrl	Public profile URL
authorReputation	Reputation reported by the API
createdAt	ISO 8601 creation time
lastActivityAt	ISO 8601 activity time
lastEditedAt	ISO 8601 edit time, when present
sourceUrl	Canonical question or answer URL
query	Query that produced the row
fetchedAt	Retrieval timestamp

Fields unavailable from the source are returned as null rather than invented.

How to run the Actor

1. Open the Actor in Apify Console.
2. Add one or more Stack Exchange site identifiers.
3. Enter a query, tags, an author ID, or known question URLs.
4. Choose whether to include answers.
5. Set the maximum number of questions.
6. Click Start.
7. Open the Dataset tab to preview or download the rows.

Start with 5 questions while refining filters, then increase the limit for production runs.

Input parameters

Input	Type	Default	Description
sites	string[]	['stackoverflow']	Up to 10 sites searched in order
query	string	—	Words in question title or body
tags	string[]	—	Up to 5 required tags
authorId	integer	—	Question author user ID
minScore	integer	—	Minimum question score
dateFrom	date	—	Earliest creation date
dateTo	date	—	Latest creation date
questionUrls	string[]	—	Known public question URLs
includeAnswers	boolean	true	Add answer rows
maxQuestions	integer	25	Question limit, 1–500
sort	string	relevance	Relevance, activity, creation, or votes
order	string	desc	Descending or ascending
maxRequestRetries	integer	2	Temporary request retries, 0–5

Provide at least one of query, tags, authorId, or questionUrls.

Search filters are combined with AND.

All supplied tags must be present.

Answer rows do not count against maxQuestions.

Example input: API topic research

{
  "sites": ["stackoverflow"],
  "query": "Stack Exchange API",
  "includeAnswers": true,
  "maxQuestions": 5,
  "sort": "relevance"
}

This input exports up to five matching questions plus their available answers.

Example input: tag and score monitoring

{
  "sites": ["stackoverflow"],
  "query": "fetch",
  "tags": ["javascript"],
  "minScore": 5,
  "includeAnswers": true,
  "maxQuestions": 10,
  "sort": "votes",
  "order": "desc"
}

Save this input as an Apify Task and schedule it to create periodic snapshots.

Example output

{
  "recordType": "answer",
  "site": "stackoverflow",
  "questionId": 12345678,
  "answerId": 12345679,
  "parentQuestionId": 12345678,
  "title": "How can I retry a fetch request?",
  "bodyHtml": "<p>Use a bounded retry loop...</p>",
  "bodyText": "Use a bounded retry loop...",
  "tags": ["javascript", "fetch"],
  "score": 12,
  "viewCount": null,
  "answerCount": null,
  "isAnswered": null,
  "isAccepted": true,
  "acceptedAnswerId": 12345679,
  "authorId": 246810,
  "authorName": "Example Author",
  "authorUrl": "https://stackoverflow.com/users/246810/example-author",
  "authorReputation": 4200,
  "createdAt": "2025-01-15T12:00:00.000Z",
  "lastActivityAt": "2025-01-16T09:30:00.000Z",
  "lastEditedAt": null,
  "sourceUrl": "https://stackoverflow.com/questions/12345678/example#12345679",
  "query": "fetch",
  "fetchedAt": "2025-01-17T10:00:00.000Z"
}

The example uses anonymized values. Actual output contains current public source data.

How much does it cost to export Stack Exchange questions and answers?

The Actor uses pay-per-event pricing.

A one-time Start event is charged once per run.

An Item extracted event is charged for each useful question or answer row written to the dataset.

Failed, rejected, duplicate, and empty records are not charged as items.

The current BRONZE price is $0.001 per run start plus $0.0016 per item. Higher and lower account tiers are shown in Console before a run.

For planning, total Actor charge is:

$0.001 + (question rows + answer rows) × $0.0016 at BRONZE.

Example BRONZE Actor charges are approximately $0.009 for 5 rows, $0.041 for 25 rows, and $0.161 for 100 rows. Platform infrastructure usage, if applicable to your account, is shown separately by Apify.

Because answers are separate useful rows, a five-question run can contain more than five billable items.

Use includeAnswers: false when your workflow needs only question metadata and bodies.

Using explicit question URLs

Known question URLs bypass search discovery.

Supported examples include:

• https://stackoverflow.com/questions/QUESTION_ID/...
• https://math.stackexchange.com/questions/QUESTION_ID/...
• https://serverfault.com/questions/QUESTION_ID/...
• https://superuser.com/questions/QUESTION_ID/...

The Actor derives the API site and question ID from each URL.

Query, tag, author, score, and date filters still apply to fetched questions.

Invalid or unsupported hosts fail closed with a clear input error.

Recurring monitoring workflow

Create an Apify Task with a stable query and filters.

Schedule it daily or weekly.

Export each run's dataset to your warehouse or webhook destination.

Use questionId, answerId, lastActivityAt, and lastEditedAt as comparison keys.

The Actor returns snapshots; it does not maintain history or send alerts by itself.

Apify schedules, webhooks, and integrations provide the automation layer.

API usage with cURL

Replace APIFY_TOKEN with your token in your environment.

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~stackexchange-questions-answers-export/runs?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sites":["stackoverflow"],"query":"Stack Exchange API","maxQuestions":5,"includeAnswers":true}'

To wait for results synchronously, use the run-sync-get-dataset-items endpoint.

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~stackexchange-questions-answers-export/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"sites":["math"],"tags":["calculus"],"maxQuestions":5}'

JavaScript API example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/stackexchange-questions-answers-export').call({
  sites: ['stackoverflow'],
  query: 'fetch',
  tags: ['javascript'],
  minScore: 5,
  includeAnswers: true,
  maxQuestions: 10,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Python API example

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/stackexchange-questions-answers-export').call(run_input={
    'sites': ['stackoverflow'],
    'query': 'Stack Exchange API',
    'includeAnswers': True,
    'maxQuestions': 5,
})
items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

Connect through MCP

Add the Actor to Claude Code:

claude mcp add --transport http apify \
  "https://mcp.apify.com?tools=automation-lab/stackexchange-questions-answers-export"

Claude Desktop setup

Add this server object to Claude Desktop's MCP configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/stackexchange-questions-answers-export"
    }
  }
}

Cursor setup

Open Cursor settings, choose Tools & MCP, add a remote MCP server, and use the same Apify MCP URL shown above.

VS Code setup

Open VS Code's MCP configuration, add an HTTP server named apify, and use the same Actor-specific URL. Confirm that MCP support is enabled in your VS Code installation.

Example prompts:

• “Search Stack Overflow for recent questions about the Stack Exchange API and include answers.”
• “Export high-scoring JavaScript fetch questions and accepted answers.”
• “Collect five recent calculus questions from Math Stack Exchange.”

Integrations

Send completed datasets to Google Sheets for editorial research.

Load JSONL into a warehouse for longitudinal analysis.

Trigger a webhook when a scheduled run completes.

Use Zapier or Make to route high-scoring questions to a review queue.

Use sourceUrl to preserve human-verifiable provenance in downstream systems.

Reliability and retries

The Actor uses the official Stack Exchange API rather than page scraping.

It validates response shapes before producing data.

Temporary network and API failures are retried with bounded exponential delay.

When the API asks clients to back off, the Actor waits for the requested interval.

Deterministic input errors are not retried.

A run fails after retry exhaustion rather than returning misleading partial success.

Limits and source behavior

Anonymous Stack Exchange API requests have a shared quota reported by the source.

Large or frequent jobs may encounter throttling.

The Actor supports up to 500 questions per run.

The API may omit deleted users, deleted posts, or fields unavailable to anonymous clients.

Search relevance and answered state are defined by Stack Exchange.

Result counts can change as posts are edited, scored, answered, or removed.

Answers are requested in batches of up to 100 question IDs.

No login-only, private, or deleted content is retrieved.

Responsible use and legality

Stack Exchange content is publicly accessible but remains subject to source terms and content licensing.

Review the applicable Stack Exchange Terms of Service and content attribution requirements before republishing data.

Keep sourceUrl and author attribution when your use requires them.

Do not use public profile information for harassment, spam, or sensitive profiling.

Comply with privacy, database, copyright, and AI-training rules that apply to your jurisdiction and use case.

You are responsible for your inputs and downstream use.

Troubleshooting

The run says I must provide a search input.

Add at least one of query, tags, authorId, or questionUrls.

A site identifier fails.

Use the Stack Exchange API site parameter, such as stackoverflow or math, not a full URL.

I received fewer questions than requested.

The source may have exhausted matching results, or filters may be narrow. Check tags, dates, author ID, and score.

There are more rows than maxQuestions.

The limit counts questions. Each included answer is saved as a separate row.

A temporary API error exhausted retries.

Run again later, reduce frequency, or split a large multi-site job into scheduled Tasks.

An answer has no author ID.

The source can omit owner information for removed accounts. Nullable fields preserve that state.

FAQ

Does this scrape every Stack Exchange site?

It searches the site identifiers you provide. It does not crawl the entire network automatically.

Can I fetch one known question?

Yes. Put its public question URL in questionUrls and set maxQuestions to at least 1.

Are answer bodies included?

Yes when includeAnswers is true. Both HTML and plain text are returned.

Can I filter answers by score?

minScore filters questions. Answers are exported with their scores but are not independently filtered.

Does the Actor deduplicate historical runs?

No. Each run is an independent snapshot. Deduplicate downstream by site plus question or answer ID.

Does it need a proxy or Stack Exchange account?

No. It calls the public API directly.

Can I use the output for RAG?

The normalized text and canonical links are suitable building blocks, but you must handle licensing, attribution, chunking, and retrieval quality for your application.

Related Automation Lab Actors

For scholarly literature rather than developer Q&A, use ArXiv Papers Scraper.

For biomedical literature search, use PubMed Article Search Scraper.

For extracting supplied article pages across domains, use Multi-Site Article Content Extractor.

These Actors solve adjacent research workflows; they are not Stack Exchange data sources.