Q&A Knowledge Extractor (Stack Exchange)

Extract clean, production-ready Q&A datasets from Stack Overflow and the entire Stack Exchange network for RAG, fine-tuning, and AI agent development.

This Apify Actor fetches question-answer pairs from the Stack Exchange API and delivers them as RAG-ready JSON records with full licensing attribution, quality metadata, and incremental diff support. Each pair is coupled (question ↔ best/accepted answer), normalized for immediate use, and includes code examples intact.

Perfect for building retrieval-augmented generation (RAG) systems, fine-tuning language models, training AI agents, and growing proprietary knowledge bases without maintenance overhead.

---

What You Get

• Coupled Q&A pairs: Each record contains one question and its best (accepted or highest-scoring) answer, ready to use as a training example or RAG context window.
• Code-safe formatting: Markdown with fenced code blocks preserved intact — no corrupted Python snippets or mangled SQL.
• Full attribution: Every record includes author names, profile URLs, and exact license version per content. Comply with CC BY-SA licensing automatically.
• Incremental extraction: Run the Actor multiple times. Only new or updated Q&A pairs are fetched and charged — grow your dataset without re-processing old content.
• RAG chunking (optional): Automatically split answer text into overlapping chunks on natural boundaries (paragraphs, code blocks) for vector embedding and retrieval.
• Quality filters: Minimum score thresholds, tag filtering, and accepted-answer-only mode eliminate low-quality noise.

---

Use Cases

1. Retrieval-Augmented Generation (RAG)

Build knowledge-grounded chatbots and search systems that cite real Stack Overflow solutions. The Actor provides clean, pre-chunked context windows ready to embed.

User query: "How to parse JSON in Python?"
↓ RAG retrieval
→ Returns top-scoring Stack Overflow answers + metadata
→ LLM generates response citing sources

2. Fine-tuning Language Models

Create domain-specific instruction datasets by filtering on tags (Python, React, databases) and score thresholds. Each Q&A pair becomes a training example.

Example: Fine-tune a model on production Docker best practices by filtering tag: "docker" and minQuestionScore: 10.

3. AI Agents & Multi-Tool Learning

Equip agents with task-specific knowledge bases. The Actor outputs clean, parseable records agents can query during reasoning.

Agent: "I need to debug a Flask authentication issue."
→ Queries the local Q&A dataset
→ Returns 5 relevant Stack Overflow answers
→ Incorporates into reasoning chain

4. Internal Knowledge Base

Populate a company knowledge base with Stack Overflow solutions relevant to your tech stack. Incremental mode keeps it fresh without re-scraping.

5. Academic Research

Extract Q&A datasets for studying software engineering practices, API design patterns, or how developers solve real problems at scale.

---

Key Features

Feature	Details
Data source	Stack Exchange API v2.3 (official, stable)
Supported sites	Stack Overflow, Server Fault, Super User, Ask Ubuntu, and 200+ other Stack Exchange sites
Q&A coupling	Automatic pairing of questions with best/accepted answers
Incremental mode	Store a high-water-mark; next run only fetches new/updated pairs — save money & time
Filtering	Tags (AND), score thresholds, date ranges, free-text search, accepted-answer-only
Output schema	Structured JSON with question metadata, answer body, licensing, attribution, optional chunks
Code handling	Markdown with fenced code blocks intact — never corrupts code samples
License compliance	CC BY-SA attribution built into every record; seamless license version detection
RAG-ready	Optional chunking on paragraph/code-block boundaries; overlap support for context preservation
Pricing	Pay-per-result: $0.005 per new/updated Q&A pair; incremental mode means you only pay once
Error handling	Graceful quota management, schema-drift detection, canary sanity checks

---

Example: Input & Output

Input Configuration

{
  "site": "stackoverflow",
  "tags": ["python", "pandas"],
  "query": "how to merge dataframes",
  "minQuestionScore": 5,
  "minAnswerScore": 10,
  "acceptedOnly": true,
  "incremental": true,
  "enableChunking": false,
  "maxItems": 100
}

Output Record (JSON)

{
  "_schemaVersion": 1,
  "site": "stackoverflow",
  "questionId": 11227809,
  "question": {
    "title": "Why is processing a sorted array faster than an unsorted array?",
    "bodyMarkdown": "A... branch misprediction explanation... ```code block``` preserved.",
    "tags": ["c++", "performance", "cpu-cache"],
    "score": 27000,
    "viewCount": 1900000,
    "createdAt": "2011-06-27T13:51:36Z",
    "lastActivityAt": "2024-02-10T08:00:00Z",
    "url": "https://stackoverflow.com/q/11227809",
    "author": {
      "name": "GManNickG",
      "url": "https://stackoverflow.com/users/123456/gmannickG"
    }
  },
  "answer": {
    "answerId": 11227902,
    "bodyMarkdown": "Excellent explanation of cache lines and branch predictors... ```code``` intact.",
    "score": 35000,
    "isAccepted": true,
    "createdAt": "2011-06-27T13:56:42Z",
    "url": "https://stackoverflow.com/a/11227902",
    "author": {
      "name": "Mysticial",
      "url": "https://stackoverflow.com/users/555555/mysticial"
    }
  },
  "license": {
    "name": "CC BY-SA 4.0",
    "url": "https://creativecommons.org/licenses/by-sa/4.0/"
  },
  "attribution": "Question by GManNickG (https://stackoverflow.com/users/123456/gmannickG) and answer by Mysticial (https://stackoverflow.com/users/555555/mysticial) on Stack Overflow, licensed under CC BY-SA 4.0.",
  "scrapedAt": "2024-06-07T10:00:00Z"
}

---

Licensing & Attribution

This Actor respects CC BY-SA licensing. Every output record includes:

1. License metadata (license object) with the correct CC version (4.0 for content created after May 2, 2018; 3.0 for older content).
2. Attribution string (attribution field) listing question author, answer author, Stack Overflow URL, and license version.

Your Responsibilities

• Include the attribution in any dataset you publish or distribute.
• Maintain the CC BY-SA license on the Q&A content when sharing your output dataset. (Your model, RAG application, or analysis doesn't have to be CC BY-SA — only the underlying Q&A text does.)
• Do not use nofollow or obfuscate links to the original questions.

Stack Overflow & Stack Exchange Terms

• Commercial use via the API is allowed per the Stack Exchange API Terms of Service and Attribution Required blog post.
• The Actor respects API rate limits and quota: ~300 requests/day per IP without an API key, ~10,000/day with a free key (register at stackapps.com).
• See the licensing discussion in the PRD for full details.

---

Getting Started

1. Configure Your Input

Choose your data source and filters:

Parameter	Purpose	Example
site	Stack Exchange site to query	"stackoverflow", "serverfault"
tags	Filter by tags (AND)	["python", "pandas"]
query	Free-text search	"memory leak"
minQuestionScore	Minimum question score	5
minAnswerScore	Minimum answer score	10
acceptedOnly	Only paired with accepted answers	true
incremental	Store high-water-mark for delta runs	true
enableChunking	Split answers into RAG chunks	false (set true if embedding)
chunkSize	Target chunk size (chars)	1200
maxItems	Hard limit on results	0 (no limit)
apiKey	Optional Stack Exchange app key	—

2. Run the Actor

Use the Apify Console or CLI:

$apify run

Or via the Apify platform: click "Start actor" in the store listing.

3. Retrieve Results

The Actor pushes Q&A pairs to the Apify Dataset. Download as JSON, CSV, or access via API:

$apify dataset get-items

4. Use in Your Application

RAG Example (pseudo-code)

# Load the dataset
qapairs = load_dataset('apify_results.json')

# Embed and index for retrieval
for pair in qapairs:
  # Chunk the answer if needed
  chunks = pair.get('chunks') or [pair['answer']['bodyMarkdown']]
  for chunk in chunks:
    embedding = embedding_model.encode(chunk)
    index.add(embedding, metadata={'pair_id': pair['questionId']})

# At query time
user_query = "How to optimize pandas merge?"
query_embedding = embedding_model.encode(user_query)
retrieved = index.search(query_embedding, top_k=5)
# Pass retrieved to LLM for RAG

---

FAQ

Q: How much does it cost?

A: $0.005 per new or updated Q&A pair. With incremental mode enabled (default), subsequent runs only charge for new pairs — a dataset of 10,000 pairs costs ~$50 to build once, then additional updates cost only for the new pairs added.

Q: Can I use this data commercially?

A: Yes. The Stack Exchange content is CC BY-SA licensed; you may use it commercially in closed or open applications, provided you include attribution (which the Actor does automatically for each record).

Q: What's the incremental mode?

A: The Actor stores the latest activity date from each run. Next run fetches only Q&A pairs updated after that date, deduplicates them, and charges only for new/changed pairs. Grow your dataset without re-processing.

Q: Can I use multiple Stack Exchange sites in one run?

A: Not yet — one site per run. Use multiple Actor runs with different site parameters to multi-source (or contact support for multi-site feature request).

Q: How often does the Stack Exchange API update?

A: Continuously. Questions and answers are updated in real-time; the Actor can refresh your dataset as often as you want (respecting the rate-limit quota).

Q: Do code blocks stay intact?

A: Yes. The Actor fetches body_markdown (not HTML), preserving fenced code blocks (```python ```) exactly as Stack Overflow displays them. HTML entities are decoded for readability.

Q: What if there's no accepted answer?

A: If acceptedOnly=false, the Actor pairs the question with its highest-scoring answer instead. If acceptedOnly=true (default), questions without accepted answers are skipped.

Q: Can I chunk the Q&A text for embeddings?

A: Yes. Enable enableChunking: true and set chunkSize (default 1200 chars). The Actor splits text on paragraph and code-block boundaries, never mid-block. Each record gets a chunks array ready to embed.

Q: What if the Stack Exchange API breaks?

A: The Actor includes a canary check (validates against a known-good answer) and schema-drift detection. If something changes, you'll see a [Canary] FAILED warning in logs — not a silent failure.

---

Technical Details

• Language: Node.js 20+ (TypeScript)
• HTTP client: got-scraping with automatic gzip decompression, retries, and quota awareness
• Tests: 138+ unit + integration tests via Vitest; all fixtures use real Stack Exchange API responses
• Error handling: Graceful quota exhaustion, schema-drift warnings, per-record isolation
• Build: Multi-stage Docker (builder installs all deps, runtime installs prod deps only)

---

Support & Issues

• Stack Exchange API docs: https://api.stackexchange.com/docs
• Register a free API key: https://stackapps.com/apps/oauth/register
• Attribution requirements: https://stackoverflow.blog/2009/06/25/attribution-required/
• CC BY-SA 4.0 license: https://creativecommons.org/licenses/by-sa/4.0/

For Actor-specific issues or feature requests, open an issue on the project GitHub or contact the maintainer.

---

Summary

Use Q&A Knowledge Extractor to build better RAG systems, fine-tune smarter models, and empower your AI agents with real, production-proven solutions. Clean data, transparent pricing, automatic licensing — from Stack Overflow to your application in minutes.

Start extracting now. 🚀