Stack Overflow / Stack Exchange Scraper

A tag filter on Stack Exchange is an AND, not an OR. Ask for javascript and promise and only the questions carrying both come back, which is why two tags is usually the whole query.

Query by keyword, by tag, or both, and the questions arrive as flat JSON with the body already converted to plain text. The site field switches it to Server Fault, Super User, Ask Ubuntu, MathOverflow or Software Engineering instead.

It reads the official Stack Exchange API. Read the quota section before you plan a big run. That's the one real constraint here.

What you get back

One row per question:

{
  "ok": true,
  "questionId": 14220321,
  "title": "How to await an async function in a constructor?",
  "url": "https://stackoverflow.com/questions/14220321/...",
  "score": 3184,
  "answerCount": 42,
  "viewCount": 2916440,
  "isAnswered": true,
  "tags": ["javascript", "async-await", "promise"],
  "ownerName": "somebody",
  "ownerReputation": 51204,
  "createdAt": "2013-01-09T09:15:29.000Z",
  "body": "I have a class that needs to..."
}

body is the question text with the HTML markup removed and code blocks kept on their own lines. ownerName and ownerReputation are null for deleted or anonymous users.

Answers are not in the payload. You get answerCount and isAnswered, so you know how many answers a question has and whether one was accepted. You don't get the answer text. If that's what you came for, this isn't the actor.

Input

{
  "query": "async await",
  "site": "stackoverflow",
  "tags": "javascript,promise",
  "sortBy": "votes",
  "maxItems": 100
}

Field	Default	Notes
query	empty	Keywords matched against title and body. Can be empty if you supply tags.
site	stackoverflow	Also serverfault, superuser, askubuntu, mathoverflow, softwareengineering.
tags	empty	Comma-separated. A question must carry all of them, not any.
sortBy	votes	votes, relevance, creation or activity. Always descending.
maxItems	50	1 to 1,000. Pulled 100 per request.
proxyConfiguration	off	The API is open. A proxy only helps if your IP is the thing being throttled.

Either query or tags will do on its own, but you need at least one of them. The Console form arrives with async await and javascript filled in as an example; an API call sends neither unless you put them there.

Tags alone is a good way to get the top-voted questions in a technology without guessing keywords.

The quota

Unauthenticated calls to the Stack Exchange API get 300 requests per day, counted per IP address. At 100 questions a request that's about 30,000 questions a day, shared with anything else running from the same address. The actor reads quota_remaining off every response, logs it, and stops paginating when it hits zero rather than hammering a closed door.

It also honours the API's backoff field, so if Stack Exchange asks for a pause the run sleeps instead of getting throttled harder. Long runs may sit idle for a few seconds at a time because of this.

If you get throttled anyway, the run emits a diagnostic row with errorCode: RATE_LIMITED and stops. Nothing is charged for it.

Billing

$2.00 per 1,000 questions, so $0.002 each, with no run-start fee. Only questions actually returned are billed. Diagnostic rows and empty results cost nothing.

Delivery

Results go to the run's dataset as JSON, CSV, Excel or XML. Set notionConnector and notionParentId and each question is also written as a Notion page after the run. Authorize the connector once under Settings → API & Integrations → MCP connectors. Leave them empty and it's skipped.

What people do with it

Mostly to find the questions their own docs should have answered. Search your product name, sort by votes, read what keeps coming up.

Tag volume is also a cheap demand signal. Run the same query against react and against vue and compare how many questions come back.

The third use is building evaluation sets. body is already plain text, so it works as a prompt straight out of the dataset, and score gives you a rough quality weight to sort on.