Quora Questions Scraper: scrape Quora question pages by URL, topic, or keyword. Get question text, topic tags, and answer bodies with author, upvotes, and timestamps. Public pages only, no login or Quora+ required. QA forum scraper. Question answer dataset export tool.
0.2.4 — 2026-09-29 (banked on branch; unshippable until the billing block lifts)
Fix (latent structural match, no live customer signature readable): a
concurrent-rotation race in QuoraSession._rotate() that could FAIL the
run.main.py shares each QuoraSession instance across up to
CONCURRENCY (3) in-flight fetch() calls via
sessions[index % len(sessions)] — that round-robin is only race-free
when task completion order matches launch order, which
asyncio.gather does not guarantee. Two attempts could then observe a
challenge/403/429 response on the SAME underlying curl-cffi session at
once; the old _rotate() unconditionally close()d self._session
with no lock or generation guard, so a second concurrent rotation
closed an already-dead handle, which raises inside curl_cffi
(TypeError — "cdata pointer") uncaught, above the except RuntimeError
guard in main._process_question, crashing the whole run. This is the
same defect fixed in offerup-listings-scraper (commit 8a9bcc97,
Apify QA run SZEHSLbsEVm1L0C9P) and flagged fleet-wide in
ops/os/DATED-ACTIONS.md ("The same defect is a TEMPLATE bug") — our
own August run logs read clean because customer runs are the ones that
race (reference-customer-run-logs-unreadable), so no readable log
carries the "cdata pointer" / "Session is closed" signature; this ships
as a mechanical port ahead of any reproduced live failure.
_rotate(stale) is now lock-guarded and only rotates when stale still
matches the current session (a generation check), teardown goes through
a best-effort _close_quietly, and aclose() uses the same
best-effort path. Three new regression tests in tests/test_client.py
port the OfferUp shape: concurrent challenges rotate exactly once, a
raising close() never propagates, and a stale rotate token is a
no-op that leaves the live session untouched. 70 tests green, ruff +
pyright clean.
0.2.3 — 2026-08-31
Cloud QA on build 0.2.2 reached SUCCEEDED with 3 real rows (run
5A12dw3zifEfsU6R5) — the first genuinely working cloud run — but
eyeballing the dataset found two silent data-quality bugs the parser
tests hadn't caught because the earlier fixture's structure was
simplified in a way that hid them:
answer_author_name shipped as "" on every row. Each answer card
actually carries TWO a[href*="/profile/"] links — a document-order-
first avatar-only link with empty text, then the name-bearing link.
css_first always grabbed the avatar. New _author_link_with_name
prefers whichever profile link actually has visible text.
answer_upvotes was None on every row. The upvote button does
NOT live inside the same ancestor as the profile link — it sits TWO
levels further out. _answer_card_ancestor required only a profile
link, so it returned a scope too small to ever contain the upvote
button. It now requires BOTH a profile link and the upvote button
before accepting an ancestor as the answer's card.
tests/fixtures/question_page.real_structure.html rebuilt to actually
reproduce both structural facts (two profile links, upvote button two
levels out) instead of the simplified single-level card the previous
fixture used — which is exactly why these two bugs passed local tests
but failed on real data. New assertions lock in the real author name
and upvote count. 67 tests green, ruff clean, pyright clean.
0.2.0 — 2026-08-31
Root cause of the 0/0 rows that closed out 2026-08-30's investigation:
0.1.3's cloud run (P4dbLhqPt6uFChk1l) confirmed Camoufox clears
Quora's Cloudflare challenge, but the JSON-LD-primary parser design in
design.md turned out to be a wrong guess — it just failed silently
instead of loudly (question parsed via H1, suggestedAnswer empty, 0
answer rows, run correctly failed loud on REQ-6 rather than reporting a
hollow success).
Ground-truthed against a REAL page this time (src/recon_capture.py,
a one-off Camoufox cloud capture persisted to KVS — see
docs/specs/quora-questions-scraper/notes.md):
No JSON-LD, no __NEXT_DATA__, no usable data island exists on the
real render.window.ansFrontendGlobals only carries page config,
never question/answer content. The question and every answer are
plain DOM.
Quora's own puppeteer_test_* classes are the stable target —
test hooks the site ships for its own end-to-end tests, present on
the question title (puppeteer_test_question_title) and every answer
body (puppeteer_test_answer_content), unlike the surrounding
CSS-in-JS classes (s1e38h8g etc.) which are hashed and unusable.
src/parser.py's extraction priority is now: test-id CSS (primary,
ground-truthed) → JSON-LD (secondary, kept for resilience) → bare H1
guessed selectors (last resort, unchanged from before).
Two real bugs the guessed selectors had baked in, both found from
the recon capture:
The "Continue Reading" gate is a CSS toggle, not a real content
gate — Quora ships a short preview AND the complete answer text in
the same DOM node, joined by that marker. The old code always kept
the text before the marker and flagged it truncated; that
discarded the real content on every answer that had it.
_split_truncated now prefers the text after the marker, and
only falls back to "before + truncated=True" when nothing follows
(the genuinely login-gated case, still covered by the existing
question_page_gated.synthetic.html fixture).
The upvote count renders as a digit-rolling ticker: a same-length
visibility--hidden ghost span (a width placeholder) plus the real
visible span, both inside the same button. Reading all <span>
text concatenated them into a garbled number (9953 instead of the
real 53, confirmed against the recon capture). _extract_upvotes
now skips the hidden span.
Fault isolation, not misattribution. Author/timestamp/upvotes are
read from the nearest ancestor (within ANSWER_CARD_SEARCH_DEPTH = 6)
that links to an author profile. An unbounded walk on the real recon
page mis-attributed a "related answers" section's author to an
unrelated answer further up the page; bounding the search means that
case now correctly returns None for those fields while still
emitting the real answer text — a partial row, never a dropped one.
New fixture tests/fixtures/question_page.real_structure.html models
the real recon structure (not a byte-for-byte copy — trimmed for repo
size) and 3 new tests in tests/test_parser.py cover the primary path,
the full-text-after-marker behaviour, and the bounded-ancestor-search
fault isolation. 66 tests green (was 63), ruff clean, pyright clean.
.actor/pay_per_event.json (actor-start: $0.20,
result-scraped: $0.002
→ $2.20/1,000) is unchanged and already correct; still a
publish-time activation on the Console Monetization tab, out of this
fix's scope.
Second bug, found by cloud QA on build 0.2.1 (run JeBdVCVejlO6YWFvC,
FAILED): Camoufox cleared the Cloudflare challenge (status=200) but the
parser STILL found no title — a genuine hydration race, not a selector bug.
page.content() was captured right after wait_until="domcontentloaded" +
a fixed 1.5s sleep; Quora's Relay app (10+ webpack chunks) doesn't always
finish rendering the question/answer DOM in that window, even though the
same wait pattern happened to work during the manual recon capture.
src/browser.py::fetch_page now waits (best-effort, h1 marker, 8s cap,
never raises) for the page to actually hydrate before reading content() —
_wait_for_hydration. A timeout just means proceed with whatever DOM
exists; the parser layer still decides whether that's enough. 1 new test
(test_fetch_page_swallows_hydration_timeout_and_still_returns_html,
67 total). Pushed as build 0.2.2.
0.1.1 — 2026-08-30
Root cause of all 12/12 cloud FAILED runs, found by reading the actual
platform run logs (P4oObkxiIk9z7OdHw, then a fresh reproduction on build
0.0.2 — run ncnbKI3rjyndmAav5) instead of trusting the 2026-08-27
UNPROVEN writeup:
The 08-27 residential + Camoufox-escalation fix was never actually
live. It was merged to main but the account was over its usage cap
from 2026-08-27 through 2026-08-29, so apify push never ran — every
cloud run still executed the OLD build (0.0.1, datacenter proxy, no
Camoufox path). Re-pushed as build 0.0.2 now that the cap is lifted
($80 ceiling, $41.37 used).
A real bug the 08-27 fix didn't anticipate. Once actually run in the
cloud, ncnbKI3rjyndmAav5 showed curl-cffi rotate through 4 browser
profiles: 3 hit Cloudflare's managed challenge (cf-mitigated: challenge
/ "Just a moment"), but the 4th (safari180) got a bare 200 that
triggers neither the challenge nor geo-splash marker — confirmed by local
reproduction (RESIDENTIAL, US-pinned) that Quora served an un-hydrated
React/Relay shell (<noscript>Please enable Javascript..., an og:title
present but no server-rendered H1 or JSON-LD). Because the escalation to
Camoufox only fired on ChallengeExhaustedError, that "successful" shell
response never reached the browser engine, and the run died on a raw
parse error instead of ever trying the engine that executes JS.
src/main.py's _fetch_html now returns (html, used_browser);
_process_question routes a curl-cffi parse failure through one
Camoufox retry (_retry_parse_via_browser) before giving up. A parse
failure on browser-rendered HTML still fails outright — the content
genuinely isn't there at that point. 3 new/updated tests in
tests/test_main.py.
PPE pricing gap confirmed still open on the platform. Both cloud runs
logged
Ignored attempt to charge for an event — the Actor does not use the pay-per-event pricing
. The repo-side fix from 0.1.0
(.actor/pay_per_event.json + Actor.charge() calls in src/main.py)
is already correct and unchanged; activating PPE on the Apify Console
Monetization tab is a publish-time action out of this fix's scope.
.actor/actor.json version bumped 0.0 → 0.1 to match this
CHANGELOG's numbering (the 0.1.0 entry never actually bumped it).
See docs/specs/quora-questions-scraper/notes.md for the full
Camoufox-vs-Cloudflare probe results (local + cloud, run IDs and
billing-confirmed PROXY_RESIDENTIAL_TRANSFER_GBYTES).
0.1.0 — 2026-08-27 (UNPROVEN — see notes.md)
Prepared on a day the Apify account is over its monthly usage hard limit;
every fix below is local-green but not yet verified against the live
target or the platform. Two defects, both real:
Pricing..actor/pay_per_event.json set actor-start to $0.02
against the fleet standard of $0.20 (set 2026-08-20). The Actor was never
published, so this changes freely — README + scaffold.json synced to the
new effective rate, $2.20 / 1,000 results (result-scraped unchanged
at $0.002).
Proxy tier. Default was auto (datacenter); Quora's Cloudflare
managed challenge does not clear on it (confirmed by run
P4oObkxiIk9z7OdHw: challenge detected on attempt 0). proxyConfiguration
now defaults to apifyProxyGroups: ["RESIDENTIAL"] in both
.actor/input_schema.json (prefill + default) and src/models.py.
proxyCountryCode also gained a prefill (was default-only).
Engine. Direct recon (no proxy) confirmed Quora serves a genuine
Cloudflare JS-execution challenge, not just an IP-reputation wall — a
plain unproxied GET from this machine got cf-mitigated: challenge + a
"Just a moment" body. Added a Camoufox escalation path
(src/browser.py + src/browser_fetch.py) that main.py reaches for
only after curl-cffi's QuoraSession exhausts retries against a real
challenge — curl-cffi stays the cheap default for everything else.
Diagnostics.src/client.py now tallies every retry attempt's
outcome instead of just the last one, and raises a new
NetworkExhaustedError distinct from ChallengeExhaustedError — run
P4oObkxiIk9z7OdHw mixed 2 real 30s connect timeouts among 3 challenge
hits, and the old code's final exception erased that mix.
Incidental: fixed 3 pre-existing pyright errors in parser.py
(behavior-preserving) while getting to local-green.
Local: 61/61 pytest, ruff clean, pyright 0 errors. No cloud run, no
apify push — forbidden today by the usage cap. See notes.md for the
exact replay sequence for cap-lift day.
icon.svg deferred to the actor-icon-designer agent (not authored
here per hard rule).
src/models.py, src/client.py, src/parser.py, src/discovery.py,
src/main.py, src/scraper.py, src/recon_cloud.py are not yet
implemented — see docs/specs/quora-questions-scraper/tasks.md
T02-T09. python -m src will fail with ModuleNotFoundError until
T09 lands src/main.py; this is the intended task boundary, not a
scaffolding defect.
