Threads Reply Scraper — Conversation Graph

Export the full reply tree of any public Threads post — no Meta login — as a conversation graph plus engagement counts, to JSON or CSV. A Threads post scraper built on the threads.net SSR HTML payload. We retry and rotate so the thread lands.

Pricing

Pay per event

Rating

0.0

(0)

Developer

DevilScrapes

Actor stats

Bookmarked

Total users

Monthly active users

2 days ago

Last modified

[0.5] — 2026-08-21 (fix: proxy group was a small datacenter-ish USA pack, not residential — Meta was blocking the pool itself, not failing to rotate)

Fixed

Root cause (CEO report 2026-08-21: "30-day success rate ~72% — 18 of 66 runs failed"; unchanged from before the [0.4] 403-rotation fix shipped 2026-08-15, which fixed a real bug but did not move the success rate). GET /v2/acts/{id}/runs only returns runs started under our own token (3 total, all our own QA, all SUCCEEDED) — the 18 failures are invisible customer runs, so the bug had to be reproduced and the infra verified directly:
- A live cloud run against the exact Store-prefill input (build 0.4.1, useProxy: true, default depth/cap) — run Vg7vXsyrLiCcL27Ua — SUCCEEDED, but scripts/os/run_usage.py showed zero PROXY_RESIDENTIAL_TRANSFER_GBYTES billed against it, despite the code requesting a proxy on every run.
- GET /v2/users/me shows the Console labels BUYPROXIES94952 as "Proxies from USA 2" (27 IPs) — a small static/datacenter pack, not residential. RESIDENTIAL shows availableCount: 0 in the same response, which is the known cardinality display artifact (see memory reference-proxy-availablecount-trap / reference-residential-is-entitled), not a real entitlement gap — a local probe authenticated against both groups successfully.
- src/main.py::PROXY_GROUP was set to BUYPROXIES94952 since this Actor's original 2026-05-16 scaffold, when it was the only proxy group provisioned on the account (FREE tier). The account moved to STARTER on 2026-08-20, which provisions RESIDENTIAL, but nothing here was ever updated to use it — every "rotate on block" fix ([0.3], [0.4]) rotated within the same small non-residential pool. Meta blocks datacenter IPs aggressively (this Actor's own useProxy schema copy already said so) — rotating through 5-27 exits that are all in the same flagged range never produces a genuinely fresh identity. amazon-reviews-scraper — the sibling client.py explicitly says its retry/rotation logic mirrors — defaults PROXY_GROUP to RESIDENTIAL; this half of the pattern was never matched.
Fix: PROXY_GROUP = "RESIDENTIAL" in src/main.py. Updated the useProxy copy in .actor/input_schema.json, src/models.py, scaffold.json, and README.md — all previously advertised the BUYPROXIES94952 pool as "residential" by name, which was never true.
New regression test test_create_proxy_configuration_uses_residential_group — asserts _create_proxy_configuration requests groups=["RESIDENTIAL"]; fails on pre-fix code (groups=["BUYPROXIES94952"]), passes post-fix. 48/48 hermetic + 1 live -m smoke green.
No behavior change to the retry/backoff/rotation state machine in client.py — that logic (403+429 rotate, 408/429/5xx retry, per-URL fault isolation) was already correct; it was rotating within the wrong pool.

[0.4] — 2026-08-15 (fix: HTTP 403 never actually rotated — the 0.3 fix's core claim was unreachable code)

Fixed

Root cause (CEO report 2026-08-15: "30-day success rate 68% — 21 of 66 runs failed"; success rate did NOT recover after the 2026-08-10 [0.3] proxy-rotation fix — ops/reports/data/*.json shows success_rate_30d actually dropping from 0.734 (08-09, pre-fix) to 0.687/0.682/0.672 across the days immediately after 0.3 shipped, before a marginal recovery to 0.682 today). recent_failures on our own runs was empty of new signal (3 FAILED runs total, all either the pre-fix bug already documented in [0.3] or a deliberate own negative-input test), so the regression had to be found by code inspection plus a live cloud reproduction, not guessed.
src/client.py::ThreadsSession._attempt gated retries with if status not in RETRY_STATUSES: raise RuntimeError(...) before ever consulting ROTATE_STATUSES. RETRY_STATUSES is {408, 429, 500, 502, 503, 504} — it does not include 403. Since ROTATE_STATUSES = {403, 429} is only checked inside _handle_retriable_status, which is itself only reached after the RETRY_STATUSES gate passes, every HTTP 403 — Meta's most common anti-bot block signal, arguably more common than 429 for an already-flagged residential exit — raised immediately on attempt 1 with zero retry and zero rotation. The ThreadsSession rotation logic the [0.3] CHANGELOG entry credited with fixing the customer-failure spike was therefore unreachable for 403s the entire time; [0.3]'s own reproduction and cloud QA only exercised 429 and connection-timeout cases, never 403, so the gap shipped untested. Confirmed against actors/amazon-reviews-scraper/src/client.py — the sibling module this code claims to "structurally mirror" — whose equivalent gate correctly reads
```
if status not in RETRY_STATUSES and status not in ROTATE_STATUSES: raise
```
(i.e. a status in ROTATE_STATUSES can never be treated as non-retriable). threads-reply-tree had dropped the ROTATE_STATUSES half of that condition.
Fix: add and status not in ROTATE_STATUSES to the same gate, matching amazon-reviews-scraper. A 403 now reaches _handle_retriable_status, which rotates the browser-impersonation profile and mints a fresh Apify Proxy session_id before backing off and retrying — exactly the behavior the [0.3] docstrings already claimed.
New regression test test_retry_on_403_rotates_session (mirrors the existing test_retry_on_429_rotates_session) — fails on pre-fix code with RuntimeError: HTTP 403 ... (non-retriable), passes post-fix. 48/48 tests green (47 hermetic + 1 live -m smoke).
No behavior change for 404/400/401/410 (still immediately non-retriable, per test_no_retry_on_404) or for 429/5xx (unchanged).

[0.3] — 2026-08-10 (fix: fixed proxy session reused across every retry — sustained blocks failed the whole run)

Fixed

Root cause (CEO report 2026-08-10: "30-day success rate 69% — 20 FAILED of 68 customer runs"; 5 new failures landed since 08-06 on build 0.2.1). Our own runs mostly succeed (GET /v2/acts/{id}/runs only returns runs started under our token), so this had to be reproduced directly against the live Store build. Two throwaway cloud runs against the exact same real, currently-live post URL minutes apart: bmn46iJqvQeE47f68 SUCCEEDED in ~5s; NaX3kHbV8fQQNymwO hit five consecutive HTTP 429s from Threads and FAILED with exit 91 ("No rows emitted..."); a third run (BbQqCwfKct710S0ab) hit sustained curl: (28) Connection timed out on every attempt instead. In every case the log shows the identical failure repeating attempt after attempt with no change in outcome — because main.py opened one curl-cffi session (and therefore one fixed residential proxy exit IP) before the retry loop and reused it for every attempt and every URL in the batch. A blocked/rate-limited exit was retried against itself five times, in direct violation of the house anti-blocking rule ("rotate the session_id on every block"). Real content parsing was confirmed unaffected — six freshly-fetched real posts all parsed correctly during triage; this was purely a transport-identity bug.
src/client.py replaced the stateless fetch_post_html(session, url) helper with a stateful ThreadsSession that rotates the browser impersonation profile and mints a fresh Apify Proxy session_id on every 403/429/network-level error, before the next retry attempt. Structurally mirrors actors/amazon-reviews-scraper/src/client.py's AmazonReviewSession, an already-shipped instance of this exact pattern.
src/main.py now pins country_code="US" on the Apify Proxy configuration (feedback-pin-proxy-country) — defensive, since a geo-random exit can silently change what a target serves even without an outright error.
Added _guard_against_missing_replies: if a post's own direct_reply_count says replies exist but the walk captured none, the Actor now logs ERROR and dumps the raw conversation payload to the KVS (missing-replies-<id> key) instead of letting that look like a clean, complete scrape. Posts that genuinely have zero replies are unaffected.

[0.2] — 2026-08-05 (fix: threads.com URLs hard-rejected the whole batch)

Fixed

Root cause (CEO report 2026-08-05: "30-day success rate 76% — 13 FAILED of 56 runs"): recent_failures on our own runs was empty, so this had to be reproduced directly. Live check confirmed https://www.threads.net/... now 301-redirects to https://www.threads.com/... — Meta migrated the canonical share-link domain in 2024, and every "Copy Link" / address-bar URL a customer pastes today is threads.com. ActorInput.post_urls's regex only accepted threads.net, and Pydantic validates the whole list eagerly — a single threads.com URL anywhere in postUrls raised ValidationError and killed the entire run via SystemExit(1) before a single HTTP request was made. Reproduced locally with
```
apify run
```
: pre-fix, a threads.com-only input exits 91 (FAILED) inside ActorInput.model_validate; post-fix, the same input exits 0 with 7 rows scraped.
POST_URL_PATTERN now accepts both threads.net and threads.com.
ActorInput._normalize_urls no longer fails the whole postUrls batch on one malformed item (wrong domain, wrong shape, non-string). It skips the bad item with a logged warning and keeps every valid URL — matching the REQ-7 "fail loud only when nothing is left" pattern already used elsewhere in this Actor. Still raises when zero items in the batch are usable.
main._scrape_one_url now also catches unexpected parser exceptions (ValueError/TypeError/KeyError/AttributeError/IndexError) per-URL and skips that URL instead of letting a single malformed post object crash _drain_inputs for every other URL in the batch.
Updated input_schema.json description, the postUrls field description, and the README "Finding a Threads post URL" section to say both domains work.
No changes to the happy-path fetch/parse logic — threads.net still works unchanged (verified redirect + SSR payload extraction both succeed), and the QA fixture (threads.net) still scrapes 7 rows.

[0.1.0] — 2026-05-16

Added

Initial release.
HTML SSR extraction of Threads conversation tree via curl-cffi chrome131 impersonation.
Per-post reply tree: root + every visible reply chain, with depth
- parent_reply_id linkage for graph reconstruction.
max_depth and max_replies_per_node caps for cost control.
PPE pricing: $0.05 / start + $0.005 / row.
Residential proxy (BUYPROXIES94952) recommended ON by default.

threads search post scraper

scrapeengine/threads-search-post-scraper

ScrapeEngine

🧵 Threads Search Post Scraper

scraperforge/threads-search-post-scraper

ScraperForge

🧵 Threads Search Post Scraper

scrapemesh/threads-search-post-scraper

ScrapeMesh

Threads Scraper - Scrape threads.net Posts, No Login

logiover/threads-scraper

No-login Threads scraper & unofficial API alternative. Export public Threads posts, replies, profiles & engagement to CSV, JSON or Excel.

Logiover

Threads Scraper - Posts, Likes & Replies

thirdwatch/threads-scraper

Scrape public posts from any Meta Threads (threads.net / threads.com) profile. Get post text, like counts, timestamps, author details, media URLs, and thread links via the public guest GraphQL API. No login, API key, or proxy needed.