Scrape Reddit posts from any subreddit with ease. This actor collects titles, upvotes, timestamps, authors, comment counts, and post URLs. Perfect for research, trend tracking, content analysis, and automations that rely on fresh Reddit data.
All notable changes to the Reddit Posts Search Scraper actor are documented here.
[0.4] - 2026-08-29
Fixed - the actual fix: anonymous session warm-up, not a domain swap
User correctly pushed back on the 0.3 diagnosis ("Reddit's WAF blocks every
proxy tier including residential") and pointed at the real mechanism: real
scrapers get past this by collecting a real anonymous session first, then
hitting www.reddit.com (not old.reddit.com) with those cookies.
Researched (Reddit deprecated cookie-less .json access in late May 2026
specifically to stop unaccountable scraping) and verified live end to end:
A bare GET to https://www.reddit.com/ sometimes serves a lightweight
JS-challenge interstitial instead of the real homepage. It is not
obfuscated — its own inline script is plaintext: it doubles a seed
string ((async e=>e+e)("<seed>")) and resubmits that as solution
alongside the page's own token. Solved this without a JS runtime by
extracting the seed/token via regex and resubmitting the same GET.
That resubmission returns real Set-Cookie headers for a normal
anonymous session (loid, session_tracker, csrf_token, token_v2,
csv, edgebucket) — the same ones a real logged-out browser tab gets.
No login/account needed.
With those cookies in the session's cookie jar,
www.reddit.com/r/ technology/top.json
returned HTTP 200 with real post data —
confirmed directly with curl before touching the actor's code.
Fix (src/scraper.py): added _warm_up_reddit_session(), called once
per aiohttp.ClientSession before ever requesting a .json URL — visits
the homepage, solves the challenge if shown, and lets the session's own
cookie jar carry the resulting cookies into the next request on that same
session. Removed the old.reddit.com domain-swap fallback entirely (the
real fix works on www.reddit.com directly, so swapping hosts was solving
the wrong problem) — every attempt now uses www.reddit.com.
src/parallel_processor.py's comment-fetch path gets the identical fix
(imports and calls the same _warm_up_reddit_session, also dropping its
own old.reddit.com fallback).
[0.3] - 2026-08-29
Fixed - broken /r/r/<subreddit>/ fallback URL
Live Apify verification of 0.2 with a residential proxy: www.reddit.com
still 403'd on every tier as expected, and the new old.reddit.com
fallback correctly triggered — but built
https://old.reddit.com/r/r/technology/top.json (double r/) for the
input "r/technology" (one of the actor's own documented accepted input
formats), so it 404/redirected to the login-wall regardless of proxy tier.
Root cause (src/scraper.py, fetch_reddit_posts_async): the subreddit
path was always built as f"/r/{subreddit_or_keyword}/..." without first
checking whether the caller had already passed the "r/"-prefixed form —
even though the very next line already checks
subreddit_or_keyword.startswith("r/") for a different purpose. Fixed to
strip a leading r/ before building the path.
[0.2] - 2026-08-29
Fixed — actor was returning 0 results on every run
Root causes identified by live testing against real Reddit endpoints:
requirements.txt listed asyncio>=3.4.3 (and typing>=3.7.4) as pip
dependencies. Both asyncio and typing have been part of the Python
standard library since 3.4/3.5 respectively; the PyPI packages of the same
name are unmaintained backports for pre-3.4/3.5 Python (PyPI's own metadata
for the current asyncio release literally says "Deprecated backport of
asyncio; use the stdlib package instead"). Testing showed today's pip/
setuptools resolve this without a hard install failure on this base image,
but it remains a documented anti-pattern that has broken builds on other
pip/setuptools combinations (the old asyncio==3.4.3 sdist depends on the
long-removed use_2to3 setuptools flag) and adds an unnecessary,
unmaintained package that shadows a stdlib module name for zero benefit.
Removed both lines.
Reddit now blocks unauthenticated .json requests from non-residential
(datacenter/hosting) IPs — the actual cause of the 0-result runs. Live
probing (2026-08-29, aiohttp and curl_cffi with real Chrome TLS
impersonation, from a hosting-provider IP) showed:
https://www.reddit.com/r/<sub>/<sort>.json, https://www.reddit.com/search.json,
https://api.reddit.com/..., and https://www.reddit.com/....rss all return a
clean HTTP 403 "Blocked" response directly from Reddit's own edge
(Server: snooserv), regardless of User-Agent (a generic Chrome UA and a
proper descriptive custom UA were both blocked identically) and
regardless of TLS fingerprint.
https://old.reddit.com/...json instead returns HTTP 200 but silently
redirects to an HTML login-wall page (/login/?reason=lor2&dest=...)
instead of JSON — a "soft block" that the previous code had no way to
detect, since it only checked for HTTP 403/429 before parsing the body
as JSON.
Both behaviors were confirmed to originate from Reddit itself (response
headers/cookies are scoped to reddit.com, not a network-level block).
The block/soft-block detection gap meant the actor's existing
no-proxy → datacenter → residential proxy fallback ladder never engaged
for the failure mode Reddit actually uses today (a 200 + HTML page), and
the comments endpoint (fetch_comments_with_retry_async) never triggered
proxy escalation at all — a 403 there wasn't even in the retry
classification list (only "429"/"Too Many Requests" was), so a blocked
comments fetch failed the post permanently on the very first attempt with
no retry.
Changed
requirements.txt: removed asyncio>=3.4.3 and typing>=3.7.4.
src/scraper.py (fetch_reddit_posts_async):
Added Content-Type based soft-block detection: a 200 response that is not
application/json (Reddit's login-wall page) is now treated the same as
a 403/429 and drives the same proxy-fallback retry path, instead of
falling through to an unhandled/uncategorized JSON-decode error.
Added FALLBACK_BASE_URL = "https://old.reddit.com": after the proxy
ladder is exhausted against the primary www.reddit.com host, one final
attempt is made against the old.reddit.com mirror before giving up,
since live testing showed the two hosts currently exhibit different
blocking behavior.
Modernized the request headers: bumped the spoofed Chrome version and
added Accept / Accept-Language (defense-in-depth; testing showed
headers alone do not bypass Reddit's IP-reputation based blocking, but
there is no reason not to send a fuller, more realistic header set).
The comments fetch now explicitly detects 403/429 and the same
Content-Type soft-block signature, and calls the proxy manager's
handle_blocked_request() to escalate the proxy tier — previously this
function never escalated the proxy or specially handled 403 at all.
Added a 403/BLOCKED case to the retry classification in
process_reddit_post so a blocked comments fetch is now retried (with
the same backoff used for rate limits) instead of failing the post
permanently after one attempt.
The last comments-fetch attempt also fails over to old.reddit.com,
matching the posts-listing fetch.
Synced the comments-fetch header set with the posts-listing fetch
(Chrome version, Accept, Accept-Language).
Verified
All edited files pass python -m py_compile.
The updated fetch_reddit_posts_async and fetch_comments_with_retry_async
were run directly (not mocked) against live www.reddit.com /
old.reddit.com: the 403 detection, the new soft-block detection, the
proxy-escalation call sequence, and the final old.reddit.com fallback
attempt were all confirmed to fire in the correct order with clear log
output.
Known limitation (please read)
From every network tested during this fix (a hosting-provider IP, not a
residential one), all unauthenticated Reddit read endpoints — new-Reddit
JSON, old-Reddit JSON, the RSS feed, and the internal mobile/desktop API —
are blocked or soft-blocked, regardless of headers or TLS fingerprint. This
is consistent with Reddit's current (2026) anti-scraping posture: it is an
IP-reputation gate, not a hard OAuth requirement, and it is not something
any request header can work around. The actor's existing residential-proxy
fallback (proxyConfiguration input, already implemented in
proxy_manager.py) is the intended way through it, and the fixes above make
sure that fallback ladder actually gets triggered by the block patterns
Reddit uses today. This was not re-verified end-to-end against a live
residential proxy (no proxy credentials were available while testing) — if
runs still return 0 results with proxyConfiguration.useApifyProxy enabled
and a residential group selected, that would point to Apify's residential
pool itself being rate-limited/blocked by Reddit, which is outside this
actor's code.
