Reddit Research & Search

Search Reddit for posts matching one or more keywords, scoped to specific subreddits or sitewide, with comments — no Reddit API key, no login, no browser required.

How it works — and the one thing you need to know

Reddit's modern search page (www.reddit.com/search/) responds to requests from well-reputed IPs with a lightweight, automatically-solvable JS challenge (HTTP 200) rather than a hard block. This actor solves that challenge itself, with plain HTTP requests — no headless browser, no manual intervention.

IP reputation is mixed, including within Apify's own proxy pools — session rotation matters a lot. Apify's RESIDENTIAL proxy group (and platform default IPs) are not uniformly blocked as originally assumed: individual IPs within the pool have different reputations with Reddit. Verified directly across several runs: the same apifyProxyGroups: ["RESIDENTIAL"] configuration got anywhere from 0% to 75% of requests through depending on which IPs the session pool happened to draw. To make the most of this, the actor retires a session (forcing a fresh IP on retry) the moment a request comes back blocked, instead of hammering the same bad IP across all retries — this alone took one test run from consistent total failure to ~75% success using nothing but Apify's own residential proxy.

For the most reliable results, supply your own proxy in the proxyConfiguration input field (a paid mobile or premium residential provider — not affiliated with Apify's shared pool — gives the most consistent odds). But Apify's own RESIDENTIAL group, left on with its default (unfiltered) settings, is a genuinely usable fallback now that session rotation is in place — narrowing it with apifyProxyCountry can hurt more than it helps if that country's slice of the pool happens to be having a rough patch (observed directly: a DK-restricted pool failed twice in a row while the unfiltered pool succeeded immediately after). Without any proxy at all, runs from Apify's platform IPs will still usually fail — supply at least the default RESIDENTIAL group.

Input

Field	Type	Default	Notes
queries	array of strings	["XPENG G6"]	Search terms; each searched separately.
subreddits	array of strings	[] (sitewide)	Restrict to these subreddits (no r/ prefix).
postUrls	array of strings	[]	Specific post URLs to fetch directly, bypassing search.
sort	enum	"new"	relevance, new, top, hot, comments.
timeRange	enum	"year"	hour, day, week, month, year, all.
maxPosts	integer	50	Cap on unique posts fetched across all queries/subreddits. Applied before relevance filtering (see below), so the final delivered count can be lower than maxPosts if some of what Reddit returned turns out irrelevant.
includeComments	boolean	false	Fetch each post's top comments too.
maxCommentsPerPost	integer	20	Only used when includeComments is on.
proxyConfiguration	object	Apify RESIDENTIAL	On by default and usable out of the box; your own non-Apify proxy gives the most consistent results — see above.

Output

One dataset record per post:

{
  "id": "1vm5al0",
  "type": "post",
  "subreddit": "TeslaModel3",
  "title": "Tesla Model 3 Bra, Black or Carbon Fiber Style",
  "author": "Solstice_25",
  "score": 1,
  "createdAt": "2026-08-12T05:37:34.199000+0000",
  "permalink": "https://www.reddit.com/r/TeslaModel3/comments/1vm5al0/.../",
  "url": "https://www.reddit.com/r/TeslaModel3/comments/1vm5al0/.../",
  "text": "Which would look better...",
  "numComments": 1,
  "comments": [],
  "matchedQueries": ["Tesla"],
  "source": "reddit_search",
  "nsfw": false
}

matchedQueries only lists queries actually verified to appear in the post's title or text (see relevance filtering below) — it's a reliable signal, not just an echo of which search produced the result.

Architecture

src/
  main.ts              orchestration: input → discovery → crawl → dedup → dataset
  crawler/
    challenge.ts        detect + solve Reddit's JS challenge
    redditClient.ts      fetch-with-challenge-solving, reusable across request types
  discovery/
    buildRequests.ts     turn input into initial search URLs
  parsers/
    searchResults.ts     parse search-results HTML → posts + pagination cursor
    postDetail.ts         parse a post page → full text/url + comments
  normalization/
    normalizePost.ts     raw parsed data → final output shape
    relevance.ts          filter out Reddit's unrelated "recommended" padding results
  storage/
    dedupe.ts            ID-based dedup, matchedQueries merging across duplicate hits

The challenge mechanism, in detail

Reddit's edge sometimes serves a tiny HTML page (~8KB) instead of the real content: an inline <script> computes a solution value and auto-submits a hidden form back to the same path with that solution plus a server-issued token attached. The interesting part: the "solution" is just the seed value self-concatenated (seed + seed) — no real cryptographic work, no proof-of-work — and the seed is embedded in plain text right in the script. crawler/challenge.ts extracts the seed and token with a regex and rebuilds the solved URL; no JS execution or browser is involved at any point.

Session cookies from one solved challenge carry over to subsequent requests (different queries, post-detail pages, pagination) without needing to re-solve every time — verified directly. redditClient.ts still defensively checks every response and re-solves if a session ever gets re-challenged.

When a request comes back blocked (hard 403, not the solvable challenge), main.ts retires the current session before the automatic retry, so the retry gets a different underlying IP from the proxy pool rather than repeating the same bad one. This is the single highest-leverage reliability fix found so far — see "How it works" above.

Relevance filtering

Reddit's search results mix in some unrelated "recommended"/related content alongside genuine query matches — verified directly (e.g. a completely unrelated monitor-review post returned for a "Tesla" search, with zero mention of the word anywhere in its title or body). normalization/relevance.ts checks, after a post's full text is fetched, whether any word from each matched query actually appears in the title or body; posts that fail this check for every query they were discovered under are dropped before being counted or delivered (logged in the run summary as filteredOutAsIrrelevant). The check is deliberately lenient (any query word, not the whole phrase) so it doesn't wrongly drop a genuine match that only restates part of a multi-word query — verified against real output where the match was legitimately body-only (e.g. "Tesla" appearing only in a stock portfolio listing, not the post title).

This filter only applies to posts discovered via search (source: "reddit_search"); posts supplied directly via postUrls (source: "direct_url") were explicitly requested and are never filtered.

Data delivery and charging

Actor.pushData() always runs for every relevant post found — it is never gated on Actor.charge() succeeding. This matters because Actor.charge() is a no-op (with a logged warning, not an error) on any actor version where pay-per-event monetization hasn't been configured in Apify Console yet; earlier code mistakenly treated a charge as a precondition for saving data, which meant every found post was silently discarded on an unmonetized actor. Charging is now best-effort and independent of delivery: it fires when monetization is configured and does nothing but log otherwise, either way every relevant post reaches the dataset.

Known limitations

• IP reputation varies, including within Apify's own proxy pools. Session rotation on block substantially improves odds but doesn't guarantee every run succeeds — a non-Apify proxy (mobile or premium residential) still gives more consistent results. See "How it works" above.
• Pagination uses a cursor param scraped from a lazy-loaded partial in each page's HTML (/svc/shreddit/search/?...&cursor=...); undocumented and could change without notice.
• Comment extraction depends on Reddit's <shreddit-comment> custom-element markup and attribute names, which are also undocumented and could change.
• If Reddit changes the challenge mechanism (a harder/real proof-of-work, different seed encoding, etc.), crawler/challenge.ts will need updating — it's isolated specifically so that's a small, contained change.
• The relevance filter is a simple word-overlap heuristic, not true semantic relevance — it will still pass a post that happens to contain a common query word in an unrelated context (e.g. a generic word used as part of a multi-word brand query).

Legal

This Actor is an independent, third-party tool. It is not affiliated with, endorsed by, or sponsored by Reddit, Inc.

It retrieves data by automating Reddit's public search page rather than through Reddit's official Data API (see "How it works" above for why). Reddit's User Agreement restricts automated access to the site outside of that API, so using this Actor may not comply with Reddit's terms. You are responsible for reviewing Reddit's terms and applicable law before using this Actor, and for how you use the data it returns.

Output includes Reddit usernames attached to post/comment content, which can constitute personal data under GDPR and similar laws even though the accounts are pseudonymous. This Actor does not resolve usernames to real identities or collect contact details — but if your use case involves EU/UK individuals, treat the output as personal data for compliance purposes (lawful basis, retention limits, data subject rights) rather than as anonymous text.