Pricing

$2.00 / 1,000 mention-scrapeds

Try for free

Go to Apify Store

Reddit Research & Search

Try for free

Search Reddit for posts matching keywords, scoped to specific subreddits or sitewide, with comments — no API key, no login, no browser required. Reddit's per-IP reputation varies; an occasional run may return partial results, and your own proxy gives the most consistent results.

Pricing

$2.00 / 1,000 mention-scrapeds

Rating

0.0

(0)

Developer

Mikkel Bech-Hansen

Actor stats

Bookmarked

Total users

Monthly active users

9 days ago

Last modified

How it works — and the one thing you need to know

Reddit's modern search page (www.reddit.com/search/) responds to requests from well-reputed IPs with a lightweight, automatically-solvable JS challenge (HTTP 200) rather than a hard block. This actor solves that challenge itself, with plain HTTP requests — no headless browser, no manual intervention.

Reddit now has a second, harder gate as well, and it is not solvable. When an IP is rate-limited or has poor reputation, Reddit skips the solvable challenge and serves either a hard 403 or a "Prove your humanity" page wrapping a real Google reCAPTCHA — also with HTTP 200, so it looks like a normal response unless you check for it. Nothing in a plain-HTTP actor can pass that; it needs a browser and a human or a paid solver. The actor detects it, treats it as a block (retire session → back off → retry on a fresh IP), and if every attempt ends there it fails the run rather than delivering an empty dataset. An empty dataset would be indistinguishable from "this brand has no mentions", which is the worse outcome for a monitoring tool.

A 403 does not always mean the IP, though. Reddit also checks the request that redeems a solved challenge, and can refuse that one with the same block page — a well-reputed IP can be handed the challenge, solve it correctly, and still be turned away. The two cases are indistinguishable on the wire, so the run summary reports blockedAfterSolvingChallenge separately from blockedByReddit, and the error message names the hop it came from. Read that first when a run fails; see Known limitations.

The single most important thing to get right is the proxy. As of 2026-09-16, Reddit refuses Apify's shared platform egress IP outright — a 403 on first contact, before any challenge is issued — so a platform run with the proxy switched off fails every request. The default proxy setting works; leave it on. Full measurements under Known limitations.

Reddit rate-limits per IP in windows, and it rate-limits the challenge solve far harder than it rate-limits ordinary page fetches. A burst of requests from one IP will start drawing 403s (a compact <body class=theme-beta> block page, not the solvable challenge), and that IP then stays blocked for a while before recovering. Measured on a cool IP: the first solve succeeds and the next five in the same window all come back 403, while ordinary fetches made with an already-solved cookie sail through. So the single most important thing this actor does is solve once and reuse the cookie — the whole run is one solve plus one plain request per page. The actor also backs off exponentially on each block (2s → 4s → 8s → 16s → 20s) instead of burning its retries instantly, and keeps concurrency low. This is why a proxy pool matters even though any single IP works fine when it is fresh.

IP reputation is mixed, including within Apify's own proxy pools — session rotation matters a lot. Apify's RESIDENTIAL proxy group (and platform default IPs) are not uniformly blocked as originally assumed: individual IPs within the pool have different reputations with Reddit. Verified directly across several runs: the same apifyProxyGroups: ["RESIDENTIAL"] configuration got anywhere from 0% to 75% of requests through depending on which IPs the session pool happened to draw. To make the most of this, the actor retires a session (forcing a fresh IP on retry) the moment a request comes back blocked, instead of hammering the same bad IP across all retries — this alone took one test run from consistent total failure to ~75% success using nothing but Apify's own residential proxy.

For the most reliable results, supply your own proxy in the proxyConfiguration input field (a paid mobile or premium residential provider — not affiliated with Apify's shared pool — gives the most consistent odds). But Apify's own RESIDENTIAL group, left on with its default (unfiltered) settings, is a genuinely usable fallback now that session rotation is in place — narrowing it with apifyProxyCountry can hurt more than it helps if that country's slice of the pool happens to be having a rough patch (observed directly: a DK-restricted pool failed twice in a row while the unfiltered pool succeeded immediately after). Without any proxy at all, runs from Apify's platform IPs now fail outright, every request — this is no longer a matter of odds. Supply at least the default RESIDENTIAL group.

Input

Field	Type	Default	Notes
`queries`	array of strings	`["XPENG G6"]`	Search terms; each searched separately.
`subreddits`	array of strings	`[]` (sitewide)	Restrict to these subreddits (no `r/` prefix).
`postUrls`	array of strings	`[]`	Specific post URLs to fetch directly, bypassing search.
`sort`	enum	`"new"`	`relevance`, `new`, `top`, `hot`, `comments`.
`timeRange`	enum	`"year"`	`hour`, `day`, `week`, `month`, `year`, `all`.
`maxPosts`	integer	`50`	Cap on unique posts fetched across all queries/subreddits. Applied before relevance filtering (see below), so the final delivered count can be lower than `maxPosts` if some of what Reddit returned turns out irrelevant.
`includeComments`	boolean	`false`	Fetch each post's top comments too.
`maxCommentsPerPost`	integer	`20`	Only used when `includeComments` is on.
`proxyConfiguration`	object	Apify RESIDENTIAL	On by default and usable out of the box; your own non-Apify proxy gives the most consistent results — see above.

Output

One dataset record per post:

{
  "id": "1vm5al0",
  "type": "post",
  "subreddit": "TeslaModel3",
  "title": "Tesla Model 3 Bra, Black or Carbon Fiber Style",
  "author": "Solstice_25",
  "score": 1,
  "createdAt": "2026-08-12T05:37:34.199000+0000",
  "permalink": "https://www.reddit.com/r/TeslaModel3/comments/1vm5al0/.../",
  "url": "https://www.reddit.com/r/TeslaModel3/comments/1vm5al0/.../",
  "text": "Which would look better...",
  "numComments": 1,
  "comments": [],
  "matchedQueries": ["Tesla"],
  "source": "reddit_search",
  "nsfw": false
}

matchedQueries only lists queries actually verified to appear in the post's title or text (see relevance filtering below) — it's a reliable signal, not just an echo of which search produced the result.

Architecture

src/
  main.ts              orchestration: input → discovery → crawl → dedup → dataset
  crawler/
    challenge.ts        detect + solve Reddit's JS challenge
    redditClient.ts      fetch-with-challenge-solving, reusable across request types
  discovery/
    buildRequests.ts     turn input into initial search URLs
  parsers/
    searchResults.ts     parse search-results HTML → posts + pagination cursor
    postDetail.ts         parse a post page → full text/url + comments
  normalization/
    normalizePost.ts     raw parsed data → final output shape
    relevance.ts          filter out Reddit's unrelated "recommended" padding results
  store/
    dedupe.ts            ID-based dedup, matchedQueries merging across duplicate hits

Do not name a directory under src/ storage, dist, node_modules, apify_storage or crawlee_storage. apify push builds its upload list by re-feeding the output of git ls-files --others --ignored --directory back in as .gitignore patterns. That output is a path (storage/), but as a pattern storage/ is unanchored and matches a directory of that name at any depth — so a local storage/ directory (which any local test run creates) silently strips src/storage/ from the upload. That is not hypothetical: it is why this file used to live in src/storage/ and why build 1.0.29 failed with Cannot find module './storage/dedupe.js'.

The challenge mechanisms, in detail

Reddit's edge sometimes serves a tiny HTML page (~8KB) instead of the real content: an inline <script> computes a solution value and auto-submits a hidden form back to the same path with that solution plus a server-issued token attached. The interesting part: the "solution" is just the seed value self-concatenated (seed + seed) — no real cryptographic work, no proof-of-work — and the seed is embedded in plain text right in the script. crawler/challenge.ts extracts the seed with a regex, copies the form's hidden inputs verbatim, and rebuilds the solved URL; no JS execution or browser is involved at any point.

The hidden-input names are read from the form rather than hardcoded, and that detail is load-bearing. On 2026-09-03 Reddit renamed the token field from token to jsc_token (and widened its value from 32 to 64 hex characters), which broke every run. Submitting the stale name is worse than submitting nothing: Reddit answers a malformed challenge with a hard 403 block page, not a retryable error. Since the page's own handler is document.forms[0] + requestSubmit() — "send whatever the form declares" — mirroring that is both the faithful behaviour and the one that survives the next rename. Only the seed stays pinned to a regex, because it comes from the script rather than the form and has to be transformed rather than copied.

The submission goes over HTTP/1.1, not HTTP/2. main.ts sets http2: false on every Reddit request. Reddit hands the challenge page to essentially anyone but does inspect the hop that redeems the solution, and on 2026-09-09 that hop appeared to be rejected purely for the HTTP/2 fingerprint of Node's http2 stack (http2-wrapper), which is what got-scraping uses by default: one IP, hop 1 held at HTTP/1.1 and only hop 2's protocol varying, HTTP/2 gave 5/5 403s (189,906 B block page) and HTTP/1.1 gave 5/5 200s (~460 KB real page).

That result did not reproduce on 2026-09-16 — three interleaved HTTP/2 trials all redeemed the challenge normally (200, ~455–474 KB). So either Reddit relaxed the check, or the original trials caught a rate limit that happened to land on the HTTP/2 arm. The pin stays because it is free at maxConcurrency: 2 and a null result on one day is not proof the check is gone — but treat it as belt-and-braces, not as the thing holding this actor together. It was never the cause of either production failure; both were IP blocks on first contact (see Known limitations).

Session cookies from one solved challenge carry over to subsequent requests (different queries, post-detail pages, pagination) without needing to re-solve every time — verified directly. redditClient.ts still defensively checks every response and re-solves if a session ever gets re-challenged.

Getting that carry-over is the single biggest lever on whether a run survives, and two separate things have to be right for it to happen.

The cookie has to survive the request, which under Crawlee it does not by default. BasicCrawler hands the request handler a sendRequest helper wired to a session-backed cookie jar — but the client that actually performs the request discards it:

// @crawlee/core/http_clients/got-scraping-http-client.js
const gotResult = await gotScraping({ ...request, cookieJar: undefined, ... });

The comment there explains that HttpCrawler pre-reads cookies into request.gotOptions, which is true for HttpCrawler but not for BasicCrawler's sendRequest. The result, confirmed on the wire, is that no request ever carried a Cookie header and no Set-Cookie was ever stored — Reddit issued a fresh edgebucket on every single hop. So every request re-solved the challenge, and since Reddit rate-limits solves hard, the actor could not get past its own first page. main.ts therefore reads and writes session.cookieJar by hand around each sendRequest call.

The session that holds the cookie has to be handed back out. Crawlee's session pool defaults to 1000 sessions and returns a brand-new one until the pool fills, which would defeat the reuse even with cookies working. main.ts pins the pool small (one session with no proxy, where every session shares the same IP anyway; four with a proxy, where each session is a distinct IP worth rotating into).

Same crawl, measured end to end: before, every run failed on its first request with a hard 403 and delivered nothing; after, 28/28 requests succeed with no retries at all, 24 posts, about 9 seconds — repeated three times back to back, which is exactly the pattern that used to fail deterministically.

The second mechanism — the reCAPTCHA interstitial — shares none of that. It is a normal HTML page (HTTP 200, ~167 KB) containing a g-recaptcha widget bound to Reddit's site key, POSTing back to the same path with ?captcha=1. There is no seed and nothing to compute: passing it requires a genuine reCAPTCHA token. challenge.ts only detects it, keying on the reCAPTCHA widget rather than the English heading so a localised page is still caught. redditClient.ts checks for it on both hops — before and after solving the JS challenge — and raises RedditBlockedError, which is what drives session retirement and backoff.

Detection is the whole point here. The interstitial parses fine as HTML and simply contains no posts, so an actor that does not recognise it will count the fetch as a success, find zero results, and finish green with an empty dataset.

When a request comes back blocked (hard 403 or the reCAPTCHA interstitial, not the solvable challenge), main.ts retires the current session and waits — exponential backoff — before the automatic retry, so the retry gets a different underlying IP from the proxy pool and gives a rate-limited IP time to recover rather than repeating the same bad one immediately.

Relevance filtering

Reddit's search results mix in some unrelated "recommended"/related content alongside genuine query matches — verified directly (e.g. a completely unrelated monitor-review post returned for a "Tesla" search, with zero mention of the word anywhere in its title or body). normalization/relevance.ts checks, after a post's full text is fetched, whether any word from each matched query actually appears in the title or body; posts that fail this check for every query they were discovered under are dropped before being counted or delivered (logged in the run summary as filteredOutAsIrrelevant). The check is deliberately lenient (any query word, not the whole phrase) so it doesn't wrongly drop a genuine match that only restates part of a multi-word query — verified against real output where the match was legitimately body-only (e.g. "Tesla" appearing only in a stock portfolio listing, not the post title).

This filter only applies to posts discovered via search (source: "reddit_search"); posts supplied directly via postUrls (source: "direct_url") were explicitly requested and are never filtered.

Data delivery and charging

Actor.pushData() always runs for every relevant post found — it is never gated on Actor.charge() succeeding. This matters because Actor.charge() is a no-op (with a logged warning, not an error) on any actor version where pay-per-event monetization hasn't been configured in Apify Console yet; earlier code mistakenly treated a charge as a precondition for saving data, which meant every found post was silently discarded on an unmonetized actor. Charging is now best-effort and independent of delivery: it fires when monetization is configured and does nothing but log otherwise, either way every relevant post reaches the dataset.

Known limitations

IP reputation varies, including within Apify's own proxy pools. Session rotation and backoff on block substantially improve odds but don't guarantee every run succeeds — a non-Apify proxy (mobile or premium residential) still gives more consistent results. See "How it works" above.
Reddit's reCAPTCHA interstitial cannot be solved by this actor, by design. Once an IP is drawing it, that IP is unusable until it recovers; the only remedies are waiting or a larger/better proxy pool. A run where every request ends at the interstitial fails loudly — it does not return a partial or empty dataset dressed up as success.
On the Apify platform, a proxy is mandatory. Turning it off fails the run, every time. Reddit refuses Apify's shared platform egress IP outright: HTTP 403 on first contact, before any challenge is issued, answered in ~6–58 ms. That is an edge deny keyed on the IP — there is nothing to solve and nothing to wait out, so all six retries draw the identical block page and the run ends with zero posts. Measured 2026-09-16, same build, same input, runs started within one minute of each other, proxy the only variable:

proxy requests failed posts run
none (direct from the platform) 1 1 0 FAILED
Apify Proxy, automatic datacenter group 28 0 24 SUCCEEDED
Apify Proxy, RESIDENTIAL (the default) 28 0 24 SUCCEEDED

Both proxied runs returned a byte-identical result set. Nothing about the actor's parsing was involved; only the route out of the container. The run summary now logs proxy: on or proxy: none (direct) so this is visible at a glance, and an unproxied block says so in the failure message instead of recommending residential traffic that was never the issue.

Running unproxied still works from a clean consumer IP — a 28-request local run completes with zero retries, verified again on 2026-09-16 — so useApifyProxy: false remains a supported choice for local use, and the actor honours it with a warning rather than overriding you. It is just not a viable configuration on the platform.
The transport is part of the contract, not just the markup. On 2026-09-08 this actor was thought to be failing because Reddit rejected the HTTP client's HTTP/2 fingerprint on the challenge-redeem hop, and http2: false was pinned in response. That diagnosis was wrong about production: the failing run's 403 was on first contact, not post-solve — the same IP block described above, misread from a stack line number. Re-tested on 2026-09-16, HTTP/2 redeemed the challenge fine in 3/3 trials. The pin is kept because it costs nothing at maxConcurrency: 2, but it is not load-bearing. The general point stands: a break here will not necessarily be in the parser, so check the route and the transport before the selectors.
The cookie fix depends on a Crawlee internal staying as it is. main.ts carries cookies by hand specifically because GotScrapingHttpClient sets cookieJar: undefined. If a future Crawlee release fixes that, this code keeps working (the jar is the session's own, so the two agree rather than conflict) — but the comment explaining why the manual handling exists would become stale, so check it when upgrading Crawlee.
Pagination uses a cursor param scraped from a lazy-loaded partial in each page's HTML; undocumented and could change without notice — it already has once. The endpoint is /svc/shreddit/search/?...&cursor=... for a sitewide search but /svc/shreddit/r/<subreddit>/search/?...&cursor=... when the search is restricted to a subreddit, and the parser has to accept both.
Comment extraction depends on Reddit's <shreddit-comment> custom-element markup and attribute names, which are also undocumented and could change.
If Reddit changes either challenge mechanism (a harder/real proof-of-work, different seed encoding, a different interstitial), crawler/challenge.ts will need updating — it's isolated specifically so that's a small, contained change. Renaming or adding a hidden form field is already handled, since the fields are copied from the form rather than hardcoded; this happened on 2026-09-03 (token → jsc_token) and took the actor down. A change to the seed's encoding or to the seed + seed transform would still break it.
A rejected challenge submission is answered with a hard 403 block page that looks exactly like an IP-reputation block. This ambiguity has already misdiagnosed a failure once — the 2026-09-08 block was a first-contact IP block read as a post-solve one — so the run summary reports blockedAfterSolvingChallenge separately and the error names the hop. Read that first: a block on first contact is about the route out (proxy off, or a refused proxy IP); a block that lands after Reddit issued a challenge and the actor solved it is not an IP problem — Reddit only offers the challenge to IPs it is willing to serve — and means something about how the submission is made is being refused, which needs a code fix rather than proxy traffic. The failure messages give different advice for the two cases accordingly.
When a post's detail page is blocked but its search result was not, the post is still delivered using search-result data alone: text is empty and comments is []. The run summary reports this as the gap between uniquePostsFound and the number of detail fetches that succeeded. Note that an empty text is also perfectly normal for link posts, which genuinely have no body.
The relevance filter is a simple word-overlap heuristic, not true semantic relevance — it will still pass a post that happens to contain a common query word in an unrelated context (e.g. a generic word used as part of a multi-word brand query).

Legal

This Actor is an independent, third-party tool. It is not affiliated with, endorsed by, or sponsored by Reddit, Inc.

It retrieves data by automating Reddit's public search page rather than through Reddit's official Data API (see "How it works" above for why). Reddit's User Agreement restricts automated access to the site outside of that API, so using this Actor may not comply with Reddit's terms. You are responsible for reviewing Reddit's terms and applicable law before using this Actor, and for how you use the data it returns.

Output includes Reddit usernames attached to post/comment content, which can constitute personal data under GDPR and similar laws even though the accounts are pseudonymous. This Actor does not resolve usernames to real identities or collect contact details — but if your use case involves EU/UK individuals, treat the output as personal data for compliance purposes (lawful basis, retention limits, data subject rights) rather than as anonymous text.

Reddit API Scraper

rl1987/reddit-api-scraper

Scrape Reddit posts, comments, search results, subreddits, and user profiles via Reddit's OAuth data API. No login required.

R.L.

Reddit Scraper - Posts, Comments & Subreddits

admo-solutions/reddit-scraper

Scrape Reddit posts, comments, subreddits, and user profiles using Reddit's public JSON API. No API key required.

Admo Solutions

Reddit Scraper - Fast, Comments,Data,Subreddits

myagizm/reddit-scraper

Scrape public Reddit posts, comments, users, communities, and search results as structured JSON. No login, no API key.

MYM

Fast Reddit Scraper ($2/1K)

tom_mobi/fast-reddit-scraper

Scrape Reddit posts, comments, subreddits, search results, and user profiles — no API key, no login, no rate limits. From $2 per 1,000 results.

Tom Mobi

5.0

Reddit Scraper — Posts, Comments & Search (No Login)

pepeschuster/reddit-scraper

Scrape Reddit posts, comments, users and search results with zero setup — no login, no API key, no OAuth. A cheap, reliable replacement for Reddit's paid API.

Pepe Schuster

Reddit Scraper

prodiger/reddit-scraper

Extract posts, comments, user profiles, and search results from Reddit. Pure HTTP, no API key required.

Arnas

454

5.0

Reddit Search Scraper — Posts, Comments & Users

logiover/reddit-search-scraper

Scrape Reddit subreddit search with no API key or login. Export posts and comments to CSV/JSON — a Reddit API alternative for keyword monitoring.

Logiover

Reddit Scraper - Posts, Comments & Subreddits

knowten/reddit-scraper

Extract Reddit posts, nested comments, subreddits, search results, and user profiles with 180+ fields. No API key required.

Knowten

Reddit Scraper - Posts, Comments, Users & Search

benthepythondev/reddit-scraper

Scrape public Reddit posts, comments, user activity, subreddits, and search results through Reddit's app-only OAuth API. Export nested threads, scores, media, Markdown, and token counts to JSON, CSV, or Excel. No Reddit login or user-supplied API key required.

Ben

187

5.0

Reddit Scraper — Posts & Comments

signalengine/reddit-scraper

Scrape posts and comments from any subreddit — no Reddit API key, no login, no proxy. A fast, free Reddit API alternative for public data, exported to JSON, CSV or Excel.

James Taylor

proxy	requests	failed	posts	run
none (direct from the platform)	1	1	0	FAILED
Apify Proxy, automatic datacenter group	28	0	24	SUCCEEDED
Apify Proxy, `RESIDENTIAL` (the default)	28	0	24	SUCCEEDED

Reddit Research & Search

How it works — and the one thing you need to know

Input

Output

Architecture

The challenge mechanisms, in detail

Relevance filtering

Data delivery and charging

Known limitations

Legal

You might also like

Reddit API Scraper

Reddit Scraper - Posts, Comments & Subreddits

Reddit Scraper - Fast, Comments,Data,Subreddits

Fast Reddit Scraper ($2/1K)

Reddit Scraper — Posts, Comments & Search (No Login)

Reddit Scraper

Reddit Search Scraper — Posts, Comments & Users

Reddit Scraper - Posts, Comments & Subreddits

Reddit Scraper - Posts, Comments, Users & Search

Reddit Scraper — Posts & Comments

Related articles