Pricing

$2.00 / 1,000 mention-scrapeds

Try for free

Go to Apify Store

Reddit Research & Search

Try for free

Search Reddit for posts matching keywords, scoped to specific subreddits or sitewide, with comments — no API key, no login, no browser required. Reddit's per-IP reputation varies; an occasional run may return partial results, and your own proxy gives the most consistent results.

Pricing

$2.00 / 1,000 mention-scrapeds

Rating

0.0

(0)

Developer

Mikkel Bech-Hansen

Actor stats

Bookmarked

Total users

Monthly active users

a day ago

Last modified

How it works — and the one thing you need to know

Reddit's modern search page (www.reddit.com/search/) responds to requests from well-reputed IPs with a lightweight, automatically-solvable JS challenge (HTTP 200) rather than a hard block. This actor solves that challenge itself, with plain HTTP requests — no headless browser, no manual intervention.

Reddit now has a second, harder gate as well, and it is not solvable. When an IP is rate-limited or has poor reputation, Reddit skips the solvable challenge and serves either a hard 403 or a "Prove your humanity" page wrapping a real Google reCAPTCHA — also with HTTP 200, so it looks like a normal response unless you check for it. Nothing in a plain-HTTP actor can pass that; it needs a browser and a human or a paid solver. The actor detects it, treats it as a block (retire session → back off → retry on a fresh IP), and if every attempt ends there it fails the run rather than delivering an empty dataset. An empty dataset would be indistinguishable from "this brand has no mentions", which is the worse outcome for a monitoring tool.

Reddit rate-limits per IP in windows, and it rate-limits the challenge solve far harder than it rate-limits ordinary page fetches. A burst of requests from one IP will start drawing 403s (a compact <body class=theme-beta> block page, not the solvable challenge), and that IP then stays blocked for a while before recovering. Measured on a cool IP: the first solve succeeds and the next five in the same window all come back 403, while ordinary fetches made with an already-solved cookie sail through. So the single most important thing this actor does is solve once and reuse the cookie — the whole run is one solve plus one plain request per page. The actor also backs off exponentially on each block (2s → 4s → 8s → 16s → 20s) instead of burning its retries instantly, and keeps concurrency low. This is why a proxy pool matters even though any single IP works fine when it is fresh.

IP reputation is mixed, including within Apify's own proxy pools — session rotation matters a lot. Apify's RESIDENTIAL proxy group (and platform default IPs) are not uniformly blocked as originally assumed: individual IPs within the pool have different reputations with Reddit. Verified directly across several runs: the same apifyProxyGroups: ["RESIDENTIAL"] configuration got anywhere from 0% to 75% of requests through depending on which IPs the session pool happened to draw. To make the most of this, the actor retires a session (forcing a fresh IP on retry) the moment a request comes back blocked, instead of hammering the same bad IP across all retries — this alone took one test run from consistent total failure to ~75% success using nothing but Apify's own residential proxy.

For the most reliable results, supply your own proxy in the proxyConfiguration input field (a paid mobile or premium residential provider — not affiliated with Apify's shared pool — gives the most consistent odds). But Apify's own RESIDENTIAL group, left on with its default (unfiltered) settings, is a genuinely usable fallback now that session rotation is in place — narrowing it with apifyProxyCountry can hurt more than it helps if that country's slice of the pool happens to be having a rough patch (observed directly: a DK-restricted pool failed twice in a row while the unfiltered pool succeeded immediately after). Without any proxy at all, runs from Apify's platform IPs will still usually fail — supply at least the default RESIDENTIAL group.

Input

Field	Type	Default	Notes
`queries`	array of strings	`["XPENG G6"]`	Search terms; each searched separately.
`subreddits`	array of strings	`[]` (sitewide)	Restrict to these subreddits (no `r/` prefix).
`postUrls`	array of strings	`[]`	Specific post URLs to fetch directly, bypassing search.
`sort`	enum	`"new"`	`relevance`, `new`, `top`, `hot`, `comments`.
`timeRange`	enum	`"year"`	`hour`, `day`, `week`, `month`, `year`, `all`.
`maxPosts`	integer	`50`	Cap on unique posts fetched across all queries/subreddits. Applied before relevance filtering (see below), so the final delivered count can be lower than `maxPosts` if some of what Reddit returned turns out irrelevant.
`includeComments`	boolean	`false`	Fetch each post's top comments too.
`maxCommentsPerPost`	integer	`20`	Only used when `includeComments` is on.
`proxyConfiguration`	object	Apify RESIDENTIAL	On by default and usable out of the box; your own non-Apify proxy gives the most consistent results — see above.

Output

One dataset record per post:

{
  "id": "1vm5al0",
  "type": "post",
  "subreddit": "TeslaModel3",
  "title": "Tesla Model 3 Bra, Black or Carbon Fiber Style",
  "author": "Solstice_25",
  "score": 1,
  "createdAt": "2026-08-12T05:37:34.199000+0000",
  "permalink": "https://www.reddit.com/r/TeslaModel3/comments/1vm5al0/.../",
  "url": "https://www.reddit.com/r/TeslaModel3/comments/1vm5al0/.../",
  "text": "Which would look better...",
  "numComments": 1,
  "comments": [],
  "matchedQueries": ["Tesla"],
  "source": "reddit_search",
  "nsfw": false
}

matchedQueries only lists queries actually verified to appear in the post's title or text (see relevance filtering below) — it's a reliable signal, not just an echo of which search produced the result.

Architecture

src/
  main.ts              orchestration: input → discovery → crawl → dedup → dataset
  crawler/
    challenge.ts        detect + solve Reddit's JS challenge
    redditClient.ts      fetch-with-challenge-solving, reusable across request types
  discovery/
    buildRequests.ts     turn input into initial search URLs
  parsers/
    searchResults.ts     parse search-results HTML → posts + pagination cursor
    postDetail.ts         parse a post page → full text/url + comments
  normalization/
    normalizePost.ts     raw parsed data → final output shape
    relevance.ts          filter out Reddit's unrelated "recommended" padding results
  store/
    dedupe.ts            ID-based dedup, matchedQueries merging across duplicate hits

Do not name a directory under src/ storage, dist, node_modules, apify_storage or crawlee_storage. apify push builds its upload list by re-feeding the output of git ls-files --others --ignored --directory back in as .gitignore patterns. That output is a path (storage/), but as a pattern storage/ is unanchored and matches a directory of that name at any depth — so a local storage/ directory (which any local test run creates) silently strips src/storage/ from the upload. That is not hypothetical: it is why this file used to live in src/storage/ and why build 1.0.29 failed with Cannot find module './storage/dedupe.js'.

The challenge mechanisms, in detail

Reddit's edge sometimes serves a tiny HTML page (~8KB) instead of the real content: an inline <script> computes a solution value and auto-submits a hidden form back to the same path with that solution plus a server-issued token attached. The interesting part: the "solution" is just the seed value self-concatenated (seed + seed) — no real cryptographic work, no proof-of-work — and the seed is embedded in plain text right in the script. crawler/challenge.ts extracts the seed with a regex, copies the form's hidden inputs verbatim, and rebuilds the solved URL; no JS execution or browser is involved at any point.

The hidden-input names are read from the form rather than hardcoded, and that detail is load-bearing. On 2026-09-03 Reddit renamed the token field from token to jsc_token (and widened its value from 32 to 64 hex characters), which broke every run. Submitting the stale name is worse than submitting nothing: Reddit answers a malformed challenge with a hard 403 block page, not a retryable error. Since the page's own handler is document.forms[0] + requestSubmit() — "send whatever the form declares" — mirroring that is both the faithful behaviour and the one that survives the next rename. Only the seed stays pinned to a regex, because it comes from the script rather than the form and has to be transformed rather than copied.

Session cookies from one solved challenge carry over to subsequent requests (different queries, post-detail pages, pagination) without needing to re-solve every time — verified directly. redditClient.ts still defensively checks every response and re-solves if a session ever gets re-challenged.

Getting that carry-over is the single biggest lever on whether a run survives, and two separate things have to be right for it to happen.

The cookie has to survive the request, which under Crawlee it does not by default. BasicCrawler hands the request handler a sendRequest helper wired to a session-backed cookie jar — but the client that actually performs the request discards it:

// @crawlee/core/http_clients/got-scraping-http-client.js
const gotResult = await gotScraping({ ...request, cookieJar: undefined, ... });

The comment there explains that HttpCrawler pre-reads cookies into request.gotOptions, which is true for HttpCrawler but not for BasicCrawler's sendRequest. The result, confirmed on the wire, is that no request ever carried a Cookie header and no Set-Cookie was ever stored — Reddit issued a fresh edgebucket on every single hop. So every request re-solved the challenge, and since Reddit rate-limits solves hard, the actor could not get past its own first page. main.ts therefore reads and writes session.cookieJar by hand around each sendRequest call.

The session that holds the cookie has to be handed back out. Crawlee's session pool defaults to 1000 sessions and returns a brand-new one until the pool fills, which would defeat the reuse even with cookies working. main.ts pins the pool small (one session with no proxy, where every session shares the same IP anyway; four with a proxy, where each session is a distinct IP worth rotating into).

Same crawl, measured end to end: before, every run failed on its first request with a hard 403 and delivered nothing; after, 28/28 requests succeed with no retries at all, 24 posts, about 9 seconds — repeated three times back to back, which is exactly the pattern that used to fail deterministically.

The second mechanism — the reCAPTCHA interstitial — shares none of that. It is a normal HTML page (HTTP 200, ~167 KB) containing a g-recaptcha widget bound to Reddit's site key, POSTing back to the same path with ?captcha=1. There is no seed and nothing to compute: passing it requires a genuine reCAPTCHA token. challenge.ts only detects it, keying on the reCAPTCHA widget rather than the English heading so a localised page is still caught. redditClient.ts checks for it on both hops — before and after solving the JS challenge — and raises RedditBlockedError, which is what drives session retirement and backoff.

Detection is the whole point here. The interstitial parses fine as HTML and simply contains no posts, so an actor that does not recognise it will count the fetch as a success, find zero results, and finish green with an empty dataset.

When a request comes back blocked (hard 403 or the reCAPTCHA interstitial, not the solvable challenge), main.ts retires the current session and waits — exponential backoff — before the automatic retry, so the retry gets a different underlying IP from the proxy pool and gives a rate-limited IP time to recover rather than repeating the same bad one immediately.

Relevance filtering

Reddit's search results mix in some unrelated "recommended"/related content alongside genuine query matches — verified directly (e.g. a completely unrelated monitor-review post returned for a "Tesla" search, with zero mention of the word anywhere in its title or body). normalization/relevance.ts checks, after a post's full text is fetched, whether any word from each matched query actually appears in the title or body; posts that fail this check for every query they were discovered under are dropped before being counted or delivered (logged in the run summary as filteredOutAsIrrelevant). The check is deliberately lenient (any query word, not the whole phrase) so it doesn't wrongly drop a genuine match that only restates part of a multi-word query — verified against real output where the match was legitimately body-only (e.g. "Tesla" appearing only in a stock portfolio listing, not the post title).

This filter only applies to posts discovered via search (source: "reddit_search"); posts supplied directly via postUrls (source: "direct_url") were explicitly requested and are never filtered.

Data delivery and charging

Actor.pushData() always runs for every relevant post found — it is never gated on Actor.charge() succeeding. This matters because Actor.charge() is a no-op (with a logged warning, not an error) on any actor version where pay-per-event monetization hasn't been configured in Apify Console yet; earlier code mistakenly treated a charge as a precondition for saving data, which meant every found post was silently discarded on an unmonetized actor. Charging is now best-effort and independent of delivery: it fires when monetization is configured and does nothing but log otherwise, either way every relevant post reaches the dataset.

Known limitations

IP reputation varies, including within Apify's own proxy pools. Session rotation and backoff on block substantially improve odds but don't guarantee every run succeeds — a non-Apify proxy (mobile or premium residential) still gives more consistent results. See "How it works" above.
Reddit's reCAPTCHA interstitial cannot be solved by this actor, by design. Once an IP is drawing it, that IP is unusable until it recovers; the only remedies are waiting or a larger/better proxy pool. A run where every request ends at the interstitial fails loudly — it does not return a partial or empty dataset dressed up as success.
Running without a proxy works from a clean consumer IP, but is fragile. Earlier versions of this note blamed Reddit for what was actually the dropped-cookie bug described above. With the cookie carried properly, a 28-request unproxied run completes with zero permanent failures and, in practice, zero retries — verified three times back to back. It is still a single IP against a per-IP rate limiter with no pool to rotate into, so a much larger crawl, or an IP that is already soured, can still exhaust its retries. Apify's platform IPs are shared and hotter with Reddit than a consumer IP; for runs on the platform, leave proxyConfiguration at its default (Apify proxy on) or supply your own.
The cookie fix depends on a Crawlee internal staying as it is. main.ts carries cookies by hand specifically because GotScrapingHttpClient sets cookieJar: undefined. If a future Crawlee release fixes that, this code keeps working (the jar is the session's own, so the two agree rather than conflict) — but the comment explaining why the manual handling exists would become stale, so check it when upgrading Crawlee.
Pagination uses a cursor param scraped from a lazy-loaded partial in each page's HTML; undocumented and could change without notice — it already has once. The endpoint is /svc/shreddit/search/?...&cursor=... for a sitewide search but /svc/shreddit/r/<subreddit>/search/?...&cursor=... when the search is restricted to a subreddit, and the parser has to accept both.
Comment extraction depends on Reddit's <shreddit-comment> custom-element markup and attribute names, which are also undocumented and could change.
If Reddit changes either challenge mechanism (a harder/real proof-of-work, different seed encoding, a different interstitial), crawler/challenge.ts will need updating — it's isolated specifically so that's a small, contained change. Renaming or adding a hidden form field is already handled, since the fields are copied from the form rather than hardcoded; this happened on 2026-09-03 (token → jsc_token) and took the actor down. A change to the seed's encoding or to the seed + seed transform would still break it.
A failed challenge submission is answered with a hard 403 block page that looks exactly like an IP-reputation block. The two are genuinely hard to tell apart from the run log alone — if blockedByReddit is non-zero, check that the challenge still solves against a freshly fetched page before concluding it is the IP.
When a post's detail page is blocked but its search result was not, the post is still delivered using search-result data alone: text is empty and comments is []. The run summary reports this as the gap between uniquePostsFound and the number of detail fetches that succeeded. Note that an empty text is also perfectly normal for link posts, which genuinely have no body.
The relevance filter is a simple word-overlap heuristic, not true semantic relevance — it will still pass a post that happens to contain a common query word in an unrelated context (e.g. a generic word used as part of a multi-word brand query).

Legal

This Actor is an independent, third-party tool. It is not affiliated with, endorsed by, or sponsored by Reddit, Inc.

It retrieves data by automating Reddit's public search page rather than through Reddit's official Data API (see "How it works" above for why). Reddit's User Agreement restricts automated access to the site outside of that API, so using this Actor may not comply with Reddit's terms. You are responsible for reviewing Reddit's terms and applicable law before using this Actor, and for how you use the data it returns.

Output includes Reddit usernames attached to post/comment content, which can constitute personal data under GDPR and similar laws even though the accounts are pseudonymous. This Actor does not resolve usernames to real identities or collect contact details — but if your use case involves EU/UK individuals, treat the output as personal data for compliance purposes (lawful basis, retention limits, data subject rights) rather than as anonymous text.

Reddit API Scraper

rl1987/reddit-api-scraper

Scrape Reddit posts, comments, search results, subreddits, and user profiles via Reddit's OAuth data API. No login required.

R.L.

Reddit Scraper - Fast, Comments,Data,Subreddits

myagizm/reddit-scraper

Scrape public Reddit posts, comments, users, communities, and search results as structured JSON. No login, no API key.

MYM

Reddit Search Results Scraper

scrapearchitect/reddit-search-results-scraper

Reddit Search Results Scraper

Scrape Architect

Fast Reddit Scraper ($2/1K)

tom_mobi/fast-reddit-scraper

Scrape Reddit posts, comments, subreddits, search results, and user profiles — no API key, no login, no rate limits. From $2 per 1,000 results.

Tom Mobi

5.0

Reddit Scraper

prodiger/reddit-scraper

Extract posts, comments, user profiles, and search results from Reddit. Pure HTTP, no API key required.

Arnas

386

5.0

Reddit Search Scraper — Posts, Comments & Users

logiover/reddit-search-scraper

Scrape Reddit subreddit search with no API key or login. Export posts and comments to CSV/JSON — a Reddit API alternative for keyword monitoring.

Logiover

Reddit Scraper — Posts & Comments

signalengine/reddit-scraper

Scrape posts and comments from any subreddit — no Reddit API key, no login, no proxy. A fast, free Reddit API alternative for public data, exported to JSON, CSV or Excel.

James Taylor

Reddit Scraper - Posts, Comments, Users & Search

benthepythondev/reddit-scraper

Scrape public Reddit posts, comments, user activity, subreddits, and search results through Reddit's app-only OAuth API. Export nested threads, scores, media, Markdown, and token counts to JSON, CSV, or Excel. No Reddit login or user-supplied API key required.