Reddit Search Scraper (Apify Actor)

Searches Reddit for one or more keyword queries and returns enriched post objects (and optionally comments) as structured JSON in the Actor dataset. Uses Reddit's public .json endpoints — no OAuth/API key required.

Input

{
  "queries": ["Cheesecake", "Swimming Pool"],
  "maxPosts": 100,
  "maxComments": 100,
  "sort": "relevance",
  "timeframe": "all",
  "subredditSort": "relevance",
  "subredditTimeframe": "all",
  "includeNsfw": false,
  "scrapeComments": false,
  "strictSearch": false,
  "strictTokenFilter": false,
  "maximize_coverage": false,
  "forceSortNewForTimeFilteredRuns": false,
  "content_analysis": false,
  "sentiment_analysis": false
}

Field	Type	Meaning
queries	string[]	Keywords/phrases to search. A value like r/pics cats restricts the search to a subreddit.
maxPosts	int	Max posts collected per query (paginates in pages of 100).
sort / timeframe	enum	Reddit search sort (relevance/hot/top/new/comments) and time window (all/year/month/week/day/hour).
subredditSort / subredditTimeframe	enum	Same, applied when a query targets a specific r/subreddit.
strictSearch	bool	Keep only posts where every query token appears in the title/body.
strictTokenFilter	bool	Keep only posts containing the exact query phrase.
maximize_coverage	bool	Sweep multiple sort orders to gather more unique posts.
forceSortNewForTimeFilteredRuns	bool	Force new sort whenever timeframe != all.
includeNsfw	bool	Include over_18 posts.
scrapeComments / maxComments	bool / int	Also fetch and flatten each post's comment tree.
content_analysis	bool	Attach a content_analysis object (keywords, token stats, question/URL detection).
sentiment_analysis	bool	Attach a lexicon-based sentiment object (positive/negative/neutral).

Output

Each dataset item is either a post ("kind": "post") or a comment ("kind": "comment"). Posts include the raw Reddit fields plus derived metrics:

• age_hours, score_per_hour, comments_per_hour
• engagement_total, comment_to_score_ratio, is_high_engagement
• media_type, has_media, gallery_images, gallery_count, outbound_url_host
• title_length, body_length, word_count
• is_deleted_or_removed, content_flags
• canonical_url, old_reddit_url, retrieved_at

See src/enrich.js for the exact shape.

How it avoids being blocked

Reddit returns a hard 403 to plain HTTP clients (even ones that impersonate a TLS fingerprint) because they can't solve Reddit's JS/Cloudflare "Please wait for verification" challenge. This Actor drives a real headless Chromium (Playwright) session: it loads a Reddit page so the browser solves the challenge and earns a valid session cookie, then fetches the .json endpoints from inside the page (same-origin, credentials: 'include'). One warmed session is reused for all search pagination and comment fetches; on a block/403 it relaunches with a fresh residential proxy. See RedditBrowser in src/reddit.py.

Project layout (Python)

.actor/actor.json          Actor config
.actor/input_schema.json   Input UI
Dockerfile                 apify/actor-python-playwright:3.13 (Chromium preinstalled)
requirements.txt           apify (pydantic/browserforge pinned)
src/__main__.py            Entry point (python -m src)
src/main.py                Orchestration + Playwright session lifecycle
src/reddit.py              RedditBrowser (browser fetch) + search/pagination
src/comments.py            Comment scraping
src/enrich.py              Derived fields
src/analysis.py            Content + sentiment analysis

Run locally

pip install -r requirements.txt
apify run          # or: python -m src   (with local input in storage/)

Deploy

apify login
apify push         # builds a NEW image with the current code

Notes

• Use residential proxies. Reddit blocks datacenter IPs; the default proxyConfiguration is useApifyProxy + RESIDENTIAL.
• Because it runs a real browser, give the Actor enough memory (≥ 2–4 GB).
• If a network blocks/redirects Reddit itself (e.g. some ISPs), local runs will fail — validate on the Apify platform where the residential proxy is used.
• Content/sentiment analysis are lightweight offline heuristics — replace with a real NLP service if you need higher accuracy.