Reddit Search Scraper

Scrapes Reddit search results, post details, and comments with one fast hybrid implementation.

Current Approach

• Playwright opens Reddit first and handles the JavaScript challenge.
• Search results are parsed from the warmed browser page.
• httpcloak reuses the warmed browser cookies for fast HTML post-detail and comment requests.
• Browser detail fallback is used only when HTTP detail/comment HTML is blocked or empty.
• A single browser/context is reused across all queries in one run.
• HTTP detail/comment fetches run with bounded concurrency via detail_workers.

The old .json endpoint implementation was removed because Reddit now blocks those paths in local residential-proxy tests.

Input

{
  "queries": ["python", "openai"],
  "max_posts_per_query": 10,
  "include_comments": true,
  "max_comments_per_post": 50,
  "detail_workers": 4,
  "sort": "relevance",
  "time": "all"
}

Field	Description
queries	Search terms. A single browser session is reused across the batch.
max_posts_per_query	Number of posts to collect per query.
include_comments	Include parsed comment bodies for each post.
max_comments_per_post	Maximum comments to save per post.
detail_workers	Concurrent HTTP detail/comment workers before browser fallback.
sort	Reddit search sort: relevance, hot, new, top, or comments.
time	Reddit time filter: all, hour, day, week, month, or year.

Output

Each dataset item is a post row with metadata and optional comments:

{
  "dataType": "post",
  "id": "abc123",
  "subreddit": "python",
  "author": "example_user",
  "title": "Example Reddit post",
  "selftext": "Post body",
  "url": "https://www.reddit.com/r/python/comments/abc123/example/",
  "permalink": "https://www.reddit.com/r/python/comments/abc123/example/",
  "score": 42,
  "num_comments": 18,
  "created_iso": "2026-06-23T12:00:00+00:00",
  "search_query": "python",
  "search_rank": 1,
  "source": {
    "search": "browser_dom",
    "detail": "http_html",
    "comments": "http_html"
  },
  "comments": [
    {
      "dataType": "comment",
      "id": "t1_example",
      "author": "commenter",
      "body": "Comment body",
      "score": 5,
      "created_iso": "2026-06-23T12:30:00+00:00",
      "depth": 0,
      "permalink": "https://www.reddit.com/r/python/comments/abc123/comment/example/"
    }
  ]
}

Run stats are written to the default key-value store as RUN_STATS.

Local Smoke Test

Set REDDIT_PROXY_URL to a residential proxy URL, then run:

python3 reddit_fast_hybrid.py -q python,openai \
  -n 2 -p 1 -c --max-comments 20 --detail-workers 4 \
  -o /tmp/reddit_fast_hybrid.jsonl

Actor entrypoint local test:

$APIFY_INPUT_FILE=INPUT.json python3 main.py