Reddit Scraper (Search, Post Detail, Comments)

Scrape Reddit search results, post details, and comment threads without touching Reddit's paid Data API. This Apify Actor drives Reddit's own www.reddit.com search page and the internal shreddit endpoints its web app uses for pagination, and gives you the results as clean, structured JSON you can run on a schedule, pull through the Apify API, or push straight into a spreadsheet/BI tool via Apify's integrations.

Why use this Reddit Scraper?

• Find posts by keyword across all of Reddit, the same way reddit.com/search does, without opening a browser.
• Pull a specific post's detail fields (title, author, subreddit, score, comment count, body text) from a URL you already have.
• Pull a post's comment thread, including "load more comments" pagination, sorted the same way Reddit's UI sorts it (Best, Top, New, Controversial, Q&A, Old).
• Run it on Apify's schedule/webhook/API tooling to track a keyword or a thread over time, instead of re-scraping by hand.

Important: read before you use this

Reddit's User Agreement explicitly prohibits automated scraping without Reddit's prior written consent, and Reddit sells access to the same data through its official Data API. This Actor talks to Reddit's own internal web endpoints (not the Data API), which carries real ToS risk. Use this for personal projects, research, or learning, not for republishing/reselling scraped data or running it at a scale that could get your IP or account flagged. If you need a fully sanctioned integration, register an app at reddit.com/prefs/apps and use the official Data API instead.

Reddit also runs active anti-bot defenses (a JS "please wait for verification" challenge, and outright IP blocks). This Actor solves the simple verification challenge automatically, but a determined block will still show up as a failed run with a clear error message rather than silent empty output - see Known limitations below.

How to use Reddit Scraper

1. Open the Actor's Input tab and pick a Mode: search, postDetail, or comments.
2. Fill in the fields for that mode (see the Input table below - every field stays visible regardless of mode, but only the ones prefixed (mode: ...) for your chosen mode matter).
3. Click Start. Results stream into the run's Dataset as they're scraped.
4. For postDetail or comments, copy the url field from a search run's output straight into postUrl.
5. Download the dataset in JSON, CSV, Excel, or HTML from the Storage tab, or pull it via the Apify API for automation.

Input

Field	Mode	Description
mode	all	search, postDetail, or comments
query	search	Search keyword
maxItems	search	Max posts to return (default 25)
maxPages	search	Max pagination requests (default 3)
postUrl	postDetail, comments	Full URL or permalink of the post
commentSort	comments	confidence (Best), top, new, controversial, qa, old
maxComments	comments	Max comments to return (default 100)
maxCommentPages	comments	Max "load more comments" requests (default 5)
cookieHeader	all, optional	Raw browser Cookie: header, improves reliability against Reddit's anti-bot checks
proxyConfiguration	all, optional	Apify Proxy configuration
requestDelaySecs	all	Delay between paginated requests, default 1.5s

Example search input:

{
  "mode": "search",
  "query": "minecraft",
  "maxItems": 25
}

Output

Example dataset item for mode: search:

{
  "postId": "lyub6f",
  "title": "Playing Minecraft at 12 years old starter pack",
  "subreddit": "starterpacks",
  "author": "someuser",
  "url": "https://www.reddit.com/r/starterpacks/comments/lyub6f/playing_minecraft_at_12_years_old_starter_pack/",
  "score": 1234,
  "numComments": 56,
  "createdAt": "2021-03-01T12:00:00.000Z",
  "postType": "image",
  "thumbnailUrl": null
}

Example dataset item for mode: comments:

{
  "commentId": "gpzk3h3",
  "parentId": "lyub6f",
  "author": "commenter1",
  "score": 10,
  "depth": 0,
  "createdAt": "2021-03-01T13:00:00.000Z",
  "permalink": "https://www.reddit.com/r/starterpacks/comments/lyub6f/x/gpzk3h3/",
  "bodyText": "Nice post!"
}

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel from the Apify Console, or fetch it through the Apify API.

Data table

Field	Modes	Meaning
postId	search, postDetail	Reddit's base36 post ID
title / bodyText	search, postDetail	Post title / self-text (or link description)
subreddit / author	search, postDetail, comments	Subreddit name and post/comment author
score / numComments	search, postDetail	Upvote score and comment count
commentId / parentId / depth	comments	Comment ID, parent comment/post ID, nesting depth

Cost estimation

This Actor makes plain HTTP requests (no browser rendering), so it runs cheap and fast. A search run for 25 posts is typically 1-2 requests; a comments run depends on thread size and maxCommentPages. Check the Apify pricing page for your plan's compute unit costs, and use maxItems / maxPages / maxComments / maxCommentPages to cap usage.

Tips

• Set requestDelaySecs higher (3-5s) if you see failed runs - Reddit's anti-bot sensitivity varies over time and by IP.
• Enable proxyConfiguration (Apify Proxy) if you're running this more than occasionally from the same IP.
• Supply cookieHeader (copied from your browser's DevTools Network tab while logged into reddit.com) for the most reliable results.

Environment variables

• REDDIT_COOKIE (optional) - same purpose as the cookieHeader input field, useful if you want to set it once at the Actor level (Actor > Settings > Environment variables) instead of per run.

Known limitations

• Parsing was not live-verified against Reddit during development - the dev environment got IP-blocked by Reddit's anti-bot system mid-build (see my_actor/parsers.py docstring). Parsing targets Reddit's documented shreddit-post / shreddit-comment custom elements with a URL/meta-tag fallback, but if Reddit's markup has since changed, run one apify run per mode and check the dataset before relying on this in production.
• Reddit's internal endpoints and challenge format can change without notice; this is not the official Data API and isn't guaranteed to keep working.
• Comment pagination cursors and CSRF tokens are Reddit session-specific and are always re-extracted fresh per request - a stale cookieHeader will still work for basic requests but may reduce pagination depth.

FAQ

Is this legal? Reddit's ToS restricts automated scraping (see the notice above). This tool is intended for personal/research use of publicly visible content. Consult a lawyer for commercial use cases.

Why did my run fail with "Reddit blocked the request"? Reddit's anti-bot system flagged the request. Try enabling proxyConfiguration, supplying a fresh cookieHeader, and/or increasing requestDelaySecs.

Found a bug or have a feature request? Use the Actor's Issues tab to report it.