Reddit Post Scraper (Bulk, Filtered & Lightweight)

High-throughput, browserless extraction of Reddit posts, categories/flairs, and relational comment trees.
Built with TypeScript, Node.js 20, Crawlee, and got-scraping for ultra-fast performance and minimal compute cost.

---

⚡ Why Reddit Post Scraper?

Most Reddit scrapers on the market rely on heavy headless browsers (Puppeteer or Playwright) that consume 1GB–2GB RAM, crash on dynamic DOM updates, and rack up hefty compute bills.

Reddit Post Scraper operates on a modern, HTTP-first architecture with browser-grade TLS/JA3 impersonation and HTTP/2 session pooling. It extracts thousands of posts per minute at 1/10th the compute cost of browser scrapers.

Feature Comparison

Feature	Reddit Post Scraper	Traditional Browser Scrapers
Memory Footprint	256 MB – 512 MB RAM	1024 MB – 2048 MB RAM
Speed / Latency	150ms – 400ms per request	2.5s – 5.0s per page
Compute Cost	< $0.02 per 10,000 posts	$0.25+ per 10,000 posts
Bulk Subreddits	Yes (parallel array)	Limited / sequential
Category / Specific Flair Filter	Yes (flair: "Discussion")	No (boolean only)
Engagement Thresholds	Yes (minScore, minComments)	No (local post-filtering)
Content Type Filtering	Yes (all, text, media)	No
Relational Comment Tree	Yes (depth, parentCommentId)	Nested JSON or None
Deduplication Monitor Mode	Yes (Apify KVS caching)	No
Browser Dependency	Zero (no Chromium)	Requires Chromium binary

---

🏗️ Architecture Flow

┌────────────────────────────────────────────────────────────────────────┐
│                          Input Configuration                           │
│     [Subreddits: Bulk]   [Search Keywords]   [Direct Reddit URLs]      │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│                   Lightweight HTTP/2 Evasion Engine                    │
│      • got-scraping (Chrome TLS / JA3 / HTTP/2 protocol frames)        │
│      • Apify Residential Proxy Pool (Session retirement on 401/403)    │
│      • Exponential backoff on 429 (Retry-After header handling)        │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                  ┌─────────────────┴─────────────────┐
                  ▼                                   ▼
        [Listing Extraction]               [Optional Comment Tree]
        • 100 posts per page               • Recursive tree traversal
        • Cursor token pagination          • Capped at maxCommentDepth
        • Reverse date early-exit          • Skip [deleted] / [removed]
                  │                                   │
                  └─────────────────┬─────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│                       In-Memory Filter Pipeline                        │
│   ✓ minScore (upvotes)              ✓ Content Type (text vs media)     │
│   ✓ minComments                     ✓ NSFW Inclusion Guard             │
│   ✓ Specific Flair Matching         ✓ KVS Delta Deduplication          │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
┌────────────────────────────────────────────────────────────────────────┐
│                   Structured Apify Dataset Output                      │
│      Clean, relational records ready for AI/LLM pipelines & SQL       │
└────────────────────────────────────────────────────────────────────────┘

---

⚙️ Input Configuration

Parameter	Type	Default	Description
subreddits	array	["technology"]	List of subreddit names or URLs to scrape in bulk.
searchTerms	array	[]	Keywords to search across Reddit or within the selected subreddits.
startUrls	array	[]	Direct URLs to specific posts, comments, or subreddits.
sort	string	"hot"	Sort order: "hot", "new", "top", "rising", or "relevance".
time	string	"all"	Time window for top sort and searches: "hour", "day", "week", "month", "year", "all".
flair	string	""	Filter by specific flair string (e.g. "Discussion", "News", "AI").
postType	string	"all"	Filter content: "all", "text" (self posts), or "media" (images/videos).
minScore	integer	0	Minimum upvote threshold to eliminate zero-engagement noise.
minComments	integer	0	Minimum comment count threshold.
postedAfter	string	""	Filter out posts older than this UTC date (YYYY-MM-DD).
postedBefore	string	""	Filter out posts newer than this UTC date (YYYY-MM-DD).
crawlComments	boolean	false	Enable recursive comment extraction for scraped posts.
maxCommentsPerPost	integer	25	Maximum comments saved per post.
maxCommentDepth	integer	3	Maximum nesting depth for comment replies (1–10).
skipDeletedComments	boolean	true	Skip empty [deleted] and [removed] comments.
includeNsfw	boolean	false	Include 18+ adult content in results.
maxPosts	integer	50	Maximum posts to scrape across all queries.
monitorMode	boolean	false	Cache seen post IDs in Apify KVS to skip duplicates in cron runs.
proxyConfiguration	object	{"useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"]}	Proxy settings.

---

📊 Output Dataset Schema

The scraper emits clean, typed records discriminated by dataType: "post" | "comment".

Sample Post Record (dataType: "post")

{
  "dataType": "post",
  "id": "t3_1h5xabc",
  "postId": "1h5xabc",
  "title": "New breakthroughs in open-source AI models",
  "body": "Detailed discussion about small language models...",
  "author": "tech_researcher",
  "subreddit": "technology",
  "subredditName": "r/technology",
  "score": 1420,
  "upvoteRatio": 0.94,
  "numComments": 312,
  "url": "https://www.reddit.com/r/technology/comments/1h5xabc/new_breakthroughs_in_opensource_ai_models/",
  "permalink": "/r/technology/comments/1h5xabc/new_breakthroughs_in_opensource_ai_models/",
  "postType": "text",
  "flair": "AI & Robotics",
  "mediaUrls": [],
  "isNsfw": false,
  "isPinned": false,
  "createdAt": "2026-09-04T12:30:00.000Z",
  "createdUtc": 1757008200,
  "scorePerHour": 48.96,
  "commentsCount": 312
}

Sample Comment Record (dataType: "comment")

{
  "dataType": "comment",
  "id": "t1_m89abc",
  "commentId": "m89abc",
  "postId": "1h5xabc",
  "parentCommentId": "t3_1h5xabc",
  "subreddit": "technology",
  "author": "commenter_one",
  "body": "The benchmark results on reasoning tasks are particularly impressive.",
  "score": 85,
  "depth": 1,
  "permalink": "https://www.reddit.com/r/technology/comments/1h5xabc/new_breakthroughs_in_opensource_ai_models/m89abc/",
  "createdAt": "2026-09-04T13:10:00.000Z",
  "createdUtc": 1757010600
}

---

💡 Practical Recipes

1. High-Signal Tech Market Intelligence

Scrape high-engagement discussions from tech communities:

{
  "subreddits": ["technology", "programming", "artificial"],
  "sort": "top",
  "time": "month",
  "minScore": 100,
  "minComments": 20,
  "postType": "text",
  "maxPosts": 100
}

2. Category / Flair-Specific Filtering

Extract posts categorized under a specific flair tag (e.g. "Discussion"):

{
  "subreddits": ["machinelearning"],
  "flair": "Discussion",
  "sort": "new",
  "maxPosts": 50
}

3. Scheduled Brand Monitoring (Delta Mode)

Monitor competitor brand mentions every morning without duplicate records:

{
  "searchTerms": ["ChatGPT", "Claude AI", "Gemini"],
  "sort": "new",
  "monitorMode": true,
  "maxPosts": 200
}

---

🛡️ Anti-Bot & Proxy Guide

Reddit enforces strict anti-scraping measures on non-residential IPs. To guarantee 100% reliable execution:

• Apify Residential Proxies (apifyProxyGroups: ["RESIDENTIAL"]) are enabled by default.
• The Actor automatically retires proxy sessions upon receiving HTTP 403 Forbidden to acquire a clean residential IP.
• HTTP 429 rate limit responses are handled with automatic backoff respecting the Retry-After header.

---

❓ Frequently Asked Questions (FAQ)

Q: Do I need a Reddit API key or Reddit account?
A: No! The scraper uses public HTTP endpoints with browser-grade TLS emulation. No Reddit credentials or API registration required.

Q: Can I scrape comments as well as posts?
A: Yes. Toggle crawlComments: true. Comments will be extracted up to your configured maxCommentDepth and saved as relational records.

Q: Why are comments saved as separate records instead of nested inside the post?
A: Flat relational items prevent multi-megabyte JSON memory bloat and make querying in SQL, pandas, or vector databases trivial.

Q: Does it work with private or quarantined subreddits?
A: No. Private and quarantined subreddits require authenticated account membership. The Actor cleanly skips and logs them without crashing.

Q: How do I export data to CSV or Excel?
A: The Apify Console and API provide built-in one-click exports to CSV, JSON, Excel, and XML.