Reddit Comment Scraper

Extract Reddit comment trees, discussion hierarchies, and sentiment analysis at massive scale.

Pure HTTP Tree Walker · Crawlee & got-scraping · Zero Browser Overhead (<150MB RAM) · Apify Residential Proxy Rotation

---

⚡ Why this Actor?

Feature	What you get
Pure HTTP Engine (No Browser)	10x faster and 90% cheaper than browser scrapers (consumes <150MB RAM vs 1GB+ with Playwright).
Anti-Bot TLS Fingerprinting	Uses got-scraping to negotiate genuine browser TLS Client Hello (JA3/JA4) and HTTP/2 headers to bypass Fastly CDN blocks.
Residential Proxy Auto-Rotation	Seamlessly integrates with Apify RESIDENTIAL proxies; retires session and swaps to a clean IP immediately upon 429/403.
Deep Collapsed Comments	Automatically calls POST /api/morechildren in batches of 100 to extract deeply buried comments often missed by basic scrapers.
Full Thread Hierarchy	Outputs threadId (root comment), ancestorIds path array, and depth for zero-friction tree reconstruction in Pandas or BI tools.
Lightweight Sentiment Analysis	Evaluates in-memory polarity scores (score, comparative, label) for every comment at <0.01ms latency without expensive AI tokens.
Link & Media Extractor	Parses markdown hyperlinks, images, videos, and embedded GIFs (i.redd.it, Giphy, Imgur).
Multi-Target Ingestion	Supports direct post URLs, shortlinks (redd.it), raw post IDs, or whole subreddits.

---

🚀 Quick start

Apify Console

1. Open Reddit Comment Scraper in the Apify Store and click Try for free or Start.
2. Paste target Reddit post URLs or subreddit names in the input form.
3. Click Save & Run.
4. Download structured results in JSON, CSV, or Excel from Storage → Dataset.

---

🔄 How it works

Input (URLs / Subreddits / Post IDs)
  │
  ▼
[Input Normalizer] ──► Deduplicate IDs & extract endpoints
  │
  ▼
[Crawlee HttpCrawler] (got-scraping + SessionPool + Apify Residential Proxies)
  │
  ├──► GET /comments/{id}.json?raw_json=1&limit=500
  │      │
  │      ├─► [Comment Tree Parser] ──► Extract authors, body, scores, depth
  │      │
  │      ├─► [Sentiment Service] ──► In-memory polarity score & label
  │      │
  │      └─► [Hierarchy Mapper] ──► Calculate threadId & ancestorIds
  │
  ├──► POST /api/morechildren (Unfold collapsed comments in 100-ID batches)
  │
  ▼
[Streaming Dataset Output] ──► Actor.pushData() in 100-record chunks

---

📥 Input Reference

{
  "startUrls": [
    { "url": "https://www.reddit.com/r/technology/comments/1i3b62l/openai_operator_live_demo/" }
  ],
  "sort": "top",
  "maxCommentsPerPost": 100,
  "maxDepth": 10,
  "expandMoreComments": true,
  "enableSentiment": true,
  "extractMedia": true,
  "minScore": 0,
  "skipDeleted": true,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": ["RESIDENTIAL"]
  }
}

Field	Type	Default	Description
startUrls	Array	[]	List of target Reddit post URLs.
subreddits	Array	[]	Subreddit names to fetch latest posts from (e.g. technology, webdev).
postIds	Array	[]	Raw Reddit post IDs (e.g. 1i3b62l).
postsPerSubreddit	Integer	10	Number of recent posts to crawl when subreddits are provided.
sort	String	"confidence"	Comment sort: confidence (Best), top, new, controversial, old, qa.
maxCommentsPerPost	Integer	100	Max comments to scrape per post (0 = all reachable).
maxDepth	Integer	10	Max thread reply depth.
expandMoreComments	Boolean	true	When true, fetches collapsed "Load more" comments.
minScore	Integer	0	Excludes comments below this upvote threshold.
skipDeleted	Boolean	true	Excludes [deleted] and [removed] comments.
keywordFilter	Array	[]	Keeps only comments containing specified terms.
dateFrom / dateTo	String	null	ISO date filter range (YYYY-MM-DD).
enableSentiment	Boolean	true	Computes in-memory sentiment scores and labels.
extractMedia	Boolean	true	Parses hyperlinks and media URLs from markdown.
proxyConfiguration	Object	RESIDENTIAL	Apify Residential proxy configuration.
maxConcurrency	Integer	15	Concurrency limit (10–30 recommended).

---

📤 Output Contract

Every scraped record pushed to the dataset adheres to this structure:

{
  "id": "t1_k9z1abc",
  "postId": "t3_1i3b62l",
  "postTitle": "OpenAI Operator Live Demo",
  "postUrl": "https://www.reddit.com/r/technology/comments/1i3b62l/openai_operator_live_demo/",
  "postAuthor": "tech_editor",
  "subreddit": "technology",
  "subredditId": "t5_2qh16",
  "parentId": "t1_k9y0xyz",
  "threadId": "t1_k9top1",
  "ancestorIds": ["t1_k9top1", "t1_k9y0xyz"],
  "author": "dev_guru",
  "authorFullname": "t2_8q4l1",
  "authorFlairText": "Software Engineer",
  "body": "This is great progress! The speed and accuracy are impressive: https://example.com/demo",
  "bodyHtml": "<p>This is great progress! The speed and accuracy are impressive: <a href=\"https://example.com/demo\">https://example.com/demo</a></p>",
  "score": 45,
  "createdUtc": 1772755200,
  "createdAtIso": "2026-03-06T00:00:00.000Z",
  "permalink": "/r/technology/comments/1i3b62l/openai_operator_live_demo/k9z1abc/",
  "url": "https://www.reddit.com/r/technology/comments/1i3b62l/openai_operator_live_demo/k9z1abc/",
  "isSubmitter": false,
  "distinguished": null,
  "stickied": false,
  "depth": 2,
  "replyCount": 3,
  "edited": false,
  "wordCount": 12,
  "charCount": 92,
  "sentiment": {
    "score": 6,
    "comparative": 0.5,
    "label": "positive"
  },
  "extractedUrls": ["https://example.com/demo"],
  "mediaUrls": []
}

---

💰 Cost, Limits & Proxies

Metric	Specification	Note
Container Memory	512 MB	Sufficient for high-concurrency HTTP streaming.
Concurrency	15 – 30 workers	Adjust via maxConcurrency.
Speed	30 – 60 comments/sec	Varies based on residential proxy latency.
Compute Cost	~$0.01 – $0.02	Per 10,000 comments scraped.
Proxy Bandwidth	~15 MB	Per 10,000 comments (~$0.15 on Apify Residential proxy).
Proxy Type	Apify RESIDENTIAL	Strongly recommended. Datacenter IPs are blocked by Reddit.

---

🍳 Recipes

Recipe 1: Single Thread Deep Dive

{
  "startUrls": [{ "url": "https://www.reddit.com/r/webdev/comments/17abcde/slug/" }],
  "maxCommentsPerPost": 0,
  "maxDepth": 20,
  "expandMoreComments": true,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

Recipe 2: Subreddit Topic Sentiment Mining

{
  "subreddits": ["technology", "artificial"],
  "postsPerSubreddit": 25,
  "maxCommentsPerPost": 100,
  "sort": "top",
  "enableSentiment": true,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

---

🏛️ Architecture

src/
├── main.ts                       # Composition & Actor lifecycle
├── domain/                       # Entity schemas & sentiment logic
│   ├── comment.entity.ts         # TypeScript models
│   └── sentiment.service.ts      # In-memory polarity calculator
├── input/                        # Trust boundary input validation
│   └── comment-input.ts          # URL normalizer & deduplicator
├── parsers/                      # Hierarchy & regex parsers
│   └── comment-tree.parser.ts    # Recursive flattener & tree mapping
└── clients/                      # Network & anti-bot transport
    └── reddit-json.client.ts     # got-scraping with 429/403 backoff & morechildren

---

🛠️ Development & Testing

# Run unit tests
npx vitest run

# Validate all schemas
apify validate-schema

# Run readiness audit
node scripts/validate-actor-readiness.js

---

❓ Frequently Asked Questions (FAQ)

Q: Do I need a Reddit account or Reddit API token?
A: No. The Actor retrieves public discussions directly without requiring personal accounts or paid official Reddit API credentials.

Q: Why do I need Residential Proxies?
A: Reddit's CDN (Fastly) blocks datacenter IP blocks (AWS, Google Cloud, DigitalOcean) with HTTP 403 Forbidden. Apify Residential proxies route requests through genuine residential IPs to ensure unblocked operation.

Q: How does the Actor handle collapsed or "load more" comments?
A: The Actor identifies kind: "more" objects and automatically batches IDs into POST /api/morechildren requests (up to 100 IDs per call) until all reachable comments are retrieved.

Q: Can I use this for academic or commercial NLP datasets?
A: Yes. The output includes threadId and ancestorIds, allowing you to reconstruct tree graphs for argument mining, conversational AI, and sentiment analysis.

Q: Will this actor get blocked by Reddit?
A: The Actor uses got-scraping to negotiate real browser TLS Client Hello signatures and headers, combined with residential IP rotation on 429 or 403.

---

⚖️ Legal & Limitations

• Public Data Only: This Actor extracts publicly available comments and submissions. It cannot access private subreddits or quarantined/banned communities.
• GDPR & Privacy: Scraped data contains publicly posted usernames. Users are responsible for complying with relevant data protection regulations.