Ultimate Reddit Scraper & MCP Server 🚀

A god-tier, highly optimized, unblockable Reddit web scraper built to bypass the 403 API limitations and extract deep insights, comment threads, and rich metadata. Designed specifically for AI Workflows, RAG systems, and data scientists.

🌟 Key Features

1. Hybrid Stealth Engine (Unblockable)

Reddit actively blocks most generic API scrapers with 403 Forbidden errors. This Actor completely bypasses Reddit's Cloudflare and bot-protection mechanisms by launching a headless Playwright browser to silently solve challenges and steal secure reddit_session cookies. It then passes those cookies into an ultra-fast HTTP crawler to scrape APIs at 50+ pages per second.

2. Native AI Summarization & Sentiment (Phase 4)

Built-in LLM processing. Provide an openaiApiKey to automatically analyze posts on the fly. The Actor will natively output a aiSummary (1-sentence TL;DR) and a sentiment tag (Positive, Negative, Neutral, Toxic) for every post!

3. User Profile Deep-Scraping (Phase 4)

Target specific users (e.g. https://www.reddit.com/user/spez/) to extract their entire post and comment history seamlessly into your dataset.

4. Real-Time Webhooks (Phase 4)

Instantly forward every scraped post to Slack, Discord, or a custom API endpoint by providing a webhookUrl. Ideal for brand monitoring.

5. Media Downloader (Phase 4)

Enable downloadMedia to physically download binary image (.jpg, .png, .gif) and video (.mp4) files directly to the Apify Key-Value Store instead of just linking them.

6. Multi-Sort Deep Scraping

Reddit restricts list endpoints to a maximum of 1,000 items. To get more than 1,000 posts from a subreddit, this Actor utilizes a powerful Multi-Sort Engine. By enabling all_sorts, the Actor will seamlessly traverse through hot, new, top (all time), top (year), top (month), and top (week)—effectively scraping up to 6,000+ posts per subreddit in a single run.

7. RAG-Optimized Markdown Export

Forget trying to parse deeply nested JSON comment trees into your LLM. When you enable exportMarkdown, this Actor stitches every post and its deeply threaded comments into a beautiful, hierarchical .md document. It is saved directly to your Apify Key-Value store under RAG_EXPORT.md, ready to be dropped straight into ChatGPT, Claude, Cursor, or LangChain.

8. Authenticated Scraping (NSFW Access)

Reddit now strictly gates adult and quarantined communities behind a login wall. You can bypass this by simply passing your own logged-in reddit_session cookie into the secure input field. The Hybrid Stealth Engine will dynamically assume your identity to crawl gated content.

9. PullPush Archive Fallback

Moderators and users delete valuable information constantly. The Actor has native integration with api.pullpush.io. If it detects [deleted] or [removed] content, it silently pings the archive to recover the original author and text!

10. Native MCP Server Mode

This is not just a scraper; it's a Model Context Protocol (MCP) Server. By enabling mcpMode, the Actor transforms into a live Server-Sent Events (SSE) server that exposes powerful tools like search_reddit directly to AI Agents on your network.

---

🛠️ Detailed Input Configuration

The Actor allows extremely granular control over the crawling behavior:

Field	Type	Description
startUrls	Array	List of specific URLs to start the scrape from.
subredditUrls	Array	List of subreddits to deeply extract.
userUrls	Array	Specific Reddit usernames or profile URLs to scrape.
searchTerms	Array	Specific keywords to search across the entire Reddit network.
sortMode	Dropdown	Allows you to target hot, new, top_all, etc. Select all_sorts to bypass the 1,000 item limit.
maxPostsCount	Number	Total maximum number of posts to pull. Set to 0 for infinite scraping.
maxCommentsCount	Number	Maximum total comments to scrape.
crawlCommentsPerPost	Boolean	If true, fetches deeply nested comment threads for every scraped post.
enableAiFeatures	Boolean	Toggle native LLM sentiment analysis and TL;DR summarization.
openaiApiKey	String	Your OpenAI secret key (Required if AI features enabled).
downloadMedia	Boolean	Save raw media buffers to Key-Value Store.
webhookUrl	String	POST every scraped post instantly to this endpoint (Slack/Discord/API).
redditSessionCookie	String	(Secure/Hidden) Inject your personal reddit_session to bypass NSFW filters.
incrementalMode	Boolean	Ideal for Apify scheduling; skips data scraped in the last run.
startDate / endDate	String	Time-travel filters. E.g. 2024-01-01 to only grab posts from 2024 onward.
exportMarkdown	Boolean	Generates a clean RAG-ready .md file in the Apify Key-Value Store.
mcpMode	Boolean	Turns the scraper into a persistent SSE MCP Server.

---

📊 Dataset Output Schema

The JSON output is rigorously structured and robust. The dataset is specially configured with an Output Schema View so that it displays a clean, readable table in the Apify Console.

Key extracted fields include:

• dataType: Classifies the row as "post" or "comment".
• id: The unique Reddit ID.
• title: The post title (Null for comments).
• body: The self-text or comment content.
• authorName: The Reddit username.
• subredditName: The community it was posted in.
• upVotes: Total upvotes on the post/comment.
• commentsCount: Total number of comments attached to the post.
• mediaUrls: Direct links to high-res images and MP4 videos attached to the post.
• downloadedMediaUrls: Direct links to physically downloaded media on Apify KV store.
• aiSummary: If enabled, a 1-sentence OpenAI TL;DR.
• sentiment: If enabled, the AI-determined sentiment (Positive/Negative/Toxic).
• createdAt: ISO 8601 Timestamp.

💸 Pay-Per-Event (PPE)

This Actor utilizes Apify's revolutionary Pay-Per-Event pricing model. You only pay for the exact amount of data you successfully extract!

🚨 Critical Note on Proxies

Reddit has extremely aggressive bot-protection. If you run this Actor using standard Datacenter Proxies (the default), your scraper will likely be blocked by Cloudflare (yielding 0 results) because Cloudflare instantly flags datacenter IPs.

To guarantee successful scraping: You MUST run this Actor on the Apify platform using Residential Proxies or provide your own Residential proxy URLs. Residential IPs look like real human devices and allow the Hybrid Stealth Engine to easily bypass the blocks and scrape the data!