# Reddit MCP Scraper (`tillzero/reddit-mcp-scraper`) Actor Scrape Reddit posts, comments, user profiles, and subreddits without logging in. Features an unblockable Hybrid Stealth Engine to bypass Cloudflare 403s. Built-in AI summarization, deep multi-sort crawling, media downloads, and native MCP support for AI Agents! - **URL**: https://apify.com/tillzero/reddit-mcp-scraper.md - **Developed by:** [K S](https://apify.com/tillzero) (community) - **Categories:** AI, Developer tools, Social media - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $3.00 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Ultimate Reddit Scraper & MCP Server πŸš€ A god-tier, highly optimized, unblockable Reddit web scraper built to bypass the 403 API limitations and extract deep insights, comment threads, and rich metadata. Designed specifically for AI Workflows, RAG systems, and data scientists. ### 🌟 Key Features #### 1. Hybrid Stealth Engine (Unblockable) Reddit actively blocks most generic API scrapers with `403 Forbidden` errors. This Actor completely bypasses Reddit's Cloudflare and bot-protection mechanisms by launching a headless **Playwright** browser to silently solve challenges and steal secure `reddit_session` cookies. It then passes those cookies into an ultra-fast HTTP crawler to scrape APIs at 50+ pages per second. #### 2. Native AI Summarization & Sentiment (Phase 4) Built-in LLM processing. Provide an `openaiApiKey` to automatically analyze posts on the fly. The Actor will natively output a `aiSummary` (1-sentence TL;DR) and a `sentiment` tag (Positive, Negative, Neutral, Toxic) for every post! #### 3. User Profile Deep-Scraping (Phase 4) Target specific users (e.g. `https://www.reddit.com/user/spez/`) to extract their entire post and comment history seamlessly into your dataset. #### 4. Real-Time Webhooks (Phase 4) Instantly forward every scraped post to Slack, Discord, or a custom API endpoint by providing a `webhookUrl`. Ideal for brand monitoring. #### 5. Media Downloader (Phase 4) Enable `downloadMedia` to physically download binary image (`.jpg`, `.png`, `.gif`) and video (`.mp4`) files directly to the Apify Key-Value Store instead of just linking them. #### 6. Multi-Sort Deep Scraping Reddit restricts list endpoints to a maximum of 1,000 items. To get more than 1,000 posts from a subreddit, this Actor utilizes a powerful **Multi-Sort Engine**. By enabling `all_sorts`, the Actor will seamlessly traverse through `hot`, `new`, `top (all time)`, `top (year)`, `top (month)`, and `top (week)`β€”effectively scraping up to **6,000+ posts per subreddit** in a single run. #### 7. RAG-Optimized Markdown Export Forget trying to parse deeply nested JSON comment trees into your LLM. When you enable `exportMarkdown`, this Actor stitches every post and its deeply threaded comments into a beautiful, hierarchical `.md` document. It is saved directly to your Apify Key-Value store under `RAG_EXPORT.md`, ready to be dropped straight into ChatGPT, Claude, Cursor, or LangChain. #### 8. Authenticated Scraping (NSFW Access) Reddit now strictly gates adult and quarantined communities behind a login wall. You can bypass this by simply passing your own logged-in `reddit_session` cookie into the secure input field. The Hybrid Stealth Engine will dynamically assume your identity to crawl gated content. #### 9. PullPush Archive Fallback Moderators and users delete valuable information constantly. The Actor has native integration with `api.pullpush.io`. If it detects `[deleted]` or `[removed]` content, it silently pings the archive to recover the original author and text! #### 10. Native MCP Server Mode This is not just a scraper; it's a **Model Context Protocol (MCP) Server**. By enabling `mcpMode`, the Actor transforms into a live Server-Sent Events (SSE) server that exposes powerful tools like `search_reddit` directly to AI Agents on your network. --- ### πŸ› οΈ Detailed Input Configuration The Actor allows extremely granular control over the crawling behavior: | Field | Type | Description | |---|---|---| | **startUrls** | Array | List of specific URLs to start the scrape from. | | **subredditUrls** | Array | List of subreddits to deeply extract. | | **userUrls** | Array | Specific Reddit usernames or profile URLs to scrape. | | **searchTerms** | Array | Specific keywords to search across the entire Reddit network. | | **sortMode** | Dropdown | Allows you to target `hot`, `new`, `top_all`, etc. Select `all_sorts` to bypass the 1,000 item limit. | | **maxPostsCount** | Number | Total maximum number of posts to pull. Set to `0` for infinite scraping. | | **maxCommentsCount** | Number | Maximum total comments to scrape. | | **crawlCommentsPerPost**| Boolean | If true, fetches deeply nested comment threads for every scraped post. | | **enableAiFeatures** | Boolean | Toggle native LLM sentiment analysis and TL;DR summarization. | | **openaiApiKey** | String | Your OpenAI secret key (Required if AI features enabled). | | **downloadMedia** | Boolean | Save raw media buffers to Key-Value Store. | | **webhookUrl** | String | POST every scraped post instantly to this endpoint (Slack/Discord/API). | | **redditSessionCookie** | String | *(Secure/Hidden)* Inject your personal `reddit_session` to bypass NSFW filters. | | **incrementalMode** | Boolean | Ideal for Apify scheduling; skips data scraped in the last run. | | **startDate / endDate** | String | Time-travel filters. E.g. `2024-01-01` to only grab posts from 2024 onward. | | **exportMarkdown** | Boolean | Generates a clean RAG-ready `.md` file in the Apify Key-Value Store. | | **mcpMode** | Boolean | Turns the scraper into a persistent SSE MCP Server. | --- ### πŸ“Š Dataset Output Schema The JSON output is rigorously structured and robust. The dataset is specially configured with an **Output Schema View** so that it displays a clean, readable table in the Apify Console. Key extracted fields include: - `dataType`: Classifies the row as `"post"` or `"comment"`. - `id`: The unique Reddit ID. - `title`: The post title (Null for comments). - `body`: The self-text or comment content. - `authorName`: The Reddit username. - `subredditName`: The community it was posted in. - `upVotes`: Total upvotes on the post/comment. - `commentsCount`: Total number of comments attached to the post. - `mediaUrls`: Direct links to high-res images and MP4 videos attached to the post. - `downloadedMediaUrls`: Direct links to physically downloaded media on Apify KV store. - `aiSummary`: If enabled, a 1-sentence OpenAI TL;DR. - `sentiment`: If enabled, the AI-determined sentiment (Positive/Negative/Toxic). - `createdAt`: ISO 8601 Timestamp. ### πŸ’Έ Pay-Per-Event (PPE) This Actor utilizes Apify's revolutionary Pay-Per-Event pricing model. You only pay for the exact amount of data you successfully extract! ### 🚨 Critical Note on Proxies Reddit has extremely aggressive bot-protection. If you run this Actor using standard **Datacenter Proxies** (the default), your scraper will likely be blocked by Cloudflare (yielding 0 results) because Cloudflare instantly flags datacenter IPs. **To guarantee successful scraping:** You MUST run this Actor on the Apify platform using **Residential Proxies** or provide your own Residential proxy URLs. Residential IPs look like real human devices and allow the Hybrid Stealth Engine to easily bypass the blocks and scrape the data! # Actor input Schema ## `startUrls` (type: `array`): Specific Reddit URLs to scrape (posts, users, subreddits). E.g. https://www.reddit.com/r/technology/ ## `searchTerms` (type: `array`): Keywords to search across Reddit. ## `subredditUrls` (type: `array`): Names or URLs of subreddits to pull all history from. ## `maxPostsCount` (type: `integer`): Maximum number of posts to scrape per input. ## `maxCommentsCount` (type: `integer`): Maximum total comments to scrape per input. ## `crawlCommentsPerPost` (type: `boolean`): If enabled, extracts full comment threads for every post found. ## `fastMode` (type: `boolean`): Scrapes Reddit JSON endpoints directly instead of loading HTML. Blazing fast. ## `proxy` (type: `object`): Select proxies to be used. ## `mcpMode` (type: `boolean`): If enabled, starts a Server-Sent Events (SSE) server for the Model Context Protocol, allowing AIs to query Reddit in real time. ## `sortMode` (type: `string`): How to sort subreddit posts. ## `redditSessionCookie` (type: `string`): Optional. Your reddit_session cookie for authenticated access (NSFW scraping). ## `incrementalMode` (type: `boolean`): If enabled, stops scraping when encountering a post older than the last run. ## `startDate` (type: `string`): Optional (YYYY-MM-DD or ISO). Only scrape posts created on or after this date. ## `endDate` (type: `string`): Optional (YYYY-MM-DD or ISO). Only scrape posts created before this date. ## `exportMarkdown` (type: `boolean`): Generate a single LLM-optimized Markdown file of all scraped data and save it to Key-Value Store. ## `userUrls` (type: `array`): Reddit usernames or profile URLs to scrape (e.g. https://www.reddit.com/user/spez/). ## `enableAiFeatures` (type: `boolean`): If true, uses OpenAI to generate 1-sentence TL;DRs and sentiment analysis for scraped content. ## `openaiApiKey` (type: `string`): Required if AI Features are enabled. Your OpenAI secret key. ## `downloadMedia` (type: `boolean`): Physically download images/videos to Apify Key-Value store instead of just linking them. ## `webhookUrl` (type: `string`): Optional. Send a POST request to this URL (e.g., Slack/Discord) the moment a post is scraped. ## Actor input object example ```json { "startUrls": [], "searchTerms": [], "subredditUrls": [], "maxPostsCount": 100, "maxCommentsCount": 50, "crawlCommentsPerPost": false, "fastMode": true, "proxy": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] }, "mcpMode": false, "sortMode": "hot", "incrementalMode": false, "exportMarkdown": false, "userUrls": [], "enableAiFeatures": false, "downloadMedia": false } ```` # Actor output Schema ## `dataset` (type: `string`): View the full structured JSON dataset of scraped posts and comments. ## `datasetCsv` (type: `string`): Download the scraped data as a spreadsheet. ## `ragExport` (type: `string`): If exportMarkdown was enabled, view the stitched conversation tree here. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("tillzero/reddit-mcp-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("tillzero/reddit-mcp-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("πŸ’Ύ Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # πŸ“š Want to learn more πŸ“–? Go to β†’ https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call tillzero/reddit-mcp-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=tillzero/reddit-mcp-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Reddit MCP Scraper", "description": "Scrape Reddit posts, comments, user profiles, and subreddits without logging in. Features an unblockable Hybrid Stealth Engine to bypass Cloudflare 403s. Built-in AI summarization, deep multi-sort crawling, media downloads, and native MCP support for AI Agents!", "version": "0.0", "x-build-id": "FhTmIIA4GwB6xXVNP" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/tillzero~reddit-mcp-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-tillzero-reddit-mcp-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/tillzero~reddit-mcp-scraper/runs": { "post": { "operationId": "runs-sync-tillzero-reddit-mcp-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/tillzero~reddit-mcp-scraper/run-sync": { "post": { "operationId": "run-sync-tillzero-reddit-mcp-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "startUrls": { "title": "Start URLs", "type": "array", "description": "Specific Reddit URLs to scrape (posts, users, subreddits). E.g. https://www.reddit.com/r/technology/", "default": [], "items": { "type": "object", "required": [ "url" ], "properties": { "url": { "type": "string", "title": "URL of a web page", "format": "uri" } } } }, "searchTerms": { "title": "Search Terms", "type": "array", "description": "Keywords to search across Reddit.", "default": [], "items": { "type": "string" } }, "subredditUrls": { "title": "Subreddits to deep scrape", "type": "array", "description": "Names or URLs of subreddits to pull all history from.", "default": [], "items": { "type": "string" } }, "maxPostsCount": { "title": "Max Posts", "minimum": 0, "type": "integer", "description": "Maximum number of posts to scrape per input.", "default": 100 }, "maxCommentsCount": { "title": "Max Comments", "minimum": 0, "type": "integer", "description": "Maximum total comments to scrape per input.", "default": 50 }, "crawlCommentsPerPost": { "title": "Crawl comments per post", "type": "boolean", "description": "If enabled, extracts full comment threads for every post found.", "default": false }, "fastMode": { "title": "Fast Mode (JSON API)", "type": "boolean", "description": "Scrapes Reddit JSON endpoints directly instead of loading HTML. Blazing fast.", "default": true }, "proxy": { "title": "Proxy configuration", "type": "object", "description": "Select proxies to be used.", "default": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }, "mcpMode": { "title": "Run as MCP Server", "type": "boolean", "description": "If enabled, starts a Server-Sent Events (SSE) server for the Model Context Protocol, allowing AIs to query Reddit in real time.", "default": false }, "sortMode": { "title": "Sort Mode", "enum": [ "hot", "new", "top_all", "top_year", "top_month", "top_week", "top_day", "all_sorts" ], "type": "string", "description": "How to sort subreddit posts.", "default": "hot" }, "redditSessionCookie": { "title": "Reddit Session Cookie", "type": "string", "description": "Optional. Your reddit_session cookie for authenticated access (NSFW scraping)." }, "incrementalMode": { "title": "Incremental Monitor Mode", "type": "boolean", "description": "If enabled, stops scraping when encountering a post older than the last run.", "default": false }, "startDate": { "title": "Start Date", "type": "string", "description": "Optional (YYYY-MM-DD or ISO). Only scrape posts created on or after this date." }, "endDate": { "title": "End Date", "type": "string", "description": "Optional (YYYY-MM-DD or ISO). Only scrape posts created before this date." }, "exportMarkdown": { "title": "Export Markdown (RAG)", "type": "boolean", "description": "Generate a single LLM-optimized Markdown file of all scraped data and save it to Key-Value Store.", "default": false }, "userUrls": { "title": "User Profiles", "type": "array", "description": "Reddit usernames or profile URLs to scrape (e.g. https://www.reddit.com/user/spez/).", "default": [], "items": { "type": "string" } }, "enableAiFeatures": { "title": "Enable AI Summarization & Sentiment", "type": "boolean", "description": "If true, uses OpenAI to generate 1-sentence TL;DRs and sentiment analysis for scraped content.", "default": false }, "openaiApiKey": { "title": "OpenAI API Key", "type": "string", "description": "Required if AI Features are enabled. Your OpenAI secret key." }, "downloadMedia": { "title": "Download Media Files", "type": "boolean", "description": "Physically download images/videos to Apify Key-Value store instead of just linking them.", "default": false }, "webhookUrl": { "title": "Real-Time Webhook URL", "type": "string", "description": "Optional. Send a POST request to this URL (e.g., Slack/Discord) the moment a post is scraped." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```