# Reddit Scraper (Search, Post Detail, Comments) (`ryyos/reddit-scraper`) Actor Search Reddit by keyword and scrape post listings, post detail, and comments via Reddit's shreddit web endpoints unlogin - **URL**: https://apify.com/ryyos/reddit-scraper.md - **Developed by:** [Rio Dwi Saputra](https://apify.com/ryyos) (community) - **Categories:** Other, Developer tools, News - **Stats:** 3 total users, 2 monthly users, 0.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.50 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. - **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use). - **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`. - **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`. - **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`. - **Any other language** — the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README Scrape **Reddit** search results, post details, and comment threads without touching Reddit's paid Data API. This [Apify Actor](https://apify.com/actors) drives Reddit's own `www.reddit.com` search page and the internal `shreddit` endpoints its web app uses for pagination, and gives you the results as clean, structured JSON you can run on a schedule, pull through the Apify API, or push straight into a spreadsheet/BI tool via Apify's integrations. ### Why use this Reddit Scraper? - **Find posts by keyword** across all of Reddit, the same way `reddit.com/search` does, without opening a browser. - **Pull a specific post's detail fields** (title, author, subreddit, score, comment count, body text) from a URL you already have. - **Pull a post's comment thread**, including "load more comments" pagination, sorted the same way Reddit's UI sorts it (Best, Top, New, Controversial, Q\&A, Old). - Run it on Apify's schedule/webhook/API tooling to track a keyword or a thread over time, instead of re-scraping by hand. ### Important: read before you use this Reddit's [User Agreement](https://www.redditinc.com/policies/user-agreement) explicitly prohibits automated scraping without Reddit's prior written consent, and Reddit sells access to the same data through its official [Data API](https://www.reddit.com/dev/api). This Actor talks to Reddit's own internal web endpoints (not the Data API), which carries real ToS risk. **Use this for personal projects, research, or learning, not for republishing/reselling scraped data or running it at a scale that could get your IP or account flagged.** If you need a fully sanctioned integration, register an app at [reddit.com/prefs/apps](https://www.reddit.com/prefs/apps) and use the official Data API instead. Reddit also runs active anti-bot defenses (a JS "please wait for verification" challenge, and outright IP blocks). This Actor solves the simple verification challenge automatically, but a determined block will still show up as a failed run with a clear error message rather than silent empty output - see **Known limitations** below. ### How to use Reddit Scraper 1. Open the Actor's **Input** tab and pick a **Mode**: `search`, `postDetail`, or `comments`. 2. Fill in the fields for that mode (see the **Input** table below - every field stays visible regardless of mode, but only the ones prefixed `(mode: ...)` for your chosen mode matter). 3. Click **Start**. Results stream into the run's **Dataset** as they're scraped. 4. For `postDetail` or `comments`, copy the `url` field from a `search` run's output straight into `postUrl`. 5. Download the dataset in JSON, CSV, Excel, or HTML from the **Storage** tab, or pull it via the Apify API for automation. ### Input | Field | Mode | Description | |---|---|---| | `mode` | all | `search`, `postDetail`, or `comments` | | `query` | search | Search keyword | | `maxItems` | search | Max posts to return (default 25) | | `maxPages` | search | Max pagination requests (default 3) | | `postUrl` | postDetail, comments | Full URL or permalink of the post | | `commentSort` | comments | `confidence` (Best), `top`, `new`, `controversial`, `qa`, `old` | | `maxComments` | comments | Max comments to return (default 100) | | `maxCommentPages` | comments | Max "load more comments" requests (default 5) | | `cookieHeader` | all, optional | Raw browser `Cookie:` header, improves reliability against Reddit's anti-bot checks | | `proxyConfiguration` | all, optional | Apify Proxy configuration | | `requestDelaySecs` | all | Delay between paginated requests, default 1.5s | Example `search` input: ```json { "mode": "search", "query": "minecraft", "maxItems": 25 } ``` ### Output Example dataset item for `mode: search`: ```json { "postId": "lyub6f", "title": "Playing Minecraft at 12 years old starter pack", "subreddit": "starterpacks", "author": "someuser", "url": "https://www.reddit.com/r/starterpacks/comments/lyub6f/playing_minecraft_at_12_years_old_starter_pack/", "score": 1234, "numComments": 56, "createdAt": "2021-03-01T12:00:00.000Z", "postType": "image", "thumbnailUrl": null } ``` Example dataset item for `mode: comments`: ```json { "commentId": "gpzk3h3", "parentId": "lyub6f", "author": "commenter1", "score": 10, "depth": 0, "createdAt": "2021-03-01T13:00:00.000Z", "permalink": "https://www.reddit.com/r/starterpacks/comments/lyub6f/x/gpzk3h3/", "bodyText": "Nice post!" } ``` You can download the dataset in various formats such as JSON, HTML, CSV, or Excel from the Apify Console, or fetch it through the Apify API. #### Data table | Field | Modes | Meaning | |---|---|---| | `postId` | search, postDetail | Reddit's base36 post ID | | `title` / `bodyText` | search, postDetail | Post title / self-text (or link description) | | `subreddit` / `author` | search, postDetail, comments | Subreddit name and post/comment author | | `score` / `numComments` | search, postDetail | Upvote score and comment count | | `commentId` / `parentId` / `depth` | comments | Comment ID, parent comment/post ID, nesting depth | ### Cost estimation This Actor makes plain HTTP requests (no browser rendering), so it runs cheap and fast. A `search` run for 25 posts is typically 1-2 requests; a `comments` run depends on thread size and `maxCommentPages`. Check the [Apify pricing page](https://apify.com/pricing) for your plan's compute unit costs, and use `maxItems` / `maxPages` / `maxComments` / `maxCommentPages` to cap usage. ### Tips - Set `requestDelaySecs` higher (3-5s) if you see failed runs - Reddit's anti-bot sensitivity varies over time and by IP. - Enable `proxyConfiguration` (Apify Proxy) if you're running this more than occasionally from the same IP. - Supply `cookieHeader` (copied from your browser's DevTools Network tab while logged into reddit.com) for the most reliable results. ### Environment variables - `REDDIT_COOKIE` (optional) - same purpose as the `cookieHeader` input field, useful if you want to set it once at the Actor level (Actor > Settings > Environment variables) instead of per run. ### Known limitations - **Parsing was not live-verified against Reddit** during development - the dev environment got IP-blocked by Reddit's anti-bot system mid-build (see `my_actor/parsers.py` docstring). Parsing targets Reddit's documented `shreddit-post` / `shreddit-comment` custom elements with a URL/meta-tag fallback, but if Reddit's markup has since changed, run one `apify run` per mode and check the dataset before relying on this in production. - Reddit's internal endpoints and challenge format can change without notice; this is not the official Data API and isn't guaranteed to keep working. - Comment pagination cursors and CSRF tokens are Reddit session-specific and are always re-extracted fresh per request - a stale `cookieHeader` will still work for basic requests but may reduce pagination depth. ### FAQ **Is this legal?** Reddit's ToS restricts automated scraping (see the notice above). This tool is intended for personal/research use of publicly visible content. Consult a lawyer for commercial use cases. **Why did my run fail with "Reddit blocked the request"?** Reddit's anti-bot system flagged the request. Try enabling `proxyConfiguration`, supplying a fresh `cookieHeader`, and/or increasing `requestDelaySecs`. Found a bug or have a feature request? Use the Actor's **Issues** tab to report it. # Actor input Schema ## `mode` (type: `string`): Pilih jenis data yang mau diambil. Field lain di bawah cuma relevan sesuai mode yang dipilih (lihat prefix mode di tiap description) - Apify input schema tidak mendukung show/hide otomatis, semua field tetap tampil. ## `query` (type: `string`): (mode: search) Kata kunci pencarian, sama seperti mengetik di kolom pencarian reddit.com. ## `maxItems` (type: `integer`): (mode: search) Jumlah maksimum post yang diambil. ## `maxPages` (type: `integer`): (mode: search) Batas jumlah request pagination (tiap halaman ambil beberapa post). Dihentikan lebih awal kalau maxItems sudah tercapai atau Reddit tidak lagi mengembalikan cursor berikutnya. ## `postUrl` (type: `string`): (mode: postDetail, comments) URL lengkap atau permalink post, misal https://www.reddit.com/r/starterpacks/comments/lyub6f/playing\_minecraft\_at\_12\_years\_old\_starter\_pack/ - bisa disalin dari field "url" hasil mode search. ## `commentSort` (type: `string`): (mode: comments) Urutan pengambilan komentar. ## `maxComments` (type: `integer`): (mode: comments) Jumlah maksimum komentar yang diambil (termasuk lewat pagination "load more"). ## `maxCommentPages` (type: `integer`): (mode: comments) Batas jumlah request pagination tambahan untuk memuat komentar lanjutan. ## `cookieHeader` (type: `string`): (semua mode, opsional) Isi header Cookie mentah hasil salin dari devtools browser (login ke reddit.com, buka tab Network, salin header 'cookie' dari salah satu request). Meningkatkan keberhasilan lolos dari verifikasi/anti-bot Reddit. Kalau kosong, coba diambil dari environment variable REDDIT\_COOKIE, lalu fallback ke request tanpa cookie. ## `proxyConfiguration` (type: `object`): (semua mode, opsional) Pakai Apify Proxy supaya request tidak selalu keluar dari IP yang sama. Reddit menerapkan anti-bot yang cukup ketat, proxy direkomendasikan untuk pemakaian lebih dari sesekali. ## `requestDelaySecs` (type: `number`): (semua mode) Jeda antar request pagination, untuk mengurangi risiko rate-limit/block dari Reddit. ## Actor input object example ```json { "mode": "search", "query": "minecraft", "maxItems": 25, "maxPages": 3, "commentSort": "confidence", "maxComments": 100, "maxCommentPages": 5, "proxyConfiguration": { "useApifyProxy": false }, "requestDelaySecs": 1.5 } ``` # Actor output Schema ## `dataset` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "query": "minecraft" }; // Run the Actor and wait for it to finish const run = await client.actor("ryyos/reddit-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "query": "minecraft" } # Run the Actor and wait for it to finish run = client.actor("ryyos/reddit-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "query": "minecraft" }' | apify call ryyos/reddit-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=ryyos/reddit-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Reddit Scraper (Search, Post Detail, Comments)", "description": "Search Reddit by keyword and scrape post listings, post detail, and comments via Reddit's shreddit web endpoints unlogin", "version": "0.1", "x-build-id": "tELRsrV1MZx9zsyPn" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/ryyos~reddit-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-ryyos-reddit-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/ryyos~reddit-scraper/runs": { "post": { "operationId": "runs-sync-ryyos-reddit-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/ryyos~reddit-scraper/run-sync": { "post": { "operationId": "run-sync-ryyos-reddit-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "mode" ], "properties": { "mode": { "title": "Mode", "enum": [ "search", "postDetail", "comments" ], "type": "string", "description": "Pilih jenis data yang mau diambil. Field lain di bawah cuma relevan sesuai mode yang dipilih (lihat prefix mode di tiap description) - Apify input schema tidak mendukung show/hide otomatis, semua field tetap tampil.", "default": "search" }, "query": { "title": "Search keyword", "type": "string", "description": "(mode: search) Kata kunci pencarian, sama seperti mengetik di kolom pencarian reddit.com." }, "maxItems": { "title": "Maksimum hasil", "minimum": 1, "type": "integer", "description": "(mode: search) Jumlah maksimum post yang diambil.", "default": 25 }, "maxPages": { "title": "Maksimum halaman pencarian", "minimum": 1, "type": "integer", "description": "(mode: search) Batas jumlah request pagination (tiap halaman ambil beberapa post). Dihentikan lebih awal kalau maxItems sudah tercapai atau Reddit tidak lagi mengembalikan cursor berikutnya.", "default": 3 }, "postUrl": { "title": "URL atau permalink post", "type": "string", "description": "(mode: postDetail, comments) URL lengkap atau permalink post, misal https://www.reddit.com/r/starterpacks/comments/lyub6f/playing_minecraft_at_12_years_old_starter_pack/ - bisa disalin dari field \"url\" hasil mode search." }, "commentSort": { "title": "Urutan komentar", "enum": [ "confidence", "top", "new", "controversial", "qa", "old" ], "type": "string", "description": "(mode: comments) Urutan pengambilan komentar.", "default": "confidence" }, "maxComments": { "title": "Maksimum komentar", "minimum": 1, "type": "integer", "description": "(mode: comments) Jumlah maksimum komentar yang diambil (termasuk lewat pagination \"load more\").", "default": 100 }, "maxCommentPages": { "title": "Maksimum halaman \"load more comments\"", "minimum": 1, "type": "integer", "description": "(mode: comments) Batas jumlah request pagination tambahan untuk memuat komentar lanjutan.", "default": 5 }, "cookieHeader": { "title": "Cookie header (opsional)", "type": "string", "description": "(semua mode, opsional) Isi header Cookie mentah hasil salin dari devtools browser (login ke reddit.com, buka tab Network, salin header 'cookie' dari salah satu request). Meningkatkan keberhasilan lolos dari verifikasi/anti-bot Reddit. Kalau kosong, coba diambil dari environment variable REDDIT_COOKIE, lalu fallback ke request tanpa cookie." }, "proxyConfiguration": { "title": "Proxy configuration", "type": "object", "description": "(semua mode, opsional) Pakai Apify Proxy supaya request tidak selalu keluar dari IP yang sama. Reddit menerapkan anti-bot yang cukup ketat, proxy direkomendasikan untuk pemakaian lebih dari sesekali.", "default": { "useApifyProxy": false } }, "requestDelaySecs": { "title": "Delay antar request (detik)", "minimum": 0, "type": "number", "description": "(semua mode) Jeda antar request pagination, untuk mengurangi risiko rate-limit/block dari Reddit.", "default": 1.5 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```