# Reddit Public Post & Comment Scraper (`woundless_yellowwood/reddit-public-post-comment-scraper`) Actor Point it at any public subreddit and get structured posts and comments back — no Reddit API key, no login, no rate-limit headaches. - **URL**: https://apify.com/woundless\_yellowwood/reddit-public-post-comment-scraper.md - **Developed by:** [Proyecto Apify](https://apify.com/woundless_yellowwood) (community) - **Categories:** Social media, Developer tools, Open source - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage ## What's an Apify Actor? Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js/docs.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python/docs.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Reddit Public Post & Comment Scraper Point it at any public subreddit and get structured posts and comments back - no Reddit API key, no login, no rate-limit headaches. ### What it does - Fetches posts from any public subreddit via Reddit's `.json` endpoints (no OAuth, no API key) - Optionally extracts top-level and nested comments for each post - Deduplicates posts by Reddit post ID across multiple runs of the same subreddit (persisted in the Actor's Key-Value Store) - Skips private, banned, and quarantined subreddits with a clear log message - never crashes - Respects Reddit's public rate limits with built-in throttling (single concurrent request + random delay) and automatic exponential backoff on 429 responses - Supports two output shapes: `nested` (one row per post with a comments array) or `flat` (one row per post plus one row per comment - easier for CSV export) ### Input | Field | Type | Default | Description | |-------|------|---------|-------------| | `startUrls` | array\ | (required) | Subreddit names (`javascript`, `r/javascript`) or full Reddit post URLs. Mixed input is allowed. | | `sortMode` | enum | `hot` | `hot` \| `new` \| `top` \| `rising`. Ignored for direct post URLs. | | `maxPosts` | integer | 100 | Max posts per subreddit (1-1000). | | `includeComments` | boolean | true | Fetch comments for each post. | | `maxCommentsPerPost` | integer | 50 | Max comments extracted per post (0-500, depth-first). | | `timeRange` | enum | `all` | Used only with `sortMode: top` (`hour`/`day`/`week`/`month`/`year`/`all`). | | `outputMode` | enum | `nested` | `nested` (one row per post with comments array) or `flat` (one row per post + one row per comment). | | `dedupeAcrossRuns` | boolean | true | Skip posts already seen in previous runs. | | `minRequestDelayMs` | integer | 800 | Minimum delay between requests (with random jitter). Increase if you see 429s. | | `useApifyProxy` | boolean | true | Route requests through Apify's residential proxy pool. Required - Reddit blocks data-center IPs from `.json` endpoints. | | `proxyGroups` | array\ | `[]` | Optional Apify proxy groups (e.g. `["RESIDENTIAL"]`). | | `proxyCountryCode` | string | `""` | Optional ISO country code (e.g. `US`, `GB`). | ### Output Each dataset row contains: **Post fields**: `id`, `title`, `author`, `score`, `upvoteRatio`, `numComments`, `createdUtc`, `subreddit`, `flair`, `permalink`, `url`, `selftext`, `postHint`, `gilded`, `totalAwards`, `isStickied`, `isOver18`, `isLocked`, and more. **Comment fields** (in `comments` array for nested mode, or as separate rows in flat mode): `id`, `author`, `body`, `score`, `createdUtc`, `depth`, `parentId`, `isStickied`, `isOp`, `permalink`, `gilded`. ### Example Input: ```json { "startUrls": ["javascript", "https://www.reddit.com/r/programming"], "sortMode": "top", "timeRange": "week", "maxPosts": 50, "includeComments": true, "maxCommentsPerPost": 20, "outputMode": "nested" } ```` Output (nested mode, one row): ```json { "type": "post", "id": "1abc23", "title": "Example post title", "author": "some_user", "score": 1234, "upvoteRatio": 0.95, "numComments": 87, "createdUtc": "2025-01-15T12:34:56.000Z", "subreddit": "javascript", "permalink": "https://www.reddit.com/r/javascript/comments/1abc23/...", "url": "https://example.com/article", "selftext": "", "comments": [ { "id": "def45", "author": "another_user", "body": "Great post!", "score": 42, "depth": 0, "parentId": null } ] } ``` ### Scheduling This Actor is safe to run on a recurring schedule (daily, weekly). With `dedupeAcrossRuns: true`, each run only emits posts not seen in previous runs - ideal for monitoring subreddits over time. Reset dedup by toggling `dedupeAcrossRuns: false` or by clearing the Actor's `reddit-dedup` Key-Value Store. ### Privacy & compliance - Only publicly available Reddit data is accessed, via Reddit's unauthenticated `.json` endpoints. - No Reddit OAuth, no API key, no login - no PII beyond what Reddit already displays publicly (usernames, public post/comment text). - Private, banned, and quarantined subreddits are **skipped**, not bypassed. - This Actor does not bypass Reddit's rate limits or ToS. It uses built-in throttling and backoff to stay within Reddit's public rate limits. ### Proxy requirements (important) Reddit actively blocks datacenter IP ranges (including Apify's default proxy pool) from `.json` endpoints. To run reliably, this Actor requires **Apify's residential proxy group** (`RESIDENTIAL`), available on Apify's **Starter plan or higher**. The Free plan does not include residential proxy access. If you see repeated HTTP 403 errors in the logs: 1. Ensure your Apify account has the residential proxy group enabled. 2. Pass `proxyGroups: "RESIDENTIAL"` in the input. 3. If you have your own external proxy that Reddit doesn't block, set `useApifyProxy: false` and configure your proxy via the `proxyGroups` field. The Actor's logic (parsing, deduplication, comment extraction, rate limiting) is fully functional - the only failure mode is Reddit blocking the proxy IP range, which is outside the Actor's control. # Actor input Schema ## `startUrls` (type: `string`): One entry per line. Each entry can be a subreddit name (e.g. 'javascript' or 'r/javascript') OR a full Reddit post URL (e.g. 'https://www.reddit.com/r/javascript/comments/abc/...'). Mixed input is allowed. ## `sortMode` (type: `string`): Sort order for posts within each subreddit. Ignored for direct post URLs. ## `maxPosts` (type: `integer`): Maximum number of posts to collect per subreddit. Pagination stops after this count is reached. ## `includeComments` (type: `boolean`): If true, fetches comments for each post (slower but more complete). If false, only posts are returned. ## `maxCommentsPerPost` (type: `integer`): Maximum number of comments to extract per post (top-level + nested, depth-first). Set to 0 to skip comments even if includeComments is true. ## `timeRange` (type: `string`): Used only when sortMode is 'top'. Ignored otherwise. ## `outputMode` (type: `string`): nested = one dataset row per post with comments array; flat = one row per post AND one row per comment (easier for CSV export). ## `dedupeAcrossRuns` (type: `boolean`): If true, post IDs seen in previous runs (stored in this Actor's Key-Value Store) are skipped. Set to false to force re-scrape. ## `minRequestDelayMs` (type: `integer`): Minimum delay between Reddit requests, in milliseconds. Combined with a random jitter to respect Reddit's public rate limits. Increase if you see 429 errors. ## `useApifyProxy` (type: `boolean`): Required for reliable operation. Reddit blocks data-center IPs from .json endpoints; routing through Apify's residential proxy pool bypasses the block. Disable only if you provide your own proxy via proxyGroups. ## `proxyGroups` (type: `string`): Optional Apify proxy groups, one per line (e.g. 'RESIDENTIAL'). Leave empty for default residential pool. ## `proxyCountryCode` (type: `string`): Optional ISO country code for the proxy (e.g. 'US', 'GB'). Leave empty for any country. ## Actor input object example ```json { "startUrls": "javascript\nprogramming", "sortMode": "hot", "maxPosts": 100, "includeComments": true, "maxCommentsPerPost": 50, "timeRange": "all", "outputMode": "nested", "dedupeAcrossRuns": true, "minRequestDelayMs": 800, "useApifyProxy": true, "proxyGroups": "", "proxyCountryCode": "" } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = {}; // Run the Actor and wait for it to finish const run = await client.actor("woundless_yellowwood/reddit-public-post-comment-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = {} # Run the Actor and wait for it to finish run = client.actor("woundless_yellowwood/reddit-public-post-comment-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{}' | apify call woundless_yellowwood/reddit-public-post-comment-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=woundless_yellowwood/reddit-public-post-comment-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Reddit Public Post & Comment Scraper", "description": "Point it at any public subreddit and get structured posts and comments back — no Reddit API key, no login, no rate-limit headaches.", "version": "0.0", "x-build-id": "vIyEaZO391NC4ePTY" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/woundless_yellowwood~reddit-public-post-comment-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-woundless_yellowwood-reddit-public-post-comment-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/woundless_yellowwood~reddit-public-post-comment-scraper/runs": { "post": { "operationId": "runs-sync-woundless_yellowwood-reddit-public-post-comment-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/woundless_yellowwood~reddit-public-post-comment-scraper/run-sync": { "post": { "operationId": "run-sync-woundless_yellowwood-reddit-public-post-comment-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "startUrls" ], "properties": { "startUrls": { "title": "Subreddits or post URLs (one per line)", "type": "string", "description": "One entry per line. Each entry can be a subreddit name (e.g. 'javascript' or 'r/javascript') OR a full Reddit post URL (e.g. 'https://www.reddit.com/r/javascript/comments/abc/...'). Mixed input is allowed.", "default": "javascript\nprogramming" }, "sortMode": { "title": "Sort mode", "enum": [ "hot", "new", "top", "rising" ], "type": "string", "description": "Sort order for posts within each subreddit. Ignored for direct post URLs.", "default": "hot" }, "maxPosts": { "title": "Maximum posts per subreddit", "minimum": 1, "maximum": 1000, "type": "integer", "description": "Maximum number of posts to collect per subreddit. Pagination stops after this count is reached.", "default": 100 }, "includeComments": { "title": "Include comments", "type": "boolean", "description": "If true, fetches comments for each post (slower but more complete). If false, only posts are returned.", "default": true }, "maxCommentsPerPost": { "title": "Max comments per post", "minimum": 0, "maximum": 500, "type": "integer", "description": "Maximum number of comments to extract per post (top-level + nested, depth-first). Set to 0 to skip comments even if includeComments is true.", "default": 50 }, "timeRange": { "title": "Time range (top sort only)", "enum": [ "hour", "day", "week", "month", "year", "all" ], "type": "string", "description": "Used only when sortMode is 'top'. Ignored otherwise.", "default": "all" }, "outputMode": { "title": "Output mode", "enum": [ "nested", "flat" ], "type": "string", "description": "nested = one dataset row per post with comments array; flat = one row per post AND one row per comment (easier for CSV export).", "default": "nested" }, "dedupeAcrossRuns": { "title": "Deduplicate across runs", "type": "boolean", "description": "If true, post IDs seen in previous runs (stored in this Actor's Key-Value Store) are skipped. Set to false to force re-scrape.", "default": true }, "minRequestDelayMs": { "title": "Min request delay (ms)", "minimum": 200, "maximum": 10000, "type": "integer", "description": "Minimum delay between Reddit requests, in milliseconds. Combined with a random jitter to respect Reddit's public rate limits. Increase if you see 429 errors.", "default": 800 }, "useApifyProxy": { "title": "Use Apify residential proxy", "type": "boolean", "description": "Required for reliable operation. Reddit blocks data-center IPs from .json endpoints; routing through Apify's residential proxy pool bypasses the block. Disable only if you provide your own proxy via proxyGroups.", "default": true }, "proxyGroups": { "title": "Proxy groups (one per line)", "type": "string", "description": "Optional Apify proxy groups, one per line (e.g. 'RESIDENTIAL'). Leave empty for default residential pool.", "default": "" }, "proxyCountryCode": { "title": "Proxy country code", "type": "string", "description": "Optional ISO country code for the proxy (e.g. 'US', 'GB'). Leave empty for any country.", "default": "" } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```