# Reddit Subreddit Members Scraper (`scrapeflow/reddit-subreddit-members-scraper`) Actor - **URL**: https://apify.com/scrapeflow/reddit-subreddit-members-scraper.md - **Developed by:** [ScrapeFlow](https://apify.com/scrapeflow) (community) - **Categories:** Automation, Lead generation, Social media - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $3.99 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Reddit Subreddit Members Scraper — Async Apify Actor with Smart Proxy Fallback This actor collects Reddit user handles from subreddits, keywords, or direct user inputs, then optionally enriches each profile with public metadata. It is fully asynchronous (aiohttp), pushes results live to the Apify dataset, and ships with a multi-step proxy strategy: start direct, fallback to Apify datacenter if blocked, then escalate to residential and stay there. This README is long-form (≈1500 words) and SEO-optimized for queries like “Reddit scraper”, “Reddit members extractor”, “Apify actor Reddit”, “Reddit subreddit members”, “Reddit user scraper”, “Reddit proxy scraping”, and “async Python Reddit scraper”. ### Why This Actor - **Multi-input flexibility**: Accepts subreddit URLs (`https://www.reddit.com/r/python`), `r/`, user URLs (`https://www.reddit.com/user/spez`), `u/`, or free-text keywords to search Reddit posts. - **Post + comment coverage**: Gathers authors from both posts and comments with pagination, respecting `maxPosts`, `maxComments`, and `sort_order` (`new`, `hot`, `top`, `rising`). - **Optional enrichment**: When `fetchDetails=true`, calls `/user/{username}/about.json` to add karma, account creation time, gold status, and avatar. - **Proxy escalation**: Direct → Apify datacenter → Apify residential (sticky after switch). Logs every transition; adds extra retries on residential. - **Live data safety**: Pushes each found user immediately to the Apify dataset so partial results are preserved on crash or block. - **Async and rate-limited**: Uses aiohttp with semaphores and jittered sleeps to reduce throttling; rotates User-Agent per request. - **Function-based code**: No classes, easier to maintain and extend. - **Production-ready defaults**: Conservative timeouts, retry logic, and structured dataset view. ### What It Scrapes - **Usernames** from subreddit posts and comments. - **Profile URL** for each discovered user. - **Optional user details** (when enabled): total karma, post karma, comment karma, created UTC, gold flag, avatar URL. - **Source coverage**: - Subreddit posts (sorted by `sort_order`) - Subreddit comments - Reddit search results for provided keywords (post authors) - Direct user inputs (usernames/URLs) ### Proxy and Anti-Blocking Strategy 1. **Start direct (no proxy)** — fastest and cheapest. 2. **If blocked (403/429/503/timeout)** → switch to **Apify datacenter** proxy. 3. **If still blocked** → switch to **Apify residential** proxy, **stick to residential** for the rest of the run, and allow 3 extra retries there. 4. **Logged events** — every escalation is written to Apify logs with reason and target URL. ### Input Parameters (actor.json) - `targets` (array, required): Mixed inputs. Supports: - Subreddit URLs (`https://www.reddit.com/r/python`) - `r/` - User URLs (`https://www.reddit.com/user/spez`) or `u/` - Free-text keywords (treated as Reddit search queries) - `subreddits` (array, optional): Plain subreddit names if you prefer not to use `targets`. - `sort_order` (string, enum): `new` | `hot` | `top` | `rising` (defaults to `new`). - `maxPosts` (integer): 1–1000, default 100. Posts per subreddit or per keyword search. - `maxComments` (integer): 0–1000, default 100. Comments per subreddit. - `fetchDetails` (boolean): If true, enrich each user via `/user/{username}/about.json`. - `maxConcurrentUsers` (integer): 1–20, default 3. Concurrency for user detail fetches. - `requestDelay` (integer): 0–10 seconds, default 1. Added between detail calls; jittered by +0–0.5s. - `proxyConfiguration` (object): Apify proxy editor. Default `useApifyProxy=false` (direct). If Reddit blocks, actor escalates automatically to datacenter then residential. #### Example Input (balanced) ```json { "targets": [ { "url": "https://www.reddit.com/r/python" }, { "url": "r/webscraping" }, { "url": "asyncio" }, { "url": "u/spez" } ], "sort_order": "new", "maxPosts": 50, "maxComments": 50, "fetchDetails": true, "maxConcurrentUsers": 3, "requestDelay": 1, "proxyConfiguration": { "useApifyProxy": false } } ```` ### Output Schema (dataset) Each pushed item (basic): ```json { "username": "example_user", "userId": "t2_1700000000000", "profileUrl": "https://reddit.com/user/example_user" } ``` When `fetchDetails=true`, additional fields appear: ```json { "username": "example_user", "userId": "t2_1700000000000", "profileUrl": "https://reddit.com/user/example_user", "totalKarma": 1234, "postKarma": 900, "commentKarma": 334, "createdUTC": 1600000000, "isGold": false, "iconImg": "https://styles.redditmedia.com/..." } ``` ### How It Works (Architecture) - **Async requests**: aiohttp with rotating User-Agent and per-request proxy resolution. - **Proxy state machine**: Direct → Datacenter → Residential; sticky on residential with extra retries. - **Post and comment pagination**: Uses `after` cursors up to `maxPosts` / `maxComments`. - **Keyword search**: Queries `/search.json` with `type=link`, collects authors from posts. - **Live persistence**: Each user is `Actor.push_data(...)` immediately; no batching required. - **Concurrency control**: Semaphore on user-detail fetches to respect `maxConcurrentUsers`. - **Jittered delays**: Small random sleep after a few requests and between user detail calls to reduce 429s. - **Resilience**: Escalates proxies on 403/429/503/timeouts, logs failures, continues collecting partial data. ### Anti-Blocking Tips - Keep `fetchDetails=false` if you only need usernames; this reduces calls and blocks. - Lower `maxPosts`/`maxComments` for aggressive subreddits. - Increase `requestDelay` to 2–3s when fetching details at scale. - Start with direct; allow actor to escalate naturally. If you expect heavy blocking, set `useApifyProxy=true` to begin on datacenter. - Prefer fewer concurrent detail requests (`maxConcurrentUsers` 2–4) if seeing rate limits. ### Setup and Local Run Requirements: Python 3.10+, Apify CLI, Docker (for full build). ```bash cd Reddit-Subreddit-Members-Scraper pip install -r requirements.txt apify run ``` Supply input via `INPUT.json` or Apify UI. Results appear in the default dataset. ### Deploy on Apify 1. Push: `apify push` 2. Run in the Apify Console with your input. 3. Monitor logs for proxy transitions and counts. 4. Download results from the Dataset tab (JSON/CSV/Excel). ### Field-by-Field Input Guide - **targets**: Best entrypoint; mix subreddits, users, and keywords. Use request-list editor in Apify UI. - **subreddits**: Convenience fallback; plain names. - **sort\_order**: Choose `new` for freshness, `hot` for trending, `top` for high-signal authors, `rising` for early discoveries. - **maxPosts / maxComments**: Balance coverage vs. speed; Reddit caps per page at 100. - **fetchDetails**: Enable only when you need karma/created/avatar; otherwise stay off for speed. - **maxConcurrentUsers**: Tune to reduce 429s; 3–5 is usually safe. - **requestDelay**: Increase if blocked while enriching. - **proxyConfiguration**: Leave off by default. The actor will escalate automatically when blocked. ### Data Quality Notes - Deleted or suspended authors are skipped (`[deleted]`). - Some profiles may block or return minimal data; enrichment may be partial. - `userId` is a generated placeholder (Reddit API hides the true ID through these endpoints). ### Performance Tips - Use smaller `maxPosts` and `maxComments` across more subreddits to diversify results. - For large runs with details, consider running during off-peak hours and upping delay to 2–3s. - Keywords can be noisy; combine with subreddits for higher relevance. ### Error Handling and Logging - **HTTP 403/429/503/Timeout** → proxy escalation with log entry. - **Other HTTP** → warning + limited retries. - **Exceptions** → logged with stack trace; run continues where possible. - **Completion** → summary log to check dataset. ### Security and Compliance - Scrapes only public Reddit endpoints. - Respect Reddit’s terms and local laws; do not spam users with scraped data. - Proxies are used solely for block avoidance; residential escalation is logged. ### Extending the Actor - Add score or upvote thresholds: adjust post fetch URL parameters. - Add `top` time windows: append `t=day/week/month/year/all` on top sorting. - Add subreddit filters: pre-validate allowed subs list. - Add CSV export: post-process dataset with Apify transformations or client-side script. ### Troubleshooting FAQ **Q: I get empty results.**\ A: Check inputs; ensure subreddits exist. Lower `maxPosts`/`maxComments`, set `sort_order=new`, and allow proxy escalation. **Q: Blocks persist even on residential.**\ A: Increase `requestDelay`, reduce `maxConcurrentUsers`, and lower volume. Consider splitting runs by subreddit batches. **Q: Enrichment is slow.**\ A: Disable `fetchDetails` or reduce `maxConcurrentUsers`. Details require per-user calls. **Q: Dataset fields missing.**\ A: Fields only appear when `fetchDetails=true`. Basic mode pushes username, userId, profileUrl only. ### SEO Snapshot (keywords covered) - Reddit scraper, Reddit members scraper, Reddit user extractor, Reddit subreddit members, Reddit profile scraper, Apify Reddit scraper, Python aiohttp Reddit scraper, Reddit proxy scraping, Reddit residential proxy, Reddit datacenter proxy, async Reddit scraper, Reddit dataset export. ### Quick Start (TL;DR) 1. Provide `targets` with subreddits/users/keywords. 2. Set `maxPosts`, `maxComments`, `sort_order`. 3. Decide on `fetchDetails`. 4. Leave proxy off; actor will escalate if blocked. 5. Run and grab results from the dataset. ### Changelog - v0.1: Initial public actor with async fetch, post/comment collection, keyword search, optional user details, and direct→datacenter→residential proxy fallback. *** Built to stay resilient, transparent, and efficient for Reddit member discovery on Apify. Run it, watch the logs for proxy events, and export the dataset when done. # Actor input Schema ## `targets` (type: `array`): 📥 Bulk input supported! Add multiple items like subreddit URLs, r/, u/, or free-text keywords. ## `subreddits` (type: `array`): ➕ Optional: add plain subreddit names (e.g., python, webscraping) if you are not using targets. ## `maxPosts` (type: `integer`): 📌 Maximum number of posts to fetch per subreddit or keyword (1-1000 recommended). ## `maxComments` (type: `integer`): 🧵 Maximum comments to fetch per subreddit (0-1000). ## `fetchDetails` (type: `boolean`): ✨ Fetch /user/{username}/about.json for each discovered user to enrich output with karma and profile details. ## `proxyConfiguration` (type: `object`): 🛡️ By default, no proxy is used. If Reddit blocks requests, the actor can fallback to datacenter, then residential proxies and stay on residential mode. ## Actor input object example ```json { "targets": [ "🔥 webscraping", "🐍 r/python", "👤 u/spez" ], "subreddits": [ "python", "webscraping", "learnprogramming" ], "maxPosts": 10, "maxComments": 10, "fetchDetails": true, "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } } ``` # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "targets": [ "🔥 webscraping", "🐍 r/python", "👤 u/spez" ], "subreddits": [ "python", "webscraping", "learnprogramming" ], "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }; // Run the Actor and wait for it to finish const run = await client.actor("scrapeflow/reddit-subreddit-members-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "targets": [ "🔥 webscraping", "🐍 r/python", "👤 u/spez", ], "subreddits": [ "python", "webscraping", "learnprogramming", ], "proxyConfiguration": { "useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], }, } # Run the Actor and wait for it to finish run = client.actor("scrapeflow/reddit-subreddit-members-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "targets": [ "🔥 webscraping", "🐍 r/python", "👤 u/spez" ], "subreddits": [ "python", "webscraping", "learnprogramming" ], "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }' | apify call scrapeflow/reddit-subreddit-members-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=scrapeflow/reddit-subreddit-members-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Reddit Subreddit Members Scraper", "version": "0.1", "x-build-id": "H2mTdv84zKrolzit3" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/scrapeflow~reddit-subreddit-members-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-scrapeflow-reddit-subreddit-members-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/scrapeflow~reddit-subreddit-members-scraper/runs": { "post": { "operationId": "runs-sync-scrapeflow-reddit-subreddit-members-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/scrapeflow~reddit-subreddit-members-scraper/run-sync": { "post": { "operationId": "run-sync-scrapeflow-reddit-subreddit-members-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "targets" ], "properties": { "targets": { "title": "🔎 URLs / Usernames / Keywords", "type": "array", "description": "📥 Bulk input supported! Add multiple items like subreddit URLs, r/, u/, or free-text keywords.", "items": { "type": "string" } }, "subreddits": { "title": "📚 Additional Subreddit Names", "type": "array", "description": "➕ Optional: add plain subreddit names (e.g., python, webscraping) if you are not using targets.", "items": { "type": "string" } }, "maxPosts": { "title": "📝 Max Posts", "minimum": 1, "maximum": 10000, "type": "integer", "description": "📌 Maximum number of posts to fetch per subreddit or keyword (1-1000 recommended).", "default": 10 }, "maxComments": { "title": "💬 Max Comments", "minimum": 0, "maximum": 1000, "type": "integer", "description": "🧵 Maximum comments to fetch per subreddit (0-1000).", "default": 10 }, "fetchDetails": { "title": "👤 Fetch User Details", "type": "boolean", "description": "✨ Fetch /user/{username}/about.json for each discovered user to enrich output with karma and profile details.", "default": true }, "proxyConfiguration": { "title": "🌐 Proxy Configuration", "type": "object", "description": "🛡️ By default, no proxy is used. If Reddit blocks requests, the actor can fallback to datacenter, then residential proxies and stay on residential mode." } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```