# Subreddit Scraper - Whole Subreddits, No 1k Post Cap ($1.5/1k) (`harshmaur/reddit-subreddit-scraper`) Actor Scrape entire subreddits — thousands of posts per community, far beyond Reddit's ~1,000-post listing cap, by combining every sort and time window. Optional comments per post. Archive communities or build ML datasets. No API key. CSV/Excel/JSON. From $1.50 per 1,000 posts. - **URL**: https://apify.com/harshmaur/reddit-subreddit-scraper.md - **Developed by:** [Harsh Maur](https://apify.com/harshmaur) (community) - **Categories:** AI, Lead generation, Social media - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 2 bookmarks - **User rating**: 5.00 out of 5 stars ## Pricing from $1.50 / 1,000 result saveds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Subreddit Scraper — Scrape an Entire Subreddit Past the 1,000-Post Limit (No API Key)

Download all posts from a subreddit — thousands per community, far beyond Reddit's ~1,000-post listing cap. Optional comments, no API key, no login. From $1.50 per 1,000 results.

Try on Apify Input Schema API Docs

What it does • How to use • 1,000-post limit • Input • Output • Pricing • FAQ

--- ### What does Subreddit Scraper do? {#what-does-subreddit-scraper-do} **Subreddit Scraper** is a purpose-built subreddit scraper that takes a community name — `AskReddit`, `r/AskReddit`, or a full URL — and pulls as much of that subreddit as possible into one deduplicated dataset. Where an ordinary scrape stops at the ~1,000 posts Reddit exposes per listing, this actor walks **every sort order across every time window plus per-flair listings**, merging the results so you keep going long after a single listing runs dry. - 🗂️ **Deep coverage** — combines new, top, and controversial sorts over hour/day/week/month/year/all-time windows, then adds flair-filtered listings on top - 💬 **Optional comments** — flip one switch to also pull each post's comment thread - 🔑 **Zero setup** — no Reddit account, no OAuth app, no API key; sidesteps Reddit's 600 requests/10 min API ceiling entirely - 📊 **70+ fields per post** — full text and HTML, media and galleries, flair, flags, and derived engagement metrics - 🔄 **Export and automate** — JSON, CSV, Excel, XML, HTML downloads; integrates with n8n, Zapier, Make, the REST API, and MCP - 💰 **Pay per result** — from $1.50 per 1,000 results, no monthly fee If you want keyword search, single posts, or user profiles, grab the all-in-one [Reddit Scraper](https://apify.com/harshmaur/reddit-scraper) instead — this actor does one thing deeply: whole communities. --- ### How to scrape an entire subreddit {#how-to-scrape-an-entire-subreddit} No code needed — a deep subreddit pull takes five steps: 1. [Sign up for Apify](https://apify.com/) for free (or log in) 2. Open [Subreddit Scraper](https://apify.com/harshmaur/reddit-subreddit-scraper) 3. Type the communities you want into **Subreddits** — any format works: `AskReddit`, `r/AskReddit`, or `https://www.reddit.com/r/AskReddit/` 4. Set **Max posts** (default 1,000 — raise it up to 50,000 to go deep) and optionally enable **Scrape comments for each post** 5. Press **Start**, then download the dataset as JSON, CSV, Excel, XML, or HTML > 💡 **Tip:** Start with a small `maxPostsCount` to verify the output shape, then rerun with a high cap for the full archive pull. --- ### How Subreddit Scraper bypasses Reddit's 1,000-post listing cap {#1000-post-limit} Reddit never serves more than roughly **1,000 posts per listing**. Page through r/technology/new as long as you like — the feed simply ends. The same wall applies to top, hot, and controversial, which is why most subreddit scrapers quietly top out around a thousand items no matter what limit you ask for. This actor's workaround is a **union of listings instead of one listing**: 1. **Every sort order** — it crawls `new`, `top`, and `controversial` separately 2. **Every time window** — `top` and `controversial` are fetched for hour, day, week, month, year, and all-time, and each window surfaces a different slice of history 3. **Every flair** — the subreddit's flair-filtered listings are crawled too, reaching posts that never appeared in any global sort 4. **Deduplication** — overlapping results are merged by post ID, so each post appears exactly once in your dataset Each listing is individually capped at ~1,000 posts, but their *union* is not — on active communities this routinely yields several times more than any single listing could. Honest caveat: it still cannot guarantee literally every post ever submitted; very old posts that fall outside all current listings stay out of reach for any scraper. --- ### Input example {#input-example} The full reference lives on the [Input Schema tab](https://apify.com/harshmaur/reddit-subreddit-scraper/input-schema). A typical deep-archive run looks like this: ```json { "subreddits": ["r/buildapc", "MachineLearning", "https://www.reddit.com/r/cscareerquestions/"], "maxPostsCount": 20000, "crawlCommentsPerPost": true, "maxCommentsPerPost": 25, "includeNSFW": false, "proxy": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] } } ```` | Field | Type | What it controls | | --- | --- | --- | | `subreddits` | array | One or more communities in any format — bare name, `r/name`, or full URL | | `maxPostsCount` | integer | Total posts to save across all subreddits (default 1,000, max 50,000) | | `crawlCommentsPerPost` | boolean | Also collect comments for every post found (default off — adds run time) | | `maxCommentsPerPost` | integer | Comment cap per post when comment crawling is on (default 10) | | `includeNSFW` | boolean | Keep posts marked 18+ in the results (default off) | | `proxy` | object | Proxy settings — Apify residential proxies recommended | *** ### Output example {#output-example} Every post arrives as a structured item with 70+ fields, downloadable as **JSON, CSV, Excel, XML, or HTML**. A trimmed example: ```json { "dataType": "post", "id": "t3_1xyz789", "parsedId": "1xyz789", "title": "Finally finished my first PC build — lessons learned", "body": "After three months of part-hunting, here is everything I wish I had known...", "bodyHtml": "

After three months of part-hunting...

", "postType": "text", "flair": "Build Complete", "authorName": "first_time_builder", "authorId": "t2_abc123", "communityName": "r/buildapc", "parsedCommunityName": "buildapc", "subredditName": "buildapc", "subredditId": "t5_2s3nh", "subredditSubscribers": 7200000, "score": 2841, "upVotes": 2841, "upvoteRatio": 0.96, "commentsCount": 312, "postUrl": "https://www.reddit.com/r/buildapc/comments/1xyz789/", "nsfw": false, "over18": false, "isSelf": true, "locked": false, "stickied": false, "archived": false, "totalAwardsReceived": 1, "domain": "self.buildapc", "mediaType": "text", "hasMedia": false, "isVideo": false, "isGallery": false, "ageHours": 52.4, "scorePerHour": 54.22, "commentsPerHour": 5.95, "engagementTotal": 3153, "isHighEngagement": true, "wordCount": 540, "createdAt": "2026-06-08T09:15:00.000Z", "crawledAt": "2026-06-10T13:40:00.000Z" } ``` Posts with media additionally populate `galleryImages`, `mediaAssets`, `videoUrl`, `thumbnail`, and the raw Reddit objects (`media`, `secureMedia`, `mediaMetadata`, `galleryData`). With comment crawling enabled, comment items (`dataType: "comment"`) carry the full thread context — `parentId`, `depth`, `isSubmitter`, `controversiality`, plus the parent post's title and URL. *** ### What can you do with a full subreddit dataset? - 🔬 **Community research** — study how a community's topics, norms, and vocabulary evolved across thousands of posts, not just last week's front page - 🤖 **ML and NLP datasets** — turn a niche subreddit into a labeled training corpus; flair and engagement fields come free as features - 📈 **Trend analysis** — chart `scorePerHour` and posting volume over time to spot when a topic took off inside a community - 🗄️ **Archiving** — archive a subreddit before it goes private, gets banned, or its mods wipe history; export to CSV and keep it forever - 🕵️ **Competitor community analysis** — pull a rival product's subreddit end to end and mine complaints, feature requests, and churn signals *** ### How much does it cost to scrape a subreddit? {#pricing} Subreddit Scraper is **pay-per-result** — no subscription, no platform fee: - **Actor start:** $0.02 per run - **Result stored:** $0.0015 each So a 1,000-post pull costs **$1.52**, a 10,000-post archive **$15.02**, and a 50,000-post maximum run **$75.02**. You pay only for items actually saved. > ⏱️ **Honest note on run time:** depth has a cost in requests. Because the actor walks many listings per subreddit (sorts × time windows × flairs) before deduplicating, a deep run takes noticeably longer than a quick scrape of one listing. Budget accordingly for large `maxPostsCount` values — especially with comments enabled. *** ### FAQ {#faq} #### How many posts can I get from one subreddit? It varies by community. Each Reddit listing maxes out near 1,000 posts, but because this actor merges new, top, and controversial across six time windows plus flair listings, active subreddits commonly yield **several thousand unique posts** — often 3-10× what a single-listing scrape returns. Quiet subreddits with few total posts will simply return everything they have. #### Does it get around the 1,000 post limit? Yes, in the only way that actually works: by unioning many capped listings rather than fighting the cap on one. You will get far more than 1,000 posts from any sizable subreddit. It does **not** guarantee every post ever made — content too old to appear in any current listing is unreachable for any scraper, including this one. #### Can I download a whole subreddit to CSV? Yes. After the run finishes, export the dataset straight from Apify as **CSV** (or Excel, JSON, XML, HTML). Each post becomes one row with 70+ columns — ready for spreadsheets, pandas, or BI tools. #### Do I need a Reddit API key or account? No. There is no OAuth flow, no login, and no Reddit account involved. The actor reads publicly available data directly, which also means Reddit's 600 requests/10 min API quota never applies to your runs. #### Can it scrape comments too? Yes — enable `crawlCommentsPerPost` and set `maxCommentsPerPost` to control depth. Be aware this multiplies the work: comments are fetched per post, so a 10,000-post run with comments takes substantially longer. For comment-first jobs, [Reddit Comments Scraper](https://apify.com/harshmaur/reddit-comments-scraper) is the better fit. #### Can I scrape several subreddits in one run? Yes. `subreddits` accepts a list, and `maxPostsCount` caps the total saved across all of them. Mix formats freely — `gaming`, `r/pcgaming`, and full URLs in the same array all work. #### Does it include NSFW posts? Only if you ask. 18+ posts are excluded by default; set `includeNSFW: true` to keep them in the dataset (they are flagged via the `nsfw` and `over18` fields either way). #### Is scraping a subreddit legal? Collecting publicly visible data is generally permissible, but you are responsible for using it lawfully: respect Reddit's terms, applicable privacy regulations, and the communities you study. Apify's guide on the [enforceability of terms of use](https://blog.apify.com/enforceability-of-terms-of-use/) is a good starting point. *** ### More Reddit scrapers | Actor | When to use it instead | | --- | --- | | [Reddit Scraper](https://apify.com/harshmaur/reddit-scraper) | The all-in-one: keyword search, direct URLs, users, and subreddits in a single actor | | [Reddit Comments Scraper](https://apify.com/harshmaur/reddit-comments-scraper) | You start from specific posts and want complete comment threads, nested replies included | | [Reddit Search Scraper](https://apify.com/harshmaur/reddit-search-scraper) | You're hunting a topic or brand across all of Reddit rather than inside one community | | [Reddit User Scraper](https://apify.com/harshmaur/reddit-user-scraper) | You need a redditor's profile, post history, and comment history | ***

Try Subreddit Scraper

Built by Harsh Maur • Questions? Contact support

# Actor input Schema ## `subreddits` (type: `array`): Enter one or more subreddits in any format: AskReddit, r/AskReddit, or https://www.reddit.com/r/AskReddit/. ## `maxPostsCount` (type: `integer`): Maximum number of posts to save across all subreddits. Raise this to pull entire communities. Max: 50000. ## `crawlCommentsPerPost` (type: `boolean`): When enabled, comments are also collected for every post found. This significantly increases run time and data output. ## `maxCommentsPerPost` (type: `integer`): Maximum comments to scrape per individual post (when 'Scrape comments for each post' is enabled). ## `includeNSFW` (type: `boolean`): Include posts marked as Not Safe For Work in results. ## `proxy` (type: `object`): Either use Apify proxy, or provide your own proxy servers. ## Actor input object example ```json { "subreddits": [ "r/AskReddit" ], "maxPostsCount": 1000, "crawlCommentsPerPost": false, "maxCommentsPerPost": 10, "includeNSFW": false, "proxy": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } } ``` # Actor output Schema ## `results` (type: `string`): All scraped Reddit data including posts, comments, communities, and user profiles. Use 'dataType' field to filter by type. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "subreddits": [ "r/AskReddit" ], "maxPostsCount": 1000, "maxCommentsPerPost": 10, "proxy": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }; // Run the Actor and wait for it to finish const run = await client.actor("harshmaur/reddit-subreddit-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "subreddits": ["r/AskReddit"], "maxPostsCount": 1000, "maxCommentsPerPost": 10, "proxy": { "useApifyProxy": True, "apifyProxyGroups": ["RESIDENTIAL"], }, } # Run the Actor and wait for it to finish run = client.actor("harshmaur/reddit-subreddit-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "subreddits": [ "r/AskReddit" ], "maxPostsCount": 1000, "maxCommentsPerPost": 10, "proxy": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } }' | apify call harshmaur/reddit-subreddit-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=harshmaur/reddit-subreddit-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Subreddit Scraper - Whole Subreddits, No 1k Post Cap ($1.5/1k)", "description": "Scrape entire subreddits — thousands of posts per community, far beyond Reddit's ~1,000-post listing cap, by combining every sort and time window. Optional comments per post. Archive communities or build ML datasets. No API key. CSV/Excel/JSON. From $1.50 per 1,000 posts.", "version": "0.0", "x-build-id": "hRTPa4xEKq5TeuaV3" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/harshmaur~reddit-subreddit-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-harshmaur-reddit-subreddit-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/harshmaur~reddit-subreddit-scraper/runs": { "post": { "operationId": "runs-sync-harshmaur-reddit-subreddit-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/harshmaur~reddit-subreddit-scraper/run-sync": { "post": { "operationId": "run-sync-harshmaur-reddit-subreddit-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "subreddits" ], "properties": { "subreddits": { "title": "Subreddits", "type": "array", "description": "Enter one or more subreddits in any format: AskReddit, r/AskReddit, or https://www.reddit.com/r/AskReddit/.", "default": [], "items": { "type": "string" } }, "maxPostsCount": { "title": "Max posts", "minimum": 1, "maximum": 50000, "type": "integer", "description": "Maximum number of posts to save across all subreddits. Raise this to pull entire communities. Max: 50000.", "default": 1000 }, "crawlCommentsPerPost": { "title": "Scrape comments for each post", "type": "boolean", "description": "When enabled, comments are also collected for every post found. This significantly increases run time and data output.", "default": false }, "maxCommentsPerPost": { "title": "Max comments per post", "minimum": 0, "type": "integer", "description": "Maximum comments to scrape per individual post (when 'Scrape comments for each post' is enabled).", "default": 10 }, "includeNSFW": { "title": "Include NSFW (18+) content", "type": "boolean", "description": "Include posts marked as Not Safe For Work in results.", "default": false }, "proxy": { "title": "Proxy configuration", "type": "object", "description": "Either use Apify proxy, or provide your own proxy servers.", "default": { "useApifyProxy": true, "apifyProxyGroups": [ "RESIDENTIAL" ] } } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```