# Reddit Scraper (`rust_knot/reddit-scraper`) Actor

Scrape Reddit posts from subreddit feeds and search results, with pagination, rate-limit handling and optional proxy support.

- **URL**: https://apify.com/rust\_knot/reddit-scraper.md
- **Developed by:** [Muhammad Luay](https://apify.com/rust_knot) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $20.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

### Reddit Scraper

Scrape **Reddit posts** from subreddit feeds and search results — titles, authors, scores, comment counts, URLs, flair, and full post bodies — without an API key. The Actor runs on [Apify](https://apify.com), giving you **API access, scheduling, monitoring, and proxy rotation** out of the box. Just set a subreddit or a search query and get a clean, structured dataset of posts.

This Actor uses the server-rendered HTML of old.reddit.com, so it works even where Reddit's JSON API endpoints are blocked for your IP.

### Why use this Actor?

- **No API keys or OAuth setup** — Reddit's public JSON endpoints are frequently blocked, but this Actor reads the HTML directly.
- **Feed or search** — scrape `hot`, `new`, `top`, `rising` from any subreddit, or run keyword searches with time filters.
- **Full post data** — score, comment count, author, permalink, external URL, domain, flair, NSFW/spoiler flags, and optional full selftext bodies.
- **Built to be cheap** — plain HTTP requests, no browser needed; paginates efficiently and stops when it hits your `maxItems` limit.
- **Apify platform advantages** — schedule it to run on a cron, call it from the API, and download results as JSON, CSV, or Excel.

### How to use this Actor

1. Open the Actor in [Apify Console](https://console.apify.com), select **Input** in the left sidebar.
2. Choose a **Mode**: `Subreddit feed` (posts from a subreddit) or `Search` (keyword search).
3. Enter a **Subreddit** (e.g. `python`, no `r/` prefix), a **Search query**, and optionally adjust **Sort order**, **Time filter**, and **Maximum items**.
4. Click **Start**. The run will show a live log; results land in the **Dataset** tab.
5. Download the dataset in JSON, HTML, CSV, or Excel, or fetch it via the Apify API.

#### Reddit OAuth credentials (recommended)

The Actor uses Reddit's **official JSON API** whenever credentials are available — faster, more structured, and immune to the login-gating Reddit now applies to `old.reddit.com`. Without credentials it falls back to scraping the legacy HTML UI, which can be login-blocked.

Credentials are read as `REDDIT_CLIENT_ID_n` / `REDDIT_KEY_n` pairs (n = 1, 2, ...). Sources, in order:

1. **Environment variables** (`REDDIT_CLIENT_ID_1`, `REDDIT_KEY_1`, ...) — set them in Apify Console under the Actor's **Environment variables**, or locally in the shell.
2. **Secrets file** (local dev only) — by default `../secrets/reddit_keys.env` next to the repo, overridable with the `REDDIT_KEYS_FILE` variable. Never commit this file.

Pairs are tried in order with a cascade of auth constructions (`client_credentials` both ways, `refresh_token`); a pair that returns 401 is skipped and the next is tried, so an expired key in the pool does not break the run. Cloud deployments may use `apify secrets add` to store a single pair and reference it via environment variables.

### Input

| Field | Type | Description |
| --- | --- | --- |
| mode | enum | `subreddit` (feed) or `search` (keyword search) |
| subreddit | string | Subreddit name without the `r/` prefix. Empty in search mode = search all of Reddit |
| query | string | Search term (search mode only) |
| sort | enum | Feed: `hot`, `new`, `top`, `rising`. Search: `relevance`, `hot`, `new`, `top`, `comments` |
| timeFilter | enum | `hour`, `day`, `week`, `month`, `year`, `all` (used by `top` and search) |
| maxItems | integer | Stop after scraping this many posts (default 1000) |
| includeSelfText | boolean | Include post bodies. Search results always include them; feed mode fetches one extra page per self-post |
| proxyConfiguration | object | Optional Apify proxy routing for IPs blocked by Reddit |

### Output

One dataset record per post:

```json
{
  "type": "post",
  "id": "1ncuv8k",
  "title": "Best web scraping tools I've tried",
  "author": "DenOmania",
  "subreddit": "automation",
  "permalink": "https://old.reddit.com/r/automation/comments/1ncuv8k/...",
  "url": "https://example.com/article",
  "domain": "example.com",
  "score": 155,
  "numComments": 188,
  "createdUtc": "1785859525000",
  "createdAt": "2025-09-09T20:55:19+00:00",
  "flair": "Discussion",
  "spoiler": false,
  "nsfw": false,
  "stickied": false,
  "selftext": "I've gone through quite a few tools..."
}
```

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel.

### Data table

| Field | Type | Description |
| --- | --- | --- |
| id | string | Post ID (`t3_` suffix) |
| title | string | Post title |
| author | string | Reddit username of the author |
| subreddit | string | Subreddit the post was submitted to |
| permalink | string | URL of the post on Reddit |
| url | string | External URL (link posts) |
| domain | string | Domain of the external URL |
| score | number | Upvote score at scrape time |
| numComments | number | Comment count at scrape time |
| createdAt | string | ISO 8601 creation timestamp |
| flair | string | Link flair text |
| spoiler / nsfw / stickied | boolean | Content and listing flags |
| selftext | string | Full post body (when enabled / available) |

### Pricing / Cost estimation

How much does it cost to scrape Reddit with this Actor? Runs are **cheap**: one listing page returns ~25 posts, so 1,000 posts costs roughly 40–45 page requests plus a few KB of compute. On the Apify **free tier** you get 5 USD of monthly credits — a 1,000-post scrape typically costs well under a cent of compute. Use `maxItems` to keep large runs predictable. Reddit rate-limits aggressive requests, so the Actor spaces requests out automatically (exponential backoff on `429`/`403`); for heavy or blocked runs, enable the **Proxy configuration** input.

### Tips / Advanced options

- **Search all of Reddit**: leave `subreddit` empty in search mode.
- **Watch for rate limits**: if runs hit repeated `429`, enable the Apify proxy in the Input tab — it rotates IPs.
- **Limit cost**: set a `maxItems` cap; the Actor stops paginating as soon as it's reached.
- **Scheduling**: use Apify Scheduler to re-scrape a subreddit daily and diff new posts.
- **Selftext in feeds**: body fetching adds one request per self-post; disable `includeSelfText` for large feed crawls.

### FAQ / Disclaimers

- **Is scraping Reddit allowed?** Respect Reddit's [Terms of Service](https://www.redditinc.com/policies/user-agreement) and Reddit's rules. The Actor reads public data at a polite rate and honors rate-limit signals; use it responsibly and don't republish personal data without permission.
- **Why old.reddit.com?** Reddit blocks the `.json` API endpoints from many IPs. The old layout serves the same public data as server-rendered HTML, which is far more reliably accessible.
- **Known limitations**: live-streaming (continuously following `/new`) and user-profile scraping are not supported yet; comment scraping is planned.
- Issues or feature requests? Open one on the [Issues](https://apify.com/bearwallow/reddit-scraper/issues) tab. For custom scrapers and automation, contact us via the Apify account.

# Actor input Schema

## `mode` (type: `string`):

What to scrape: posts from a subreddit feed, or subreddit/site-wide search results.

## `subreddit` (type: `string`):

Name of the subreddit without the r/ prefix. Leave empty in search mode to search all of Reddit.

## `query` (type: `string`):

Search term. Used only in search mode.

## `sort` (type: `string`):

Feed mode: hot, new, top or rising. Search mode: relevance, hot, new, top or comments.

## `timeFilter` (type: `string`):

Time window used by 'top' sorting and by search. Ignored by other sorts.

## `maxItems` (type: `integer`):

Stop after scraping this many posts (0 = no limit).

## `includeSelfText` (type: `boolean`):

Include the selftext body of text posts. Search results always include bodies; in feed mode this fetches one extra page per self-post.

## `rpsLimit` (type: `number`):

Maximum request rate against Reddit. Keep at 1.0-2.0 to stay polite and avoid the bot filter; higher values increase block risk.

## `useCacheFallback` (type: `boolean`):

When a subreddit fetch is fully blocked, serve the last successful records (up to 100, cached from previous runs) instead of returning nothing.

## `proxyConfiguration` (type: `object`):

Optionally route requests through the Apify proxy to avoid Reddit rate limiting / IP blocks.

## `proxyUrls` (type: `string`):

Comma-separated forward-proxy URLs (user:pass@host:port form) to use as egress for RSS/HTML requests. Highly recommended: Reddit's network-security filter blocks datacenter and flagged IPs, but the RSS feed works from clean residential IPs after a session warm-up.

## Actor input object example

```json
{
  "mode": "subreddit",
  "subreddit": "python",
  "query": "web scraping",
  "sort": "hot",
  "timeFilter": "all",
  "maxItems": 1000,
  "includeSelfText": true,
  "rpsLimit": 1,
  "useCacheFallback": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "proxyUrls": "http://user:pass@proxy1.example.com:8888,http://user:pass@proxy2.example.com:8888"
}
```

# Actor output Schema

## `posts` (type: `string`):

Dataset with scraped Reddit posts (one record per post).

## `summary` (type: `string`):

Key-value store entry summarizing the run.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "subreddit": "python",
    "query": "web scraping",
    "proxyConfiguration": {
        "useApifyProxy": false
    },
    "proxyUrls": "http://user:pass@proxy1.example.com:8888,http://user:pass@proxy2.example.com:8888"
};

// Run the Actor and wait for it to finish
const run = await client.actor("rust_knot/reddit-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "subreddit": "python",
    "query": "web scraping",
    "proxyConfiguration": { "useApifyProxy": False },
    "proxyUrls": "http://user:pass@proxy1.example.com:8888,http://user:pass@proxy2.example.com:8888",
}

# Run the Actor and wait for it to finish
run = client.actor("rust_knot/reddit-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "subreddit": "python",
  "query": "web scraping",
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "proxyUrls": "http://user:pass@proxy1.example.com:8888,http://user:pass@proxy2.example.com:8888"
}' |
apify call rust_knot/reddit-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,rust_knot/reddit-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/TXGhLGrHFGbFH5gVq/builds/NlHNVaXjtEd8DngRM/openapi.json