# Reddit Fast Scraper RAG Formatter (`egsand/reddit-fast-scraper-rag-formatter`) Actor

- **URL**: https://apify.com/egsand/reddit-fast-scraper-rag-formatter.md
- **Developed by:** [Enrique G.](https://apify.com/egsand) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.20 / 1,000 reddit post & rag markdowns

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## 🚀 Reddit Fast Scraper & RAG Markdown Formatter for AI

> **The ultra-fast, cost-effective Reddit scraper built specifically for AI agents, LLM pipelines, and RAG (Retrieval-Augmented Generation).**\
> Extract posts, subreddits, keywords, and deep discussion trees without browser bloat, formatted directly into token-dense Markdown.

***

### ⚡ Why This Actor?

Most Reddit scrapers on Apify fall into two traps:

1. **They run heavy headless browsers** (Puppeteer/Playwright) for every page, consuming massive amounts of Compute Units (CU) and costing you 5x more.
2. **They dump raw, messy JSON** with endless nested `replies.data.children` structures that waste up to **70% of your LLM tokens** when fed into prompts or vector databases.

**Reddit Fast Scraper & RAG Formatter** solves both problems:

- **Pure HTTP Execution**: Minimal memory footprint, near-instant runs, and maximum cost efficiency.
- **Smart Deep Comment Tree (`morechildren`)**: Captures complete conversations instead of truncating at 20-30 comments.
- **Native RAG Markdown Output**: Converts complex thread discussions into clean, indented Markdown with YAML frontmatter, ready to drop straight into LangChain, LlamaIndex, Pinecone, Chroma, or ChatGPT/Claude/Gemini context.

***

### 📊 Feature Comparison

| Feature | Standard Reddit Scrapers | **Reddit Fast Scraper & RAG Formatter** |
| :--- | :--- | :--- |
| **Output for LLMs / RAG** | ❌ Messy recursive JSON | ✅ **Token-dense Markdown with Frontmatter** |
| **Comment Depth** | ⚠️ Capped at 20–50 comments | ✅ **Smart recursive expansion of top branches** |
| **Execution Cost** | 💸 Heavy CUs (Browser-based) | ⚡ **Ultra-lightweight Pure HTTP** |
| **Media Resolution** | ❌ Broken or muted `v.redd.it` links | ✅ **Direct HLS playlist (video+audio united)** |
| **Vector DB Ready** | ❌ Requires custom post-processing | ✅ **Ready for embeddings out of the box** |

***

### 📄 Sample RAG Markdown Output

When `outputFormat` is set to `rag_markdown` or `both`, every scraped post includes a `ragMarkdown` field structured like this:

```markdown
---
id: "t3_1k4q8z"
title: "The shift towards local small LLMs in production"
subreddit: "technology"
author: "dev_expert"
score: 4820
upvote_ratio: 0.96
created_at: "2026-10-30T10:15:00.000Z"
url: "https://www.reddit.com/r/technology/comments/1k4q8z/..."
total_comments: 142
extracted_comments: 48
media_type: "text"
---

## The shift towards local small LLMs in production
*Posted by **u/dev_expert** in **r/technology** | Score: 4820 (96% upvoted) | 2026-10-30T10:15:00.000Z*

In the past 6 months, we migrated our classification pipeline from cloud models to fine-tuned 8B local models...

---
### Discussion (48 comments extracted)

- **u/ml_engineer** [Score: 840]:
  Totally agree with this approach. Memory bandwidth is the primary bottleneck when scaling to 10k req/min.
  > - **u/dev_expert** [Score: 210]:
  >   Exactly, we solved that using vLLM with PagedAttention.
  >   > - **u/sysadmin_joe** [Score: 45]:
  >   >   Did you benchmark against TensorRT-LLM?

- **u/cloud_architect** [Score: 320]:
  What was your final hosting cost reduction?
```

***

### 🛠️ Input Options

| Field | Type | Description | Default |
| :--- | :--- | :--- | :--- |
| `startUrls` | Array | Direct Reddit URLs (posts, subreddits, or searches). | `[]` |
| `subreddits` | Array | List of subreddits to scrape (e.g. `["technology", "OpenAI"]`). | `["technology"]` |
| `searchQueries` | Array | Keywords to search across Reddit or targeted subreddits. | `[]` |
| `sort` | String | Sort order: `hot`, `new`, `top`, `rising`, `relevance`. | `"hot"` |
| `timeFilter` | String | Time filter for `top` and search: `hour`, `day`, `week`, `month`, `year`, `all`. | `"all"` |
| `maxPosts` | Integer | Max total posts to scrape. | `25` |
| `maxCommentsPerPost` | Integer | Max comments to extract per post (0 for posts only). | `50` |
| `expandComments` | Boolean | Enable smart expansion of `more comments` nodes. | `true` |
| `maxMoreChildrenExpansions`| Integer | Max batches of additional comments to resolve. | `3` |
| `outputFormat` | String | `rag_markdown`, `structured_json`, or `both`. | `"both"` |
| `includeMedia` | Boolean | Resolve HLS video playlists, audio tracks, and galleries. | `true` |
| `proxyConfiguration` | Object | Apify Proxy settings. Residential recommended for cloud runs. | `Residential` |

***

### 🤖 Integration with AI & Automation Tools

#### 1. LangChain / LlamaIndex (Python & JS)

Simply load the dataset items and map the `ragMarkdown` text directly into `Document(page_content=item['ragMarkdown'], metadata=item)`:

```python
from apify_client import ApifyClient
from langchain.docstore.document import Document

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("YOUR_USERNAME/reddit-rag-fast-scraper").call(run_input={
    "subreddits": ["ArtificialInteligence"],
    "maxPosts": 10,
    "outputFormat": "rag_markdown"
})

documents = [
    Document(page_content=item["ragMarkdown"], metadata={"source": item["url"], "id": item["id"]})
    for item in client.dataset(run["defaultDatasetId"]).iterate_items()
]
```

#### 2. n8n & Make.com

Trigger this actor on a schedule (e.g., every 6 hours), collect new community discussions, and pass the generated Markdown directly to an OpenAI or Gemini node for automated sentiment summarization or Slack/Discord alerts.

***

### 🛡️ Proxy & Anti-Blocking

Reddit implements aggressive rate limiting and Cloudflare WAF challenges.

- **On Apify Cloud**: The Actor automatically utilizes **Apify Residential Proxy** to ensure smooth, unblocked extraction.
- **Custom Session**: You can optionally provide a `sessionCookie` in the input if you want to scrape private or age-restricted (NSFW) subreddits.

# Actor input Schema

## `startUrls` (type: `array`):

Direct URLs to Reddit posts, subreddits, or search pages to scrape.

## `subreddits` (type: `array`):

List of subreddits (without 'r/'). E.g. \['technology', 'ArtificialInteligence'].

## `searchQueries` (type: `array`):

Keywords to search across Reddit or specified subreddits.

## `sort` (type: `string`):

Listing sort order.

## `timeFilter` (type: `string`):

Time window when sorting by 'top' or searching.

## `maxPosts` (type: `integer`):

Maximum total posts to extract per subreddit or search query.

## `maxCommentsPerPost` (type: `integer`):

Maximum number of comments to extract per post (0 for post only).

## `expandComments` (type: `boolean`):

Expand 'more comments' nodes using smart batching to get full discussion threads without timeout.

## `maxMoreChildrenExpansions` (type: `integer`):

Max number of additional comment batches to expand per post (prioritizes highest scored threads).

## `outputFormat` (type: `string`):

Format of the scraped dataset: RAG Markdown (optimized for LLMs), Structured JSON, or Both.

## `includeMedia` (type: `boolean`):

Extract direct media URLs, audio/video streams, and native HLS m3u8 playlists without downloading heavy binaries.

## `sessionCookie` (type: `string`):

Optional: paste your 'reddit_session' cookie for zero-block scraping or accessing NSFW/private communities.

## `proxyConfiguration` (type: `object`):

Optional Apify Proxy. Works out-of-the-box without proxies via automatic Arctic Shift fallback, or enable Residential Proxy for direct Reddit endpoints.

## Actor input object example

```json
{
  "subreddits": [
    "technology"
  ],
  "sort": "hot",
  "timeFilter": "all",
  "maxPosts": 10,
  "maxCommentsPerPost": 50,
  "expandComments": true,
  "maxMoreChildrenExpansions": 3,
  "outputFormat": "both",
  "includeMedia": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `overview` (type: `string`):

Dataset items containing all extracted Reddit posts, nested discussion trees, and token-optimized RAG Markdown.

## `ragMarkdown` (type: `string`):

Token-optimized Markdown representation of each post and its discussion tree.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "subreddits": [
        "technology"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("egsand/reddit-fast-scraper-rag-formatter").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "subreddits": ["technology"],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("egsand/reddit-fast-scraper-rag-formatter").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "subreddits": [
    "technology"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call egsand/reddit-fast-scraper-rag-formatter --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,egsand/reddit-fast-scraper-rag-formatter"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/UTxkOQGB6peiexU1I/builds/shLgUKCuQTYUt0Klt/openapi.json