🚀 Reddit Fast Scraper & RAG Markdown Formatter for AI

The ultra-fast, cost-effective Reddit scraper built specifically for AI agents, LLM pipelines, and RAG (Retrieval-Augmented Generation).
Extract posts, subreddits, keywords, and deep discussion trees without browser bloat, formatted directly into token-dense Markdown.

---

⚡ Why This Actor?

Most Reddit scrapers on Apify fall into two traps:

1. They run heavy headless browsers (Puppeteer/Playwright) for every page, consuming massive amounts of Compute Units (CU) and costing you 5x more.
2. They dump raw, messy JSON with endless nested replies.data.children structures that waste up to 70% of your LLM tokens when fed into prompts or vector databases.

Reddit Fast Scraper & RAG Formatter solves both problems:

• Pure HTTP Execution: Minimal memory footprint, near-instant runs, and maximum cost efficiency.
• Smart Deep Comment Tree (morechildren): Captures complete conversations instead of truncating at 20-30 comments.
• Native RAG Markdown Output: Converts complex thread discussions into clean, indented Markdown with YAML frontmatter, ready to drop straight into LangChain, LlamaIndex, Pinecone, Chroma, or ChatGPT/Claude/Gemini context.

---

📊 Feature Comparison

Feature	Standard Reddit Scrapers	Reddit Fast Scraper & RAG Formatter
Output for LLMs / RAG	❌ Messy recursive JSON	✅ Token-dense Markdown with Frontmatter
Comment Depth	⚠️ Capped at 20–50 comments	✅ Smart recursive expansion of top branches
Execution Cost	💸 Heavy CUs (Browser-based)	⚡ Ultra-lightweight Pure HTTP
Media Resolution	❌ Broken or muted v.redd.it links	✅ Direct HLS playlist (video+audio united)
Vector DB Ready	❌ Requires custom post-processing	✅ Ready for embeddings out of the box

---

📄 Sample RAG Markdown Output

When outputFormat is set to rag_markdown or both, every scraped post includes a ragMarkdown field structured like this:

---
id: "t3_1k4q8z"
title: "The shift towards local small LLMs in production"
subreddit: "technology"
author: "dev_expert"
score: 4820
upvote_ratio: 0.96
created_at: "2026-10-30T10:15:00.000Z"
url: "https://www.reddit.com/r/technology/comments/1k4q8z/..."
total_comments: 142
extracted_comments: 48
media_type: "text"
---

# The shift towards local small LLMs in production
*Posted by **u/dev_expert** in **r/technology** | Score: 4820 (96% upvoted) | 2026-10-30T10:15:00.000Z*

In the past 6 months, we migrated our classification pipeline from cloud models to fine-tuned 8B local models...

---
## Discussion (48 comments extracted)

- **u/ml_engineer** [Score: 840]:
  Totally agree with this approach. Memory bandwidth is the primary bottleneck when scaling to 10k req/min.
  > - **u/dev_expert** [Score: 210]:
  >   Exactly, we solved that using vLLM with PagedAttention.
  >   > - **u/sysadmin_joe** [Score: 45]:
  >   >   Did you benchmark against TensorRT-LLM?

- **u/cloud_architect** [Score: 320]:
  What was your final hosting cost reduction?

---

🛠️ Input Options

Field	Type	Description	Default
startUrls	Array	Direct Reddit URLs (posts, subreddits, or searches).	[]
subreddits	Array	List of subreddits to scrape (e.g. ["technology", "OpenAI"]).	["technology"]
searchQueries	Array	Keywords to search across Reddit or targeted subreddits.	[]
sort	String	Sort order: hot, new, top, rising, relevance.	"hot"
timeFilter	String	Time filter for top and search: hour, day, week, month, year, all.	"all"
maxPosts	Integer	Max total posts to scrape.	25
maxCommentsPerPost	Integer	Max comments to extract per post (0 for posts only).	50
expandComments	Boolean	Enable smart expansion of more comments nodes.	true
maxMoreChildrenExpansions	Integer	Max batches of additional comments to resolve.	3
outputFormat	String	rag_markdown, structured_json, or both.	"both"
includeMedia	Boolean	Resolve HLS video playlists, audio tracks, and galleries.	true
proxyConfiguration	Object	Apify Proxy settings. Residential recommended for cloud runs.	Residential

---

🤖 Integration with AI & Automation Tools

1. LangChain / LlamaIndex (Python & JS)

Simply load the dataset items and map the ragMarkdown text directly into Document(page_content=item['ragMarkdown'], metadata=item):

from apify_client import ApifyClient
from langchain.docstore.document import Document

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("YOUR_USERNAME/reddit-rag-fast-scraper").call(run_input={
    "subreddits": ["ArtificialInteligence"],
    "maxPosts": 10,
    "outputFormat": "rag_markdown"
})

documents = [
    Document(page_content=item["ragMarkdown"], metadata={"source": item["url"], "id": item["id"]})
    for item in client.dataset(run["defaultDatasetId"]).iterate_items()
]

2. n8n & Make.com

Trigger this actor on a schedule (e.g., every 6 hours), collect new community discussions, and pass the generated Markdown directly to an OpenAI or Gemini node for automated sentiment summarization or Slack/Discord alerts.

---

🛡️ Proxy & Anti-Blocking

Reddit implements aggressive rate limiting and Cloudflare WAF challenges.

• On Apify Cloud: The Actor automatically utilizes Apify Residential Proxy to ensure smooth, unblocked extraction.
• Custom Session: You can optionally provide a sessionCookie in the input if you want to scrape private or age-restricted (NSFW) subreddits.