Reddit Archive Scraper — Historical Posts for RAG

Export historical Reddit posts and optional comments as structured JSON or CSV. Choose a subreddit, explicit UTC date bounds, a post cap, and oldest-first or newest-first ordering. The linked n8n workflow converts these rows into Markdown and a Sheets export for retrieval projects. Archive coverage and freshness vary; start with a small dated sample before a large backfill.

Try it: Archive historical Python posts · n8n: import reddit-archive → Sheets/Slack (reddit-archive-rag-sheets-slack.json)

Need a packaged dump? Reddit RAG dump starts with a $250 sample, credited toward a larger order. Email benthepythondev0@gmail.com. Self-serve fees include saved rows, non-empty comment threads and the start event; see the full pricing examples below. Issues stay for bugs.

📚 What is the Reddit Archive Scraper?

It turns any subreddit or keyword into a structured historical dataset. Give it a list of subreddits (and/or a keyword), an optional afterDate/beforeDate window and a maxPosts cap, and it pages through the archive in your chosen direction, returning available matching posts up to the cap — and, optionally, its archived comments. Each row carries a type field (post or comment) so you can split or join them effortlessly. Ideal for backfilling a database, building sentiment and AI/RAG datasets, or studying how a topic was discussed over years.

What data does it extract?

Posts (type: "post"):

• id, title, selftext (body), author, subreddit
• score, upvote_ratio, num_comments
• created_utc (epoch) and created_iso (ISO timestamp)
• permalink, url, domain, link_flair_text
• Flags: is_self, is_video, over_18, locked, stickied, spoiler, total_awards_received

Comments (type: "comment", optional):

• id, post_id, parent_id, link_id, body, author, subreddit
• score, created_utc, created_iso, permalink
• is_submitter, total_awards_received

Who it’s for

AI engineers building RAG corpora who need years of Reddit, not just the latest ~1,000 posts.

Verified output and limits

Selected fields from an actual r/Python post returned on September 5, 2026:

{
  "type": "post",
  "id": "18vkgtu",
  "title": "Monday Daily Thread: Project ideas!",
  "subreddit": "Python",
  "score": 1,
  "num_comments": 2,
  "created_utc": 1704067208,
  "created_iso": "2024-01-01T00:00:08+00:00",
  "permalink": "https://www.reddit.com/r/Python/comments/18vkgtu/monday_daily_thread_project_ideas/"
}

Build 1.0.18 returned 250 unique posts oldest-first and 250 unique posts newest-first in separate cloud checks over January 2024. A comment check returned three posts and six comments, with nine result events and three comment-thread events. These checks establish correct output for the tested inputs, not complete archive coverage.

Pricing

On the Free tier, every saved post or comment costs $0.003. When includeComments is enabled and at least one eligible comment is returned for a post, its comment thread adds $0.005, once for that post. This fee is active; it is not a future change. Both the post and its comment rows also count as results.

At 512 MB, the start event is $0.00005. Larger memory settings charge one start event per GB, with a minimum of one. Apify plan discounts apply to all events. Examples before discounts:

Output	Cost
250 posts, comments off	$0.75005
3 posts + 6 comments across 3 non-empty threads	$0.04205
20 posts + 200 comments across 20 non-empty threads	$0.76005

The final example illustrates the formula, not a measured run. maxPosts caps posts rather than total dataset rows; comments increase both the row count and bill. Set Apify's maximum run charge as an additional budget limit. An empty result has no result or thread charge, but the start event applies.

⬇️ Input

Give it at least one subreddit or a keyword, then scope the window and size:

Field	Description
subreddits	One or more subreddits (without r/). Leave empty to search all of Reddit by keyword
searchQuery	Optional keyword — combine with subreddits, or use alone across all of Reddit
sortOrder	newest (default) or oldest first within the archive window
minScore	Optional minimum Reddit score for both posts and included comments
afterDate	Inclusive start, YYYY-MM-DD at 00:00 UTC, for posts
beforeDate	Exclusive end, YYYY-MM-DD at 00:00 UTC, for posts
maxPosts	Max posts across all subreddits (1–500000)
includeComments	Also fetch archived comments per post (raises result count and cost)
maxCommentsPerPost	Cap comments fetched per post (only when comments are on)

Example input

{
  "subreddits": ["FragranceClones"],
  "searchQuery": "dupe",
  "sortOrder": "newest",
  "minScore": 5,
  "afterDate": "2024-01-01",
  "beforeDate": "2025-01-01",
  "maxPosts": 5000,
  "includeComments": true,
  "maxCommentsPerPost": 50
}

⬆️ Output

Each post or comment is a separate dataset row. Use (type, id) as the unique key when merging exports. Posts keep selftext; comments keep body and link back through post_id, parent_id and link_id. Missing or deleted source text remains missing or marked as deleted.

Post date bounds do not constrain the dates of comments on those posts. Comment count is limited by maxCommentsPerPost, archive availability, and the optional score threshold. num_comments describes the archived post and can exceed the number of comment rows returned.

The Actor deduplicates within a run. It does not persist a cross-run cursor or deduplicate earlier exports. For recurring refreshes, store (type, id) in your destination and use overlapping bounded windows to accommodate late archive ingestion. Large exports should use the asynchronous run endpoint, wait for success, then page through the resulting dataset; do not assume a synchronous HTTP connection can stay open for the whole backfill.

An upstream outage, unrecognized response, or stalled pagination cursor fails the run. Inspect the run status before treating a partial dataset as complete. A successful empty run means the archive returned no eligible matches for that request; it does not prove that Reddit had no posts in that period.

💡 Use cases

• 🗃️ Historical backfill: seed a database with years of a subreddit's posts and comments in a single run.
• 🤖 AI / RAG training data: build large, topic-specific historical corpora for fine-tuning or retrieval.
• 📈 Research & sentiment datasets: analyse how opinions and trends shifted across long time spans.
• 🔎 Brand & product monitoring: study archived mentions of a brand, product or keyword over time.

💰 Cost tips

Pricing is pay-per-result, so you're charged per post or comment returned. Comments are usually the bulk of the count — leave includeComments off if you only need posts, or set maxCommentsPerPost to cap them. Use afterDate/beforeDate to scope exactly the window you need and avoid pulling more than you'll use.

❓ FAQ

How do I scrape historical Reddit data? Enter one or more subreddits (and/or a searchQuery), optionally set a date window, then Run. It follows the selected time order and returns structured posts (and comments if enabled).

Why not just use Reddit's official API? Reddit's API hard-caps any listing at ~1000 posts — a few weeks for a busy subreddit. The archive lets you reach months or years of history that the live API simply won't return.

Where does the data come from? The public PullPush and Arctic Shift archives. The Actor automatically selects a healthy backend; coverage and freshness depend on those third-party services.

Can I search all of Reddit by keyword? Yes — leave subreddits empty and set searchQuery to search across all of Reddit, or combine both to filter a keyword within specific subreddits.

Can I get comments too? Yes — turn on includeComments. Each post's archived comments are returned as separate rows with type: "comment"; use maxCommentsPerPost to cap them. minScore applies to comments too, so low-signal replies do not enter a RAG pack.

How do I separate posts from comments? Every row has a type field set to post or comment, plus post_id/link_id on comments to join them back to their thread.

Do I need an API key or login? No. It reads the public archive — no Reddit account, app credentials or API key required.

How far back does it go? As far as the archive holds for that subreddit or keyword; use afterDate to set the earliest date you want. Set sortOrder to oldest when you want to begin at the start of that window.

Can I run it on a schedule or via API? Yes — schedule recurring runs in Apify, call it via the API/SDK, or connect it to Make, Zapier or n8n.

Can you deliver a packaged dump? Yes. A sample on your subreddit is $250 (credited). Larger windows are a fixed quote. Prices: https://benthepythondev00.github.io/reddit-rag-dump.html — inbound email only.

Is scraping Reddit data legal? It reads publicly archived data. Use it for lawful purposes and follow Reddit's and each archive provider's terms.

🔗 You might also like

• Reddit Scraper — live posts, comments & AI-ready markdown
• Hacker News Intelligence — HN stories & comment threads
• OpenAlex Scraper — academic papers & citations
• arXiv Scraper — scientific papers, abstracts & PDFs

---

Keywords: reddit rag, reddit RAG pack, reddit RAG dump, historical reddit dump, Pushshift alternative, PullPush, reddit historical posts, Reddit archive scraper, historical Reddit posts, Arctic Shift, Reddit posts scraper, Reddit comments scraper, subreddit scraper, Reddit API alternative, RAG training data, weekly Reddit refresh, Reddit data export, keyword search Reddit.