Reddit Archive Scraper — Historical Posts for RAG

Build a markdown/JSON RAG pack from Reddit posts and comments — the last three months of r/whatever, or years past the official API’s ~1,000-post cap. Set afterDate for a weekly refresh; leave the window open for a full backfill.

Try it: Archive historical Python posts · n8n: import reddit-archive → Sheets/Slack (reddit-archive-rag-sheets-slack.json)

Need a packaged dump instead of babysitting a 50k–1M row run? Reddit RAG dump — sample $250, dump $2–4k. Email benthepythondev0@gmail.com. Store PPE stays $2.40/1k Gold. Issues stay for bugs.

📚 What is the Reddit Archive Scraper?

It turns any subreddit or keyword into a structured historical dataset. Give it a list of subreddits (and/or a keyword), an optional afterDate/beforeDate window and a maxPosts cap, and it pages backward through time from the archive, returning every matching post — and, optionally, its archived comments. Each row carries a type field (post or comment) so you can split or join them effortlessly. Ideal for backfilling a database, building sentiment and AI/RAG datasets, or studying how a topic was discussed over years.

What data does it extract?

Posts (type: "post"):

• id, title, selftext (body), author, subreddit
• score, upvote_ratio, num_comments
• created_utc (epoch) and created_iso (ISO timestamp)
• permalink, url, domain, link_flair_text
• Flags: is_self, is_video, over_18, locked, stickied, spoiler, total_awards_received

Comments (type: "comment", optional):

• id, post_id, parent_id, link_id, body, author, subreddit
• score, created_utc, created_iso, permalink
• is_submitter, total_awards_received

Who it’s for

AI engineers building RAG corpora who need years of Reddit, not just the latest ~1,000 posts.

Sample output (post)

{
  "type": "post",
  "id": "1abc234",
  "title": "Interview about Deep learning case study [D]",
  "selftext": "I will have interview in Deep Learning specialist position...",
  "author": "example_user",
  "subreddit": "MachineLearning",
  "score": 42,
  "num_comments": 12,
  "created_iso": "2024-06-01T12:00:00Z",
  "permalink": "https://www.reddit.com/r/MachineLearning/comments/1abc234/..."
}

One-run proof

Cloud run gJAFTYmczF0JICN6B: 119 items (25 posts + 94 comments) in ~38s.
Example: https://apify.com/benthepythondev/reddit-archive-scraper/examples/archive-historical-python-posts
n8n: https://gist.github.com/benthepythondev00/e28968a940fe4195ffb5a997fe309503

Pricing note

Base price is per dataset result (post or comment). Optional enrichment: comment-thread when comments are fetched for a post (scheduled 2026-08-27, Free $0.005).

⬇️ Input

Give it at least one subreddit or a keyword, then scope the window and size:

Field	Description
subreddits	One or more subreddits (without r/). Leave empty to search all of Reddit by keyword
searchQuery	Optional keyword — combine with subreddits, or use alone across all of Reddit
sortOrder	newest (default) or oldest first within the archive window
minScore	Optional minimum Reddit score for both posts and included comments
afterDate	Earliest date YYYY-MM-DD (lower bound)
beforeDate	Latest date YYYY-MM-DD (start point / upper bound)
maxPosts	Max posts across all subreddits (1–500000)
includeComments	Also fetch archived comments per post (raises result count and cost)
maxCommentsPerPost	Cap comments fetched per post (only when comments are on)

Example input

{
  "subreddits": ["FragranceClones"],
  "searchQuery": "dupe",
  "sortOrder": "newest",
  "minScore": 5,
  "afterDate": "2024-01-01",
  "beforeDate": "2025-01-01",
  "maxPosts": 5000,
  "includeComments": true,
  "maxCommentsPerPost": 50
}

⬆️ Output

Every post and comment is one clean row (view as a table, or export JSON / CSV / Excel):

{
  "type": "post",
  "id": "1d8bw4c",
  "title": "Best clone of Cool Water?",
  "selftext": "Looking for an affordable alternative...",
  "author": "someuser",
  "subreddit": "FragranceClones",
  "score": 14,
  "upvote_ratio": 0.93,
  "num_comments": 8,
  "created_utc": 1717322040,
  "created_iso": "2024-06-02T10:14:00+00:00",
  "permalink": "https://www.reddit.com/r/FragranceClones/comments/1d8bw4c/...",
  "url": "https://www.reddit.com/r/FragranceClones/comments/1d8bw4c/...",
  "domain": "self.FragranceClones",
  "link_flair_text": "Discussion",
  "is_self": true,
  "over_18": false,
  "total_awards_received": 0
}

💡 Use cases

• 🗃️ Historical backfill: seed a database with years of a subreddit's posts and comments in a single run.
• 🤖 AI / RAG training data: build large, topic-specific historical corpora for fine-tuning or retrieval.
• 📈 Research & sentiment datasets: analyse how opinions and trends shifted across long time spans.
• 🔎 Brand & product monitoring: see everything said about a brand, product or keyword over the years.

💰 Cost tips

Pricing is pay-per-result, so you're charged per post or comment returned. Comments are usually the bulk of the count — leave includeComments off if you only need posts, or set maxCommentsPerPost to cap them. Use afterDate/beforeDate to scope exactly the window you need and avoid pulling more than you'll use.

❓ FAQ

How do I scrape historical Reddit data? Enter one or more subreddits (and/or a searchQuery), optionally set a date window, then Run. It pages backward through the archive and returns structured posts (and comments if enabled).

Why not just use Reddit's official API? Reddit's API hard-caps any listing at ~1000 posts — a few weeks for a busy subreddit. The archive lets you reach months or years of history that the live API simply won't return.

Where does the data come from? The public PullPush and Arctic Shift archives. The Actor automatically selects a healthy backend; coverage and freshness depend on those third-party services.

Can I search all of Reddit by keyword? Yes — leave subreddits empty and set searchQuery to search across all of Reddit, or combine both to filter a keyword within specific subreddits.

Can I get comments too? Yes — turn on includeComments. Each post's archived comments are returned as separate rows with type: "comment"; use maxCommentsPerPost to cap them. minScore applies to comments too, so low-signal replies do not enter a RAG pack.

How do I separate posts from comments? Every row has a type field set to post or comment, plus post_id/link_id on comments to join them back to their thread.

Do I need an API key or login? No. It reads the public archive — no Reddit account, app credentials or API key required.

How far back does it go? As far as the archive holds for that subreddit or keyword; use afterDate to set the earliest date you want. Set sortOrder to oldest when you want to begin at the start of that window.

Can I run it on a schedule or via API? Yes — schedule recurring runs in Apify, call it via the API/SDK, or connect it to Make, Zapier or n8n.

Can you deliver a packaged dump? Yes. A sample on your subreddit is $250 (credited). Larger windows are a fixed quote. Prices: https://benthepythondev00.github.io/reddit-rag-dump.html — inbound email only.

Is scraping Reddit data legal? It reads publicly archived data. Use it for lawful purposes and follow Reddit's and each archive provider's terms.

🔗 You might also like

• Reddit Scraper — live posts, comments & AI-ready markdown
• Hacker News Intelligence — HN stories & comment threads
• OpenAlex Scraper — academic papers & citations
• arXiv Scraper — scientific papers, abstracts & PDFs

---

Keywords: reddit rag, reddit RAG pack, reddit RAG dump, historical reddit dump, Pushshift alternative, PullPush, reddit historical posts, Reddit archive scraper, historical Reddit posts, Arctic Shift, Reddit posts scraper, Reddit comments scraper, subreddit scraper, Reddit API alternative, RAG training data, weekly Reddit refresh, Reddit data export, keyword search Reddit.