Reddit Historical Archive Scraper — RAG & Comment Search

Build RAG corpora and research datasets from historical Reddit posts and comments for AI engineers, analysts, and agents. Each row includes documentId, chunkText, title, body, subreddit, score, createdAt, sourceUrl, and archiveSource. The Actor reads public Arctic Shift and PullPush archives over HTTP — no Reddit login, OAuth, browser, or buyer-supplied credential.

The Actor is designed as a focused standalone workflow. Point it at subreddits, keywords, comment-body queries, post IDs, usernames, or Reddit URLs, set a date window, and export embeddable chunks from the Dataset tab. Its specialty is historical archive coverage and provenance-aware corpus building.

Best fit for this Actor

• Backfill a niche subreddit past Reddit’s shallow listing window into embedding chunks.
• Full-text search archived comment bodies for a brand or topic phrase.
• Dump one archived post thread as flat post and comment rows for a corpus.

When the next step is a live hot, new, or rising feed rather than an archive backfill, continue with Reddit Posts Scraper.

When you need provider-backed live posts, comments, and search in one Actor after the historical pack, use Reddit Posts & Comments Scraper.

Why agents choose this Actor

Use it when the task is to retrieve historical Reddit discussion with a bounded date window, then hand useful text directly to an embedding, retrieval, or review workflow. The input is deliberately narrow: target the archive by subreddit, post query, comment-body query, user, post ID, or Reddit URL, and set maxItems before the call. The return shape is stable and flat: one validated post or comment per dataset row, with a provenance field and a source URL for review.

Live-feed neighbors

• When a workflow needs live hot, new, or rising feeds, use Reddit Posts Scraper.
• When a workflow needs combined live posts-and-comments collection, use Reddit Posts & Comments Scraper.

This archive contract covers historical public data rather than account authentication or moderation actions.

Practical scenario

An AI engineer needs H1 discussion from r/MachineLearning as embedding chunks. She sets subreddits to MachineLearning, bounds afterDate and beforeDate to that half-year, and keeps maxItems at 3 for a cheap sample. The dataset returns three posts with chunkText ready to embed, plus sourceUrl and archiveSource. She raises maxItems and optionally enables includeComments once the sample looks right.

Quick start input

{
  "subreddits": ["MachineLearning"],
  "afterDate": "2024-01-01",
  "beforeDate": "2024-06-30",
  "maxItems": 3,
  "includeComments": false,
  "sortOrder": "newest"
}

Provide at least one of subreddits, searchQueries, commentSearchQueries, postIds, usernames, or startUrls. Empty input finishes as INVALID_INPUT with zero archived-item charges.

Input reference

Field	Type	What it controls
subreddits	array	Subreddit names (with or without r/). Prefill: MachineLearning.
searchQueries	array	Keyword search on archived post title and self-text.
commentSearchQueries	array	Full-text search inside archived comment bodies.
postIds	array	Base36 IDs from /comments/{id}/.
usernames	array	Authors to archive (userContent selects posts, comments, or both).
startUrls	array	Reddit URLs auto-classified as subreddit, post, or user.
afterDate / beforeDate	string	YYYY-MM-DD or ISO window bounds.
sortOrder	enum	newest (default) or oldest.
minScore	integer	Keep higher-engagement rows in the corpus.
includeComments	boolean	Also fetch archived comments for each post. Default false.
maxCommentsPerPost	integer	Cap comments per post when comments are on. Default 50.
maxItems	integer	Hard row cap for the run. Prefill 3.
requestDelayMs	integer	Delay between archive requests. Default 800.
proxyConfiguration	object	Apify Proxy; Residential helps when archives throttle datacenter IPs.

What data you receive

One dataset item is one archived post or comment.

{
  "type": "post",
  "documentId": "reddit:post:1hqoku0",
  "chunkText": "# Where to start?\n\nLooking for beginner ML resources...",
  "title": "Where to start?",
  "body": "Looking for beginner ML resources...",
  "author": "example_user",
  "subreddit": "MachineLearning",
  "score": 42,
  "numComments": 12,
  "createdAt": "2024-03-15T12:00:00.000Z",
  "createdUtc": 1710504000,
  "permalink": "/r/MachineLearning/comments/1hqoku0/where_to_start/",
  "sourceUrl": "https://www.reddit.com/r/MachineLearning/comments/1hqoku0/where_to_start/",
  "postId": "1hqoku0",
  "archiveSource": "arctic-shift",
  "retrievedAt": "2026-08-21T12:00:00.000Z",
  "queryTarget": "subreddit:MachineLearning"
}

Field	Meaning
documentId	Stable RAG id (reddit:post:{id} or reddit:comment:{id}).
chunkText	Ready-to-embed text (title plus body for posts; body for comments).
archiveSource	arctic-shift or pullpush.
sourceUrl	Canonical https://www.reddit.com... link.

OUTPUT and RUN_SUMMARY in the default key-value store hold outcome, itemsPushed, and chargedEventCounts.

The useful-content filter retains posts with a usable title or body and comments with a usable body; this keeps each returned corpus row meaningful for retrieval.

Use through the API

curl -X POST "https://api.apify.com/v2/acts/khadinakbar~reddit-archive-scraper/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"subreddits":["LocalLLaMA"],"afterDate":"2024-01-01","beforeDate":"2024-03-31","maxItems":50}'

Download rows as JSON, CSV, Excel, or HTML from the Dataset tab.

Use with AI agents through Apify MCP

Pull historical posts from r/MachineLearning between afterDate and beforeDate. Return documentId, chunkText, sourceUrl, createdAt, archiveSource, and outcome. Cap maxItems for a cheap sample.

Connect via https://mcp.apify.com. Read OUTPUT.outcome and itemsPushed to interpret empty datasets. Charge signal: $0.005 per archived-item plus start.

Agent checklist

1. Choose exactly one primary target first; combine targets only when deduplication across them is intentional.
2. Set afterDate, beforeDate, and maxItems before calling so both scope and cost are bounded.
3. Read OUTPUT.outcome: COMPLETE means the requested work finished; PARTIAL preserves useful rows after a cap or route issue; VALID_EMPTY means the archive returned no qualifying row; INVALID_INPUT supplies a repair; UPSTREAM_FAILED records an unavailable archive route.
4. Use documentId as the idempotent corpus key, chunkText for embeddings, and sourceUrl plus archiveSource when citing or auditing a result.

Connect the workflow

• When you need live Reddit hot or new listings after the archive backfill, continue with Reddit Posts Scraper.
• For provider-backed live posts, comments, and search in one Actor, use Reddit Posts & Comments Scraper.
• For live keyword discovery across Reddit, start with Reddit Search Scraper.

Pricing

Pay per event plus platform usage. Confirm current event prices on the live Pricing tab; that tab is the source of truth if this page ever lags.

• apify-actor-start: $0.00005 per run
• archived-item: $0.005 per persisted post or comment row ($5 per 1,000 rows)

A three-item quality sample is about $0.01505 in result events plus Apify platform usage (compute and proxy). Platform usage is paid by the run user. The public archive routes use zero per-record provider fee.

How it works

1. Collect jobs from subreddits, queries, post IDs, usernames, and Reddit URLs.
2. Query Arctic Shift first; fall back to PullPush when needed.
3. Normalize RAG fields, dedupe by documentId, and stop at maxItems.
4. Atomically persist and charge archived-item for each validated useful row. Input-repair, useful-empty, and unavailable-route outcomes finish without result-event charges.

Best results

• Keep a tight date window and optional minScore so the corpus stays on-topic.
• Use maxItems: 3 for a cheap agent check, then raise the cap for bulk packs.
• Enable includeComments only after posts look good — comments multiply row count and cost.
• Keep Residential proxy on when archive hosts throttle shared datacenter IPs.

Builder's note

I built this around Arctic Shift’s public search and comment-tree endpoints because Reddit’s live listings stop after a shallow page window and native search does not reach comment bodies. PullPush stays as a Pushshift-compatible fallback when Arctic Shift is slow or rate-limited. The important product choice was shipping documentId and chunkText on every row so an agent can embed results without a second transform step.

Legal and responsible use

Use this Actor on publicly archived Reddit data you are authorized to process, follow applicable law and Reddit’s terms, and keep the output in your own compliance workflow. Removed content may appear as stubs or be absent. This Actor is independent of Reddit, Inc.

Issues and feature requests: use the Actor Issues tab on Apify.