# Reddit Historical Archive Scraper — RAG & Comment Search (`khadinakbar/reddit-archive-scraper`) Actor

Archive historical Reddit posts and comments for RAG and research. Search by date, subreddit, author, post, or comment body beyond live listings. Use a live sibling for hot/new feeds. Returns RAG-ready rows with documentId, chunkText, sourceUrl, createdAt, and provenance. $0.005 per item plus usage.

- **URL**: https://apify.com/khadinakbar/reddit-archive-scraper.md
- **Developed by:** [Khadin Akbar](https://apify.com/khadinakbar) (community)
- **Categories:** AI, Social media, MCP servers
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 archived reddit items

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Reddit Historical Archive Scraper — RAG & Comment Search

Build RAG corpora and research datasets from historical Reddit posts and comments for AI engineers, analysts, and agents. Each row includes `documentId`, `chunkText`, `title`, `body`, `subreddit`, `score`, `createdAt`, `sourceUrl`, and `archiveSource`. The Actor reads public Arctic Shift and PullPush archives over HTTP — no Reddit login, OAuth, browser, or buyer-supplied credential.

The Actor is designed as a focused standalone workflow. Point it at subreddits, keywords, comment-body queries, post IDs, usernames, or Reddit URLs, set a date window, and export embeddable chunks from the Dataset tab. Its specialty is historical archive coverage and provenance-aware corpus building.

### Best fit for this Actor

- Backfill a niche subreddit past Reddit’s shallow listing window into embedding chunks.
- Full-text search archived comment bodies for a brand or topic phrase.
- Dump one archived post thread as flat post and comment rows for a corpus.

When the next step is a live hot, new, or rising feed rather than an archive backfill, continue with [Reddit Posts Scraper](https://apify.com/khadinakbar/reddit-posts-scraper).

When you need provider-backed live posts, comments, and search in one Actor after the historical pack, use [Reddit Posts & Comments Scraper](https://apify.com/khadinakbar/reddit-posts-comments-scraper).

### Why agents choose this Actor

Use it when the task is to retrieve historical Reddit discussion with a bounded date window, then hand useful text directly to an embedding, retrieval, or review workflow. The input is deliberately narrow: target the archive by subreddit, post query, comment-body query, user, post ID, or Reddit URL, and set `maxItems` before the call. The return shape is stable and flat: one validated post or comment per dataset row, with a provenance field and a source URL for review.

#### Live-feed neighbors

- When a workflow needs live `hot`, `new`, or `rising` feeds, use [Reddit Posts Scraper](https://apify.com/khadinakbar/reddit-posts-scraper).
- When a workflow needs combined live posts-and-comments collection, use [Reddit Posts & Comments Scraper](https://apify.com/khadinakbar/reddit-posts-comments-scraper).

This archive contract covers historical public data rather than account authentication or moderation actions.

### Practical scenario

An AI engineer needs H1 discussion from `r/MachineLearning` as embedding chunks. She sets `subreddits` to `MachineLearning`, bounds `afterDate` and `beforeDate` to that half-year, and keeps `maxItems` at `3` for a cheap sample. The dataset returns three posts with `chunkText` ready to embed, plus `sourceUrl` and `archiveSource`. She raises `maxItems` and optionally enables `includeComments` once the sample looks right.

### Quick start input

```json
{
  "subreddits": ["MachineLearning"],
  "afterDate": "2024-01-01",
  "beforeDate": "2024-06-30",
  "maxItems": 3,
  "includeComments": false,
  "sortOrder": "newest"
}
```

Provide at least one of `subreddits`, `searchQueries`, `commentSearchQueries`, `postIds`, `usernames`, or `startUrls`. Empty input finishes as `INVALID_INPUT` with zero `archived-item` charges.

### Input reference

| Field | Type | What it controls |
| --- | --- | --- |
| `subreddits` | array | Subreddit names (with or without `r/`). Prefill: `MachineLearning`. |
| `searchQueries` | array | Keyword search on archived post title and self-text. |
| `commentSearchQueries` | array | Full-text search inside archived comment bodies. |
| `postIds` | array | Base36 IDs from `/comments/{id}/`. |
| `usernames` | array | Authors to archive (`userContent` selects posts, comments, or both). |
| `startUrls` | array | Reddit URLs auto-classified as subreddit, post, or user. |
| `afterDate` / `beforeDate` | string | `YYYY-MM-DD` or ISO window bounds. |
| `sortOrder` | enum | `newest` (default) or `oldest`. |
| `minScore` | integer | Keep higher-engagement rows in the corpus. |
| `includeComments` | boolean | Also fetch archived comments for each post. Default `false`. |
| `maxCommentsPerPost` | integer | Cap comments per post when comments are on. Default 50. |
| `maxItems` | integer | Hard row cap for the run. Prefill 3. |
| `requestDelayMs` | integer | Delay between archive requests. Default 800. |
| `proxyConfiguration` | object | Apify Proxy; Residential helps when archives throttle datacenter IPs. |

### What data you receive

One dataset item is one archived post or comment.

```json
{
  "type": "post",
  "documentId": "reddit:post:1hqoku0",
  "chunkText": "# Where to start?\n\nLooking for beginner ML resources...",
  "title": "Where to start?",
  "body": "Looking for beginner ML resources...",
  "author": "example_user",
  "subreddit": "MachineLearning",
  "score": 42,
  "numComments": 12,
  "createdAt": "2024-03-15T12:00:00.000Z",
  "createdUtc": 1710504000,
  "permalink": "/r/MachineLearning/comments/1hqoku0/where_to_start/",
  "sourceUrl": "https://www.reddit.com/r/MachineLearning/comments/1hqoku0/where_to_start/",
  "postId": "1hqoku0",
  "archiveSource": "arctic-shift",
  "retrievedAt": "2026-08-21T12:00:00.000Z",
  "queryTarget": "subreddit:MachineLearning"
}
```

| Field | Meaning |
| --- | --- |
| `documentId` | Stable RAG id (`reddit:post:{id}` or `reddit:comment:{id}`). |
| `chunkText` | Ready-to-embed text (title plus body for posts; body for comments). |
| `archiveSource` | `arctic-shift` or `pullpush`. |
| `sourceUrl` | Canonical `https://www.reddit.com...` link. |

`OUTPUT` and `RUN_SUMMARY` in the default key-value store hold `outcome`, `itemsPushed`, and `chargedEventCounts`.

The useful-content filter retains posts with a usable title or body and comments with a usable body; this keeps each returned corpus row meaningful for retrieval.

### Use through the API

```bash
curl -X POST "https://api.apify.com/v2/acts/khadinakbar~reddit-archive-scraper/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"subreddits":["LocalLLaMA"],"afterDate":"2024-01-01","beforeDate":"2024-03-31","maxItems":50}'
```

Download rows as JSON, CSV, Excel, or HTML from the Dataset tab.

### Use with AI agents through Apify MCP

> Pull historical posts from r/MachineLearning between afterDate and beforeDate. Return documentId, chunkText, sourceUrl, createdAt, archiveSource, and outcome. Cap maxItems for a cheap sample.

Connect via <https://mcp.apify.com>. Read `OUTPUT.outcome` and `itemsPushed` to interpret empty datasets. Charge signal: `$0.005` per `archived-item` plus start.

#### Agent checklist

1. Choose exactly one primary target first; combine targets only when deduplication across them is intentional.
2. Set `afterDate`, `beforeDate`, and `maxItems` before calling so both scope and cost are bounded.
3. Read `OUTPUT.outcome`: `COMPLETE` means the requested work finished; `PARTIAL` preserves useful rows after a cap or route issue; `VALID_EMPTY` means the archive returned no qualifying row; `INVALID_INPUT` supplies a repair; `UPSTREAM_FAILED` records an unavailable archive route.
4. Use `documentId` as the idempotent corpus key, `chunkText` for embeddings, and `sourceUrl` plus `archiveSource` when citing or auditing a result.

### Connect the workflow

- When you need live Reddit hot or new listings after the archive backfill, continue with [Reddit Posts Scraper](https://apify.com/khadinakbar/reddit-posts-scraper).
- For provider-backed live posts, comments, and search in one Actor, use [Reddit Posts & Comments Scraper](https://apify.com/khadinakbar/reddit-posts-comments-scraper).
- For live keyword discovery across Reddit, start with [Reddit Search Scraper](https://apify.com/khadinakbar/reddit-search-scraper).

### Pricing

Pay per event plus platform usage. Confirm current event prices on the live Pricing tab; that tab is the source of truth if this page ever lags.

- `apify-actor-start`: $0.00005 per run
- `archived-item`: **$0.005** per persisted post or comment row ($5 per 1,000 rows)

A three-item quality sample is about $0.01505 in result events plus Apify platform usage (compute and proxy). Platform usage is paid by the run user. The public archive routes use zero per-record provider fee.

### How it works

1. Collect jobs from subreddits, queries, post IDs, usernames, and Reddit URLs.
2. Query Arctic Shift first; fall back to PullPush when needed.
3. Normalize RAG fields, dedupe by `documentId`, and stop at `maxItems`.
4. Atomically persist and charge `archived-item` for each validated useful row. Input-repair, useful-empty, and unavailable-route outcomes finish without result-event charges.

### Best results

- Keep a tight date window and optional `minScore` so the corpus stays on-topic.
- Use `maxItems: 3` for a cheap agent check, then raise the cap for bulk packs.
- Enable `includeComments` only after posts look good — comments multiply row count and cost.
- Keep Residential proxy on when archive hosts throttle shared datacenter IPs.

### Builder's note

I built this around Arctic Shift’s public search and comment-tree endpoints because Reddit’s live listings stop after a shallow page window and native search does not reach comment bodies. PullPush stays as a Pushshift-compatible fallback when Arctic Shift is slow or rate-limited. The important product choice was shipping `documentId` and `chunkText` on every row so an agent can embed results without a second transform step.

### Legal and responsible use

Use this Actor on publicly archived Reddit data you are authorized to process, follow applicable law and Reddit’s terms, and keep the output in your own compliance workflow. Removed content may appear as stubs or be absent. This Actor is independent of Reddit, Inc.

Issues and feature requests: use the Actor Issues tab on Apify.

# Actor input Schema

## `subreddits` (type: `array`):

Subreddit names to archive historically (e.g. MachineLearning). Accepts with or without r/ prefix. Defaults empty. NOT live Reddit hot listings — this pages the public archive by date.

## `searchQueries` (type: `array`):

Keywords matched against archived post titles and self-text (e.g. retrieval augmented generation). Combine with subreddits and date bounds. Defaults empty. NOT comment-body search — use commentSearchQueries for that.

## `commentSearchQueries` (type: `array`):

Full-text search inside archived comment bodies (e.g. embeddings). Reddit native search cannot do this. Defaults empty. NOT post-title search — use searchQueries for posts.

## `postIds` (type: `array`):

Reddit post base36 IDs from /comments/{id}/ (e.g. 1hqoku0). Fetches each archived post and, when includeComments is on, its comment tree. Defaults empty.

## `usernames` (type: `array`):

Reddit usernames whose archived posts and/or comments to pull (e.g. spez). Accepts with or without u/ prefix. Defaults empty. Controlled by userContent.

## `startUrls` (type: `array`):

Reddit URLs auto-classified as subreddit, post, or user (e.g. https://www.reddit.com/r/MachineLearning/). Mix types in one run. Defaults empty.

## `afterDate` (type: `string`):

Only return items created on or after this date. Format YYYY-MM-DD or ISO timestamp (e.g. 2024-01-01). Leave empty for no lower bound. NOT a Reddit sort timeframe like week/month.

## `beforeDate` (type: `string`):

Only return items created on or before this date. Format YYYY-MM-DD or ISO timestamp (e.g. 2024-06-30). Leave empty for no upper bound.

## `sortOrder` (type: `string`):

Archive sort by created time. newest returns recent archived items first; oldest pages forward from the start of the window. Default newest.

## `minScore` (type: `integer`):

Keep only posts/comments with Reddit score at or above this value. Leave empty for no filter. Useful to drop low-engagement noise from a RAG corpus.

## `includeComments` (type: `boolean`):

When scraping subreddit listings, keyword posts, or post IDs, also fetch archived comments for each post. Increases row count and cost. Default false for lean RAG post packs.

## `maxCommentsPerPost` (type: `integer`):

Cap archived comments fetched per post when includeComments is true. Default 50. Set higher for deep thread dumps.

## `userContent` (type: `string`):

When usernames are set, choose posts only, comments only, or both. Default both. Ignored when no usernames are provided.

## `maxItems` (type: `integer`):

Hard cap on billed dataset rows across all targets for this run. Example: 3 for an agent-quality sample or 500 for a bounded corpus. Prefill 3; maximum 50000. This controls returned records, not a date filter.

## `requestDelayMs` (type: `integer`):

Milliseconds to wait between archive HTTP requests. Default 800 keeps polite rate limits. Lower only if you accept more 429s.

## `proxyConfiguration` (type: `object`):

Apify Proxy settings. Archives often block datacenter IPs — Residential US is recommended. Default uses Apify Proxy.

## Actor input object example

```json
{
  "subreddits": [
    "MachineLearning"
  ],
  "searchQueries": [],
  "commentSearchQueries": [],
  "postIds": [],
  "usernames": [],
  "startUrls": [],
  "afterDate": "2024-01-01",
  "beforeDate": "2024-06-30",
  "sortOrder": "newest",
  "includeComments": false,
  "maxCommentsPerPost": 50,
  "userContent": "both",
  "maxItems": 3,
  "requestDelayMs": 800,
  "proxyConfiguration": {
    "useApifyProxy": true,
    "apifyProxyGroups": [
      "RESIDENTIAL"
    ]
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset rows with documentId, chunkText, scores, dates, and source URLs.

## `summary` (type: `string`):

OUTPUT record with outcome, itemsPushed, charges, and warnings.

## `runSummary` (type: `string`):

Machine-readable RUN\_SUMMARY mirroring OUTPUT for integrations.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "subreddits": [
        "MachineLearning"
    ],
    "afterDate": "2024-01-01",
    "beforeDate": "2024-06-30",
    "maxItems": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("khadinakbar/reddit-archive-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "subreddits": ["MachineLearning"],
    "afterDate": "2024-01-01",
    "beforeDate": "2024-06-30",
    "maxItems": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("khadinakbar/reddit-archive-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "subreddits": [
    "MachineLearning"
  ],
  "afterDate": "2024-01-01",
  "beforeDate": "2024-06-30",
  "maxItems": 3
}' |
apify call khadinakbar/reddit-archive-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,khadinakbar/reddit-archive-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/4N5gVlxyADb7GwCcQ/builds/XJZGPp2dJ0ln4aOF9/openapi.json