Reddit Scraper — Posts, Comment Threads & User History

Export archived Reddit posts, comment threads, scoped keyword matches and public user activity as JSON, CSV, Excel or other Apify dataset formats. No Reddit API key, login or proxy setup is required.

The source is Arctic Shift, not a live Reddit session. Recent records can lag, deleted material may be absent, and engagement/profile values describe archived snapshots. An exhausted query does not establish complete Reddit coverage.

Version 0.3 adds familiar input and output names used by harshmaur/reddit-scraper, with shared result caps, verified comment ancestry, field selection, provenance and interrupted-write recovery. These names ease migration; supported capabilities and differences are documented below.

Price

$1 per 1,000 saved results: $0.001 per row, plus a small memory-based start charge. A post and three comments are four results ($0.004 in result charges). Requested community and user metadata rows count at the same rate. Parent records fetched only to enrich a comment do not create extra result rows.

Monetization is active. The apify-actor-start event costs $0.00005 per GB of allocated memory, with at least one event. At the default 512 MB, that is $0.00005 per run. Empty exports and dry runs can still incur the start charge. The Actor remains private until its owner requests publication.

Apify automatically charges the apify-default-dataset-item event; the Actor does not manually charge those rows again. Filtered candidates and duplicate candidates within a matching request are not stored. Separate runs or intentionally new requests can return the same records again.

maxTotalItems is a hard shared row ceiling. maxTotalChargeUsd adds a result ceiling rounded down at $0.001 per row: $0.019 permits at most 19 results. A zero shared row or result-charge budget performs no source requests. Apify's platform spending limit is a separate run setting. Private owner development runs can incur platform usage. Consult Console for actual billing and the effective pricing record.

Quick start

The Console's prefilled sample exports three known historical posts using direct ID lookups. It is also the input used for Apify's daily automated Store test. The sample has a three-row cap, three logical source requests, a 10-second HTTP timeout and one retry per request. If one record is unavailable, it continues to the other real records. Change the example URLs and limits for your own exports.

The sample is defined in .actor/default_input.json. Its values are schema prefill entries, so they do not inject demonstration targets into API requests for your own subreddit, keyword or profile. The saved Actor example input uses the same sample. Empty API requests, including requests containing only unchanged schema defaults, resolve to this bounded sample and report defaultSampleApplied: true. Explicit targets, filters, custom controls and invalid inputs keep their usual behavior. Archive/provider outages can still prevent a non-empty result; successful tests cannot guarantee future uptime.

Export up to 20 archived posts from a known historical date:

{
  "subredditUrls": ["Python"],
  "postedAfter": "2024-01-01",
  "postedBefore": "2024-01-01",
  "maxPostsCount": 20,
  "maxTotalItems": 20,
  "outputProfile": "compact"
}

Date-only upper bounds include the whole UTC day; this example ends at January 2, 00:00 UTC. Listings without an explicit lower bound scan the last 30 days. Availability still depends on the archive.

After running, open Results and export your preferred format. The output tab links to the JSON summary, HTML coverage report and discovery provenance.

A post and its archived thread

{
  "startUrls": [{"url": "https://www.reddit.com/r/Python/comments/18vkgtu/"}],
  "crawlCommentsPerPost": true,
  "maxPostsCount": 1,
  "maxCommentsPerPost": 20,
  "maxTotalItems": 21,
  "includeParentData": true
}

Threads are chronological. Comments have joinable postId and parentId fields. depth and rootCommentId are populated only when ancestry is verified; gaps, cycles or lookup bounds yield missingParent: true and depth: null. Available replies can differ from the post's archived commentsCount.

A direct comment URL (/r/community/comments/POST/title/COMMENT/) saves that comment alone. Parent posts and ancestors can enrich it without becoming additional results. Direct URLs have no implicit 30-day lower bound; explicit date filters still apply. maxPostsCount: 0 can suppress a direct post row while allowing its thread.

Scoped keyword matching

{
  "searchTerms": ["python automation", "\"data pipeline\""],
  "withinCommunity": "Python",
  "postedAfter": "2024-01-01",
  "postedBefore": "2024-01-07",
  "searchPosts": true,
  "searchComments": true,
  "maxPostsCount": 10,
  "maxCommentsCount": 10,
  "maxTotalItems": 20
}

Every token in one search phrase must occur case-insensitively in the post title/body or comment body. Quoted phrases stay together. This is literal matching, without Reddit Boolean operators. Each term gets its own scoped search. withinCommunity or a subreddit/user target is required; global Reddit discovery is unavailable.

If upstream rejects scoped text filtering, the Actor scans the same target/date range and verifies matches locally within maxRequests. Coverage can be partial. Explicit URL/listing sources and keyword sources are additive and share a deduplicated allowance; a broad listing can consume it first. Use only searchTerms with withinCommunity for a pure keyword request.

User activity

{
  "startUrls": ["https://www.reddit.com/user/spez/"],
  "postedAfter": "2024-01-01",
  "postedBefore": "2024-01-31",
  "commentedAfter": "2024-01-01",
  "commentedBefore": "2024-01-31",
  "includeUserComments": true,
  "maxPostsCount": 10,
  "maxCommentsCount": 10,
  "maxTotalItems": 20
}

/user/NAME/comments/ restricts activity to comments; /user/NAME/submitted/ restricts it to posts. General user targets include comment history by default. includeUserProfiles adds a separate archived user metadata row when available.

Inputs

The .actor/input_schema.json defines Console controls. startUrls accepts strings or { "url": "…" } objects with GET URLs. subredditUrls accepts names, r/NAME, or subreddit URLs. Short redd.it links, canonical posts/comments and chronological scoped search URLs are supported. Other hosts, credentials in URLs and unsupported listing paths fail validation.

Input	Default	Meaning
startUrls, subredditUrls	empty	Direct records, community listings or user activity; 100 entries each
searchTerms, withinCommunity	empty	Scoped literal keyword matching
searchPosts	true	Collect listing/search posts
searchComments	false	Collect comments independently from scopes
crawlCommentsPerPost	false	Collect threads for selected posts
includeCommunity, includeUserProfiles	false	Save available archived metadata rows
includeUserComments	true	Comment activity for general user targets
postedAfter, postedBefore	last 30 days / run start for listings	Post date bounds
commentedAfter, commentedBefore	last 30 days / run start for listings	Comment bounds; threads default to parent post creation
searchSort	new	Chronological new or old; threads always oldest first
searchTime	all	Optional hour, day, week, month, year lower bound without explicit dates
includeNSFW	false	Exclude posts explicitly marked over_18; unknown stays unknown
onlyWithFlair	false	Require a post flair
minScore	unset	Minimum archived score; missing scores fail this filter
includeKeywords, excludeKeywords	empty	Require every included phrase; reject any excluded phrase
maxPostsCount	50	Saved posts per source across resumptions
maxCommentsCount	100	Saved independent comments per source
maxCommentsPerPost	100	Saved comments per thread
maxCommunitiesCount	2	Request-wide community metadata ceiling
maxTotalItems	100	Shared ceiling across all record families
maxTotalChargeUsd	unset	Additional row ceiling at $0.001 per result
maxRequests	200	Logical source requests including context; HTTP retries add attempts
includeParentData, maxParentDepth	true, 20	Cached, bounded comment context lookups
outputProfile	standard	compact, standard, full
includeRawData	profile-dependent	False for compact/standard; true for full unless overridden
outputFields	unset	Select known fields; identity and recovery provenance remain
resumeFromState	true	Resume a matching request in the same storage
dryRun	false	Validate/preview without source HTTP, dataset writes or checkpoint changes
skipInvalidTargets	false	Explicitly skip invalid targets with diagnostics
continueOnError	false	Continue other sources after errors; unresolved writes always stop

Dates accept YYYY-MM-DD, ISO datetimes and relative periods such as 7 days, 1 hour or 1 week ago. Naive datetimes use UTC. Relative dates and the default upper bound freeze when a request is created. all removes a lower bound. Relative months/years mean 30/365 days.

Zero disables the corresponding result family. Filters apply before stored-row caps. Sources run in input order, so the first source can consume a shared cap. searchComments and crawlCommentsPerPost are independent.

Advanced controls: limit (1–100 or auto) sets archive page size, windowDays defaults to 1 (0 scans the bounded range), minWindowSeconds to 60, timeoutSeconds to 30, retries to 3, retrySleepSeconds to 2 and requestDelayMs to 0. maxPagesPerKind adds a partial-scan ceiling. verboseLog enables diagnostics. baseUrl overrides the archive endpoint. Transient source timeouts/rate limits are retried with bounded backoff; oversized ranges split adaptively and grow again after successful empty scans. Saturated timestamp boundaries stop with preserved partial data instead of skipping unseen records.

Output contract

dataType is post, comment, community or user_profile. Fields describe the applicable family; missing source-dependent values remain null. Standard output uses familiar Reddit keys plus explicit recovery/provenance fields.

Group	Main keys
Identity	dataType, id, parsedId, recordKey, url
Content	title, body, bodyHtml, createdAt, edited, editedAt
Author/community	authorName, authorId, parsedAuthorId, communityName, communityId, parsedCommunityName
Engagement	score, upVotes, commentsCount, upvoteRatio, scoreHidden, totalAwardsReceived
Post state	over18, isSelf, isVideo, isGallery, spoiler, locked, archived, flair, removalStatus
Links/media	postUrl, contentUrl, domain, images, videoUrl, thumbnail, mediaAssets, galleryImages
Relationships	postId, parsedPostId, postTitle, parentId, parsedParentId, parentKind, rootCommentId, depth, missingParent, isSubmitter
Provenance	source, sourceRetrievedAt, crawledAt, sourceTarget, searchTerm
Metadata	Archived names, descriptions, creation times, available flags, karma/member counts and archive activity fields

id is a Reddit fullname (t3_POST, t1_COMMENT, t2_USER, t5_COMMUNITY). parsedId omits the prefix. postId and parentId also use fullnames for joins. Unknown IDs remain null; recordKey uses a stable name/hash fallback instead of inventing a Reddit ID.

communityName includes r/; parsedCommunityName is bare. url/postUrl are Reddit permalinks; contentUrl is an external destination. Media normalization preserves gallery order, decodes HTML entities and omits unsafe asset URLs.

upVotes is a compatibility alias for archived net score, not positive vote count. sourceRetrievedAt is the upstream observation time when available; crawledAt is when this Actor prepared the row. Age-adjusted metrics use source observation time and remain null when unknown. removalStatus: "unknown" does not assert a post is active. Archived metadata does not establish current community/profile state.

compact retains main text, IDs, relationships and provenance. standard adds normalized details. full also retains rawData by default. outputFields reduces any profile; identity/source fields, sourceTarget and comment missingParent remain for recovery. Selected fields for a different record family can be absent. Raw payloads increase export size.

The complete dictionary is in the .actor/dataset_schema.json. Console offers overview, threads, provenance and legacy views. The schema covers both output contracts; an individual row does not contain both.

Summaries, coverage and recovery

The default key-value store contains:

• OUTPUT and RUN-SUMMARY: JSON status, unique/new/recovered row counts, record families, filter/duplicate counts, unresolved ancestry, request budget and per-scan stop reasons.
• RUN-MAP.html: self-contained coverage report.
• PROVENANCE: recordKey → discovered source labels for this request. Thread rows identify their parent post URL.
• REQUEST, STATE-*, PENDING-*: internal request/checkpoint/write journals. Keep them with the original dataset for recovery.

Statuses are completed, empty, partial, dry_run or failed. A successful Actor process can have a partial export because a cap was reached. Inspect coverageComplete, truncated, scans[].stoppedReason and pendingWrite. Stops at item/page/request caps do not establish complete traversal. Context failures preserve rows with nulls and warnings.

itemsTotal is the unique saved count for this request; totalRows is newly saved this invocation. An intentionally reused dataset can also contain earlier requests. estimatedResultCostUsd estimates request result rows; Console reports actual charges.

Resuming a matching request in the same key-value store and dataset freezes dates, reconciles uncertain appends, replays unfinished pages and skips saved records. Raising row/request caps can continue it. Changed targets, dates, filters or output shape, or resumeFromState: false, creates a new request that can append records seen previously.

Only one writer may use this storage. Unexpected dataset changes or unresolved writes stop the run. Recover with the original input/storage before switching requests. New scheduled runs normally use fresh storage and do not share deduplication. Use explicit dates and downstream recordKey deduplication for recurring exports.

Migration and supported scope

Use startUrls/subredditUrls and version 2 for new integrations. Legacy targets alone keeps version 1; explicitly set outputVersion: 1 to pin an existing integration. Version 2 also accepts targets, datasets, startDate, endDate and maxItemsPerKind as aliases. Standard cap/date fields take precedence; datasets explicitly selects the legacy record families when present. Recover old pending writes before changing contracts; prefer fresh storage for upgrading.

Original key	Standard key
kind (post / comment)	dataType (post / comment)
redditId	parsedId and fullname id
description	body
author	authorName
subreddit	parsedCommunityName
commentCount	commentsCount
retrievedAt	crawledAt; upstream capture is separate sourceRetrievedAt

Live hot/top/relevance, global keyword discovery, community discovery, private/login-only data, current profile guarantees, AI sentiment/intent labels and external MCP delivery are unsupported. Enabling placeholders such as aiAnalysis, customLabels, searchCommunities, intent flags or mcpConnector fails before result writes. Unknown input keys also fail. Version 1 retains its original output and summary format.

API and development

Use Console or the authenticated Actor API. Store your Apify account token in an environment variable, not the Actor input:

curl --request POST \
  --header "Authorization: Bearer $APIFY_TOKEN" \
  --header 'Content-Type: application/json' \
  --data '{"subredditUrls":["Python"],"postedAfter":"2024-01-01","postedBefore":"2024-01-01","maxTotalItems":5}' \
  'https://api.apify.com/v2/acts/aster_data~reddit-mcp-actor/runs'

The response provides run and storage IDs. Poll the run and retrieve dataset items or summary artifacts through the authenticated API. The Actor stays private until its owner chooses to publish it.

Python 3.12+ is supported. Runtime and development dependencies are separate:

python3 -m venv .venv
.venv/bin/pip install -r requirements-dev.txt
.venv/bin/python scripts/generate_schemas.py
.venv/bin/python -m coverage run --branch --source=src -m unittest discover -s tests -q
.venv/bin/python -m coverage report -m
npm exec --package apify-cli@1.10.0 -- apify validate-schema

Tests cover input rejection, dates, output types, media, parent chains, filtering, deduplication, shared budgets, both pagination directions, timestamp saturation, adaptive splitting, HTTP failures, partial/ambiguous appends and restart recovery with real local SDK storage. Generated cases vary record distributions and malformed JSON fields. Cloud checks use bounded historical inputs and validate actual output. See the docs/releases/0.3.4/README.md and docs/tests/scale-2026-10-22/README.md. No finite suite guarantees every upstream condition.

Deploy with apify push. Apply .actor/pricing.json separately through the Actor API and read it back. Ordinary pushes must preserve private visibility. The dated docs/research/reddit-actors-2026-10-21/README.md informed this release; field counts do not establish superior completeness or speed.