Reddit VOC Collector

An auditable Apify Actor for two separate Reddit research stages using the public Arctic Shift archive:

1. discover searches selected public subreddits and writes deduplicated candidate posts.
2. collect accepts a separately reviewed post list and writes each returned post plus its public comment records.

The Actor intentionally has no combined mode. A discovery run never starts full comment collection, and its candidates are not automatically treated as approved research evidence.

Scope and evidence boundary

• Collects only records that Arctic Shift marks as belonging to a public subreddit. Quarantined or mismatched records are rejected.
• Does not authenticate to Reddit, access private communities, bypass controls, or reconstruct [deleted] / [removed] text.
• Does not infer residence, nationality, language, or demographics from subreddit membership or flair.
• Arctic Shift is a third-party archive. Its availability, freshness, and coverage can differ from Reddit. A successful run proves what the endpoint returned at retrieval time, not a complete history of Reddit.
• Discovery scores are diagnostics for analyst review. They are not relevance labels, population estimates, or market shares.

Quick start

1. Discover candidates

{
  "mode": "discover",
  "subreddits": ["CampingGear", "overlanding"],
  "queries": ["portable fridge", "12v refrigerator"],
  "includeTerms": ["fridge", "refrigerator"],
  "excludeTerms": ["minecraft"],
  "startDate": "2024-01-01",
  "endDate": "2026-08-31",
  "pageLimit": 50,
  "maxPagesPerQuery": 2,
  "maxCandidates": 200
}

Review the candidate Dataset outside the Actor. Keep the post ID, subreddit, all matched_queries, and a human-written reason for each selected post.

2. Collect reviewed threads

{
  "mode": "collect",
  "selectedPosts": [
    {
      "id": "abc123",
      "subreddit": "CampingGear",
      "reason": "Detailed first-person power and cooling setup",
      "matchedQueries": ["portable fridge", "12v refrigerator"]
    }
  ],
  "commentLimit": 5000
}

Duplicate selected IDs are merged before collection. Their matched queries are preserved and deduplicated. If duplicate entries assign one post ID to different subreddits, validation fails.

Input rules

The Actor validates input again at runtime instead of relying only on the Console form.

Mode	Required	Forbidden from the other mode
discover	subreddits, queries	selectedPosts, commentLimit
collect	selectedPosts	discovery subreddits, queries, terms, dates, and page limits

Hard bounds include 25 unique subreddits, 25 unique queries, 20 pages per pair, 1,000 potential discovery requests, 5,000 unique candidates, 100 selected posts, 10,000 comments per post, and 100,000 potential collect-mode Dataset records (unique selected posts x (commentLimit + 1)). Unknown input fields, control characters, invalid dates, invalid Reddit IDs, and invalid subreddit names fail before a request starts.

requestSettings permits only bounded delay, timeout, retry, and backoff values. The provider origin is fixed in source to https://arctic-shift.photon-reddit.com; input cannot supply a URL.

Default Dataset

Only successful customer-facing records are stored in the default Dataset:

• candidate rows in discover mode;
• post and comment rows in collect mode.

Every record has a stable dataset_record_id in addition to its source IDs, canonical HTTPS Reddit URL, UTC retrieval time, source, and matched queries. Comment rows also preserve parent_id, calculated depth, and a root-to-comment thread_path. Removed/deleted bodies are stored as null with content_status, not reconstructed. Stable record IDs support downstream reconciliation, but Apify Dataset writes are not advertised as exactly-once.

Post rows expose source-fetch completeness both as compact fields and a completeness object. comments_complete describes the comment tree returned by the source, not Dataset write confirmation. It is conservative: a count gap or surplus, rejected node, unresolved more node, orphan/cycle, reached limit, or missing expected count prevents comments_complete=true. Actual confirmed Dataset rows, charge-limit skips, and any indeterminate failed batch are recorded in POST_AUDITS and RUN_AUDIT.

No page, retry, failure, raw-response, or audit row is written to the default Dataset.

Key-value store audit records

Key	Mode	Contents
RUN_AUDIT	both	Final status, counts, sanitized errors, and stage summary
DISCOVERY_AUDIT	discover	Per-page URL/attempt/count status, caps, errors, and coverage limitations
POST_AUDITS	collect	Per-post request status, raw keys, comment completeness, and failures
CHECKPOINT	both	Completed discovery pages or processed/complete/incomplete post IDs
RAW_DISCOVERY_*_MANIFEST	discover	Manifest for one source-shaped search-page JSON response
RAW_POST_*_MANIFEST	collect	Manifest for one source-shaped post-lookup JSON response
RAW_COMMENTS_*_MANIFEST	collect	Manifest for one source-shaped comment-tree JSON response
RAW_*_NNNN.json.gz	both	Independently gzip-compressed raw JSON byte chunk

Raw KVS artifacts are created only after public-scope and requested-subreddit/post checks. Private, quarantined, and mismatched records are excluded before storage. Each manifest and chunk key includes a hash of the sanitized content, so reusing a KVS cannot silently overwrite an older raw artifact referenced by a Dataset row. Successful response JSON is preserved in source-shaped form before normalization, except that deleted/removed body, selftext, related HTML fields, and deleted author markers are set to null; _voc_redacted_fields and the manifest redaction count make this explicit. The sanitized UTF-8 JSON bytes are split into at most 4 MiB per chunk, then every chunk is gzip-compressed and accompanied by byte counts and SHA-256 hashes in an ordered manifest. This keeps large comment trees below a single-record assumption while preserving byte-exact reconstruction of the compliant raw artifact. HTTP response bodies also have a fixed 128 MiB safety limit. Errors omit headers and response bodies. RUN_AUDIT.status=partial means at least one request failed, source completeness was not established, or the Dataset output stopped at a charge limit; if every requested discovery call or selected post fails, the Actor exits non-zero.

The checkpoint is written after every discovery page and selected post and is bound to a deterministic normalized-input fingerprint. It is durable run evidence, but a new Actor run uses new default storage unless the caller explicitly reuses storage; it is not advertised as automatic cross-run resume.

Output and charging boundary

The source does not define or mutate Store pricing. If default Dataset item pricing is configured later, each successful candidate/post/comment row becomes a billable Dataset result. Audit and raw records remain in KVS and must not be priced as results. Dataset rows are written serially in bounded batches. Before every metered batch, the Actor checks the SDK's remaining default-result charge capacity and never calls pushData when that capacity is zero. It uses the returned chargedCount as the confirmed prefix; a failed batch is explicitly listed as indeterminate because an HTTP acknowledgement loss cannot prove that the batch was not stored. Verify live pricing separately before publication or any non-trivial run.

Local verification

npm install
npm test
npm run check

Tests use local mocked HTTP responses and do not call Reddit or Arctic Shift.

SOURCE_FILES deployment

create_actor_payload.mjs produces private-by-default Apify create and version payloads without reading any credential:

$node create_actor_payload.mjs

It writes /tmp/reddit_voc_actor_create.json and /tmp/reddit_voc_actor_version.json. The create payload sets isPublic: false, source type SOURCE_FILES, version 0.0, build tag latest, 1 GB memory, and a one-hour timeout. Upload credentials belong only in an Authorization header used by a separately reviewed deployment helper; never add them to these payloads, Actor input, source, or logs.