Reddit Scraper - All In One

The Actor performs any one of nine Reddit reads, chosen at run time from a dropdown: a subreddit's post feed, the popular feed, a keyword search, an account's activity, an account's comment history, a post's comment tree, one post's full record, one community's record, and one account's profile. Each mode fills in its own section of the form and writes to the same dataset. No Reddit account, application, or API credential is involved.

Each of the nine is also published as a focused Actor with the same behaviour and a form containing only its own fields. This one exists for work that switches between modes, or that is driven from a script where one Actor and one dataset is simpler than nine.

Choosing a mode

scraperType is the only required field. Every other field belongs to one of the sections below, and a mode reads only the fields of the sections that concern it; the rest are ignored, so leaving them empty is correct rather than merely tolerated. Each mode raises its own error naming the identifier it needs, so a missing username on a profile run says so.

scraperType	What it saves	Fields it reads
subredditPosts	Posts of one community, in a chosen order	subreddit, sort, time, maxItems, after
popularPosts	Posts of the cross-community popular listing	maxItems, after
search	Results for a keyword, in one of five corpora	q, searchType, maxItems, cursor
userPosts	One account's posts and comments together	username, include, maxItems, after
userComments	One account's comments	username, maxItems, after
postComments	One post's comment tree, flat, with depth and parent	post_id, subreddit, commentSort, maxItems
postDetails	One post's full record	post_id, subreddit
subredditInfo	One community's record and rules	subreddit
userProfile	One account's profile	username

{
  "scraperType": "subredditPosts",
  "subreddit": "r/dataisbeautiful",
  "sort": "top",
  "time": "MONTH",
  "maxItems": 50
}

{
  "scraperType": "postComments",
  "post_id": "https://www.reddit.com/r/technology/comments/1vqcqpz/example_title/",
  "commentSort": "top",
  "maxItems": 200
}

The sections of the form

• Subreddit — subreddit, sort, time. The sort order and the time window apply to the subreddit feed only; subredditInfo, postComments, and postDetails read the community name and nothing else from this section.
• Post — post_id, commentSort. Pasting a post's link supplies the community as well, so the Subreddit field above can stay empty for the two post modes.
• Account — username, include. include is read by userPosts alone.
• Search — q, searchType, cursor.
• Limits and resuming — maxItems, after. The four single-record modes save one row each and read neither.

What lands in the dataset

The dataset holds the union of every mode's columns, and a run fills in the ones its mode returns. Rows from the mixed and search modes carry kind, which states whether the row is a post, a comment, a community, or an account; the modes that return a single shape omit it.

Post rows carry id, fullname, permalink, url, title, subreddit, subreddit_prefixed, author, created_at, score, upvote_ratio, comment_count, post_type, flair, media, text, and nsfw, among others. Comment rows carry body, depth, parent_fullname, position, and is_deleted. Community rows carry name, description, weekly_active_users, and rules. Account rows carry karma, post_karma, comment_karma, and contributions. The full list per mode is in the focused Actor for that mode.

Behaviour worth knowing before choosing a mode

Each mode carries the behaviour of the source it reads, and four of those behaviours change what a result means:

• popularPosts samples as much as it pages. Two reads a moment apart return overlapping but not identical sets, so the run counts distinct posts and maxItems bounds posts saved rather than rows fetched. There is no fixed order to return to, and a second run will not reproduce the first.
• userPosts reads one interleaved feed. Reddit publishes an account's posts and comments as a single stream with no filter of its own, so include: "posts" is applied after the feed has been read. An active account runs roughly one post to ten comments, so a filtered run reads a lot of feed per saved row and can finish below maxItems while the account still has more. For comments specifically, userComments reads a dedicated feed and does not behave this way.
• postComments returns part of a tree. Reddit serves a comment tree in branches, and a run saves the branches it reached. Every comment row carries tree_complete, unfollowed_branches, truncated_threads, and Reddit's own total_comments, so a partial result cannot read as a complete one. unfollowed_branches is what a larger maxItems would reach; truncated_threads counts sub-threads that need their own run.
• search is one request and publishes no continuation. Reddit ranks search results rather than paging them by id, so maxItems is the only control on depth and its ceiling is 200. The people corpus in particular returns one short page and then ends; a small result there is the corpus, not a shortfall.

The subreddit feed's six sort orders were each checked against the same community and returned genuinely different sets rather than one set reordered, so sort is worth setting deliberately. time is read by top and controversial only, and steers Reddit's ranking rather than acting as a date filter — a run sorted by top with time set to WEEK returned posts considerably older than seven days. Where an exact date range matters, filter the saved rows on created_at.

The nsfw column

nsfw has three states across every mode. true and false are Reddit's report; null means the source said nothing either way, and is an absence of information rather than a statement that the record is safe for work. On most post types the mark is simply not stated, so null is the common case. No NSFW filter is offered on this Actor or on any of the nine, because filtering on a field that is frequently unknown produces a result set that looks filtered without being filtered. Rows can be filtered downstream on nsfw === true, provided null is treated as unknown rather than as safe.

Failures and partial results

An identifier that cannot be understood is rejected before any request is made, with a message naming what was received. A request that fails ends the run; rows already saved remain in the dataset, and the modes that page write their continuation point into the run log so a failed long run is continued rather than restarted. A lookup that finds nothing ends the run cleanly with an empty dataset rather than failing. Fields absent from the source are returned as null rather than omitted, so the column set is the same in every row of a run.

Cost

Charging is per row saved to the dataset, plus the platform's one-off Actor start event, at one rate regardless of mode. The four single-record modes therefore cost less here than on their focused Actors, where the rate reflects a whole run's work going into one row; the feed modes cost slightly more here than on theirs. A run that saves nothing is charged for nothing beyond the start event.

Related Actors

The nine focused Actors, each with the same behaviour and only its own fields on the form: Reddit Subreddit Feed Scraper, Reddit Popular Feed Scraper, Reddit Keyword Search Scraper, Reddit User Activity Scraper, Reddit User Comment History Scraper, Reddit Comment Tree Scraper, Reddit Post Details Scraper, Reddit Subreddit Details Scraper, and Reddit Account Profile Scraper.

Support

Questions, bug reports, and requests for a field that is not here yet: thenetaji@proton.me.