Reddit Scraper — incremental runs, posts and full comment trees

Point it at your subreddits and search queries once, put it on a schedule, and every run hands you only the posts published since the previous run. No duplicates to filter out, no history to re-download, and nothing billed twice.

Turn on Only new posts since my last run and the Actor keeps a high-water mark per subreddit and per search query in a key-value store on your own Apify account. Track 20 sources and you get 20 independent marks. Because Reddit's new listing is newest-first, the Actor also stops reading a source the moment it reaches a post you already have — a daily run on a busy community reads one page instead of twenty.

$2.50 per 1,000 rows, on every plan. No start fee. Posts and comments are the same price.

What you get

One row per post:

Switch on Also scrape comments and every reply becomes its own row, in the same columns, plus:

parentId joins straight to commentId, so the whole thread rebuilds from the flat table with one self-join — no JSON parsing, no nested objects, no ragged CSV columns. Posts and comments share one column set. The contract is exact: every parentType: "comment" row's parentId matches a commentId that appears earlier in the dataset, one depth level up.

minScore cannot break the thread, and the rows that keep it whole are free

Reddit returns downvoted comments routinely, so a score floor — including the default

minScore: 0

WHERE retainedForTree IS NOT TRUE

Comments are fetched in parallel, and the order never changes

A comment tree is one unblocking-proxy request and the unblocker answers in 40–90 seconds, so commentConcurrency (default 5) fetches several at once: 150 posts with comments takes roughly 30–35 minutes at the default, against over two hours one at a time — and about 15–18 minutes at commentConcurrency: 10. Each post's tree is buffered and written in listing order, so the dataset is identical at any setting: parents still precede their replies, and nothing is billed or counted twice. A tree that fails to load is reported in the log; its post row is still saved and the run carries on.

Author names are redacted unless you ask for them

Include author usernames is OFF by default, and every row's author, authorFullname and authorFlairText come back as null while it is off. Everything that is not a person — the title, the body, the score, the timestamp, the permalink, the subreddit — is kept, so the dataset is fully useful for market research, monitoring and trend analysis without carrying anybody's identity.

Turn it on only if you have a lawful basis for processing the identity of the people who wrote the posts. It is a single switch, and it is your decision, recorded in your run's input.

Incremental runs

The first run, spelled out

A source the Actor has never seen has no mark, so it has to start somewhere. It starts from a date, not from the beginning of the listing. By default that date is 30 days ago, so the first run on a new subreddit costs you roughly a month of posts — not everything the listing will give. You decide how much history you pay for:

Every later run ignores firstRunSince entirely and uses the mark. The run log names which of the two happened for each source, so there is never any doubt about what you were billed for.

What the marks are, and where they live

One record per source, holding the timestamp and id of the newest post you have received, the source's own identity, and a run counter. They are kept in a named key-value store, which Apify retains indefinitely on every plan, so the marks survive across runs, schedules, plan changes and Actor updates.

• They are yours. The store is created on the account that runs the Actor, alongside your datasets. You can open it, read it, edit a mark to re-pull a period, or delete it to start over.
• Keyed per source. A subreddit and a query are separate sources, and a query is keyed by its own sort and time range too — so "top of the week for X" and "new posts about X" never share a mark.
• Namespaced. Keys carry a namespace taken from your own Apify user id, and a record whose namespace does not match is ignored on read, so marks can never be crossed between accounts.
• Give different tracking sets different store names — reddit-state-competitors and reddit-state-leads — and the same subreddit is tracked independently in each.
• Cheap. One read and one write per source per run.

Where it saves you money

Reddit's new listing is newest-first, so once the Actor reaches a post older than your mark it knows everything beyond it is already in your dataset and stops reading that source immediately. Both things a run pays for fall: the rows you are billed for, and the proxy requests behind them.

Modelled on a community gaining 50 posts a day, run daily with maxPostsPerSource: 500:

That is 10x fewer billed rows and 5x fewer proxy requests per run. Across a 30-day daily schedule, including the two seeding runs at the start, it is 2,100 rows against 15,000 — 7x less, or $5.25 against $37.50. npm run measure reproduces those numbers locally from the Actor's own code, with no network.

The quieter the community and the more often you run, the bigger the gap: the stateless bill is always your cap, while the incremental bill is only what was actually posted.

Safety rails

• A source with nothing new returns zero rows and is billed nothing. No empty rows, no minimum charge.
• Two consecutive runs cannot double-count, not even a post sitting exactly on the boundary second: the mark remembers recent post ids as well as the timestamp, and Reddit timestamps are whole seconds, so that collision is common rather than theoretical.
• A run cut short still moves the mark, safely. If maxItems, maxPostsPerSource or a charge limit stops a run part-way, the mark advances to the newest post that run actually billed you for — listings are newest-first, so that post is trustworthy even though older ones on the same run were not reached. Either way, you are never billed twice: seenPostIds remembers what was already charged independently of the mark, so a re-read of the same ground never re-bills it.
• A run cut short DOES leave a permanent gap, and the run log and the Actor's OUTPUT say so. If a source hits its cap before reaching the mark (or, on a first run, before reaching firstRunSince), the posts older than where it stopped are never returned by a later run — later runs only look forward from the mark, they never go back to fill in what an earlier capped run skipped. Each source's line in the log names it plainly:

  "reached back to <date>, requested back to <date>; posts older than that will not be returned by later runs - raise 'maxPostsPerSource' ... for a complete first pull."
  The same fact is on the mark record itself (windowCovered: false, oldestReachedIso) and in the run's OUTPUT under coverage, so you can check it without reading logs. If you want a source's full history, raise maxPostsPerSource (or maxItems) so the first run is never capped, or run once with firstRunSince: "all".
• A pinned post never ends a run early. Communities pin old announcement threads to the top of every listing; the early stop skips them rather than concluding the source is up to date.
• Incremental mode needs newest-first order, and says so in the log if you asked for another.
• Leave at least a minute between incremental runs on the same source. The marks live in Apify's key-value store, and a run started only seconds after the previous one finished can read a not-yet-visible mark and reseed from scratch — the log says

  "mark record found but not yet set"
  when this happens, so it reads as a timing note rather than a mystery.

A source that returns nothing does not pass quietly

A misspelt subreddit name and a missing proxy both look like "no results", and neither should finish SUCCEEDED with an empty dataset. If a source returns no posts at all and nothing explains it — no date window, no score floor, no mark, no cap — the run fails and names the source, with the likely cause. Rows already saved are kept: they are real, and the failure is about the ones that are missing. Set failOnEmpty: false if an empty result is acceptable to you.

What it needs, and what it never does

This Actor needs a plan with Unblocker proxy access. Reddit answers a plain datacenter IP with HTTP 403 and an HTML block page, so the proxy group is the reason the Actor returns anything at all. Unblocker access is included on every plan, the free one included, and the input's proxy field is already set to it — there is nothing for you to configure. Each successful request costs about 10 proxy units, and one request returns up to 100 posts.

Requests carry a 90-second timeout by default. That is deliberate and measured: the unblocking proxy does real work per request and a shorter timeout aborts requests that were about to succeed. Raise it with requestTimeoutSecs if you see timeouts; the minimum accepted is 45 seconds.

It reads only the public JSON views of pages anybody can open in a browser — a subreddit listing, a search results page, and a post's comment tree. It never logs in, never sends a cookie from a signed-in session, and uses no Reddit API key and no OAuth token. It reads no private community, no direct message and no user inbox.

Reddit's own terms restrict automated collection of its content, and commercial use is gated on an agreement with Reddit. Satisfying yourself that your use is permitted is your responsibility as the person running the Actor.

Input at a glance

Billing

Pay per event, charged as each row is saved, with no start fee and no charge for platform usage:

Set maxItems as your cost cap, or a maximum total charge on the run. When a charge limit is reached the run stops cleanly, keeps everything already saved, and holds the incremental marks where they were so nothing is re-billed on the next run.