Substack Scraper — Posts, Comments & Newsletters

Substack as clean rows: complete post archives, full comment threads with replies, topic-based publication discovery, and the recommendation graph between newsletters.

No API key. No login. Paste a name, press start, export CSV or JSON.

---

Three things this does that other Substack scrapers don't

1. It doesn't silently skip posts

Substack's archive endpoint accepts limit up to 50 — ask for 51 and you get a hard HTTP 400, so 50 looks like the safe number. But at the first page it returns only 23 rows, with HTTP 200 and no hint it short-changed you. Page by offset += 50 and you jump straight from row 23 to row 50, never seeing the posts in between.

Approach	Posts retrieved
Page by the limit you asked for	346
Page by the rows actually returned	373 — complete, no duplicates, 9 requests

That's 7.2% of the archive missing and nothing anywhere saying so. Reproduced identically on Big Technology, Astral Codex Ten and The Pragmatic Engineer.

The same trick appears again under load: Substack throttles by returning HTTP 200 with an empty array rather than a 429, which reads exactly like "end of archive". This Actor confirms every unexpected empty response with backoff before believing it.

2. Its comment threads are actually complete

The comments endpoint returns top-level comments only — replies are nested inside them, recursively, several levels deep. Read the array you get back and stop, and a thread of 878 comments gives you 65 rows.

This Actor flattens the tree: 881 rows, each with its depth, parent and reply count, nesting up to 20 levels deep. Watch out for child_comment_count too — it counts top-level comments, not replies, which invites precisely the wrong assumption.

3. It respects the paywall — on purpose

Substack's per-post endpoint hands over the complete text of subscriber-only posts. Measured: a post marked only_paid came back with 1,116 words against a stated word count of 1,121. The whole article, free.

This Actor will not give you that. Paywalled posts return their metadata, engagement numbers and the public teaser Substack itself displays, with paywalled: true and bodyStatus: "withheld_paywalled". Free posts return their full text, because that is public.

That's a deliberate boundary. The paid body is the writer's livelihood, republished paywalled text isn't something you could safely use commercially anyway, and a scraper that harvests it is one complaint away from disappearing. You get every signal about paid posts — what they cover, how they perform, when they run — without the liability.

---

What you get

Posts

Column	Example
title / subtitle	Why AI Margins Are Collapsing
publication / publicationId	bigtechnology
url	canonical post link
audience / paywalled	only_paid · true
publishedAt	ISO 8601
authors	name, handle, bio, id
wordCount / reactionCount / commentCount / restacks	1121 · 56 · 9 · 4
tags / language / postType	["ai"] · en · newsletter
podcastUrl / podcastDuration / hasAudio	for audio posts
teaser	the public preview, always
bodyHtml / bodyText	free posts only
bodyStatus	full · withheld_paywalled · not_requested

Comments — body, authorName, authorHandle, depth, isReply, parentId, replyCount, reactionCount, createdAt, editedAt.

Publications — name, subdomain, customDomain, url, tagline, logoUrl, paymentsEnabled, communityEnabled, inviteOnly, createdAt, recommendedBy.

---

Modes

Mode	What it returns
Posts from a publication	The archive, swept completely
Search posts	Substack's own search, inside a publication
Discover publications	Newsletters matching a topic — when you don't know the URLs
Recommended publications	Who a newsletter recommends
Comments	Whole threads, replies flattened with depth and parent

---

Typical uses

• Competitive research. Every post a rival newsletter published, how often, how it performed, and what sits behind the paywall.
• Finding newsletters at all. Discovery takes a topic and returns publications — most scrapers need you to already know the URL, which is useless if the question is "who writes about climate tech".
• Newsletter growth and partnerships. The recommendation graph shows who recommends whom, which is the main discovery mechanism on Substack.
• Audience research. Complete comment threads with reply structure, not just the top 7%.
• Content and trend analysis. Word counts, tags, cadence and engagement across a whole archive.
• Feeding an LLM or agent. One row per item, consistent field names, HTML stripped and entities decoded.

---

Example input

A publication's best recent posts, with full text where it's free:

{
  "mode": "posts",
  "publications": ["bigtechnology"],
  "includeBody": true,
  "publishedWithinDays": 90,
  "minReactions": 20,
  "sortBy": "reactions",
  "maxItems": 200
}

Find publications in a topic:

{ "mode": "discover", "discoverQuery": "technology", "maxItems": 100 }

Walk the recommendation graph out from one newsletter:

{ "mode": "recommendations", "publications": ["bigtechnology"], "maxItems": 200 }

---

Settings that matter

• includeBody — off by default because it makes rows much larger. Note bodyStatus: paywalled posts come back withheld_paywalled with the public teaser, never the paid text. That's deliberate, not a gap.
• freeOnly / paidOnly — if you're building a text corpus, freeOnly is what you want, since paid posts return a teaser only and would just be empty rows you paid for.
• minReactions / minComments / minWords — applied before billing. minWords is the quickest way to drop one-line link posts from a writing analysis.
• publishedWithinDays — most archives are long. This is usually a bigger lever on cost than maxItems.
• mode: discover vs recommendations — discover searches by topic; recommendations walks the graph of who recommends whom, which surfaces newsletters that search doesn't rank.
• maxItems — your cost ceiling. Set it deliberately.

---

Reliability

Everything comes from Substack's own public JSON API. There is no HTML parsing and no bot-detection workaround anywhere in this Actor, which is why it doesn't quietly break when a page layout changes.

An 82-assertion live test suite runs against the real API before every release — including assertions that the paging gap still exists, that a paid body is never emitted even when the API offers it, and that a flattened thread matches its advertised comment count.

Publications that have left Substack while keeping their domain return 404 on every API route. This Actor says so plainly instead of returning an empty result you'd have to debug.

---

Pricing

Pay per event. You are charged for the run start and for each row returned — nothing else, and platform compute is included free. Full post bodies cost one extra request per post, so they're opt-in, and they're only fetched for posts that can actually return one.

---

Questions or a bug?

Open an issue on the Issues tab of this Actor. Include your input JSON and the run ID — that's usually enough to reproduce it immediately.