Substack Scraper — Posts, Free vs Paid & Analytics

This Substack scraper exports publications and their full post archives, then computes the Substack analytics you cannot read off the site: how often a newsletter ships, what share of it sits behind the paywall, and what engagement that actually earns.

No account, no API key. It reads Substack's own public endpoints, so it does not break when the site changes its layout.

Substack free vs paid: the ratio that matters

Any scraper can return a list of post titles. The question a newsletter operator actually has is what does a working paid newsletter look like — and that answer lives in the ratio between free and paywalled posts, the cadence behind it, and the engagement it earns.

This scraper computes that per publication:

Publication	Paywalled	Avg words	Avg reactions	Posts/week
Pirate Wires	91%	3,082	86	3.8
The Pragmatic Engineer	61%	4,154	182	2.2
ByteByteGo	17%	2,609	233	5.3
Career Brew	0%	708	77	3.6

Four technology newsletters, four completely different paywall strategies. That comparison is the product.

Cadence is reported with its measurement window. Twelve posts from a daily publication span about a day, and extrapolating that to "57 posts per week" is arithmetically correct and practically useless. Every summary carries cadenceWindowDays next to postsPerWeek — raise Maximum posts per publication to widen it.

Substack post scraper: what it exports

Per publication: name · handle · subdomain · customDomain · publicationId · heroText · language · createdAt · hasPaidPlan · plans (interval, amount, currency) · freeSubscriptionBenefits · paidSubscriptionBenefits · foundingSubscriptionBenefits · communityEnabled · logoUrl

Per publication summary: postsAnalysed · firstPostDate · lastPostDate · postsPerWeek · cadenceWindowDays · freePosts · paywalledPosts · paywalledShare · avgWordcount · avgReactions · avgComments · avgRestacks · totalReactions · totalComments · podcastPosts · voiceoverPosts

Per post: title · subtitle · url · postDate · type · audience · isFree / isPaywalled · wordcount · reactionCount · commentCount · restacks · sectionName · hasPodcast · hasVoiceover · podcastDurationSec · language

How to use this Substack scraper

Two starting points, either or both:

• Publications — handles (astralcodexten), *.substack.com URLs, or custom domains (www.thefp.com). All three work; Substack serves the same data from a publication's own domain.
• Category slugs — technology, business, finance, culture, us-politics, food and more, plus subcategories like artificial-intelligence-robotics. The scraper walks the category's leading publications, paginating until it hits your limit.

{
  "categorySlugs": ["technology"],
  "maxPublications": 50,
  "maxPostsPerPublication": 100,
  "postsOutput": "separate"
}

Set Include posts to false for publication profiles only — no archive requests, no post charges.

FAQ

Do I need a Substack account or API key? No. This reads Substack's public endpoints, the same data the site ships to any browser.

Can it scrape paywalled post content? No. It reports whether a post is paywalled (isPaywalled, audience) and the metadata around it — title, date, word count, engagement — but never the body of a paid post. The paywall split is the product; the paid text is not.

Can it export subscriber counts or email lists? No. Substack does not publish subscriber numbers and this scraper collects no email addresses at all.

Does it work with custom domains? Yes. www.thefp.com and bariweiss.substack.com both resolve to the same publication.

Can it scrape Substack comments? No. Substack's robots.txt disallows /p/*/comment/*, and comments are the one surface on a publication that is unambiguously personal. They are never fetched.

How many posts can I get per publication? Up to 2,000. Posts load 30 per request going backwards through the archive.

Is scraping Substack legal? This reads publicly published content without authentication, honours Substack's robots.txt, and collects no personal data. How you use the output is your responsibility.

No personal data

Substack publications are businesses; the people who write them are not. This scraper collects publication identity and post metadata, and deliberately drops author names, author ids, bylines and email addresses. The test suite asserts it: a run that leaked a byline or an email address fails the build.

Pricing

Pay per event: one charge per publication record and one per post record. No charge for compute time or failed requests.

Development

npm install
npm test          # offline regression tests against saved fixtures
npm start         # local run; put an INPUT.json in storage/key_value_stores/default/