Substack Scraper — posts, engagement and paywall status in one schema

Scrape any Substack publication's full archive with engagement already turned into numbers: reactions, comments, restacks, word count, and whether each post sits behind the paywall.

Why this one

It gets the whole archive. Substack's archive endpoint returns 23 posts on the first page however large a limit you ask for, and 50 on every page after it. A scraper that treats "fewer than requested" as the end of the data stops at 23 and reports that as an entire publication. This pages until the archive actually ends — verified past 573 posts on a single publication.

Reactions come back as a number. Substack returns them keyed by emoji: {"❤": 21}. That breaks CSV exports and forces you to guess which emoji a given publication uses. You get reactionCount as an integer, plus topReaction if you want the emoji.

Custom domains survive. The Free Press publishes from the bariweiss slug but lives at thefp.com. url is the canonical one, so links actually work.

Paywall status as a boolean. isPaid instead of decoding audience values like only_paid and founding yourself.

Find newsletters, don't just fetch them

Every other Substack Actor takes a list of publications you already know. The more useful question is which newsletters cover a subject at all.

{ "discoverKeywords": ["climate"], "discoverOnly": true }

returns the publications themselves — name, url, tagline, author, the Substack category they sit in, and which of your keywords matched, so a surprising result can be explained rather than guessed at. Drop discoverOnly and those publications feed straight into the archive pipeline, so one run answers "who writes about this, and what have they published lately".

Why this needs building at all. Substack's own /api/v1/publication/search and /api/v1/post/search return HTTP 200 with an empty result set for anonymous callers — they look functional and are not. The category tree does work, and the query parameter on it is accepted and ignored, returning the identical payload. So the matching happens here, across publication name, tagline, author bio and handle.

Categories are ranked by popularity, so early pages hold the newsletters people have heard of. maxPagesPerCategory reaches further down the tail for rarer subjects; discoverCategories narrows the scan when you already know where to look.

Input

{
  "publications": ["noahpinion", "platformer", "www.thefp.com"],
  "sort": "new",
  "audience": "all",
  "searchQuery": "",
  "postedWithinDays": 0,
  "maxItemsPerPublication": 100
}

Slugs, subdomains and custom domains all work: noahpinion, bariweiss.substack.com and www.thefp.com are all valid.

Output

One row per post:

{
  "postId": "212192141",
  "publication": "www.thefp.com",
  "title": "'It's Just Dog Racism'",
  "subtitle": "'The myth of the pit bull is a lot of bullshit,' said actor Jon Bernthal.",
  "url": "https://www.thefp.com/p/its-just-dog-racism",
  "postDate": "2026-08-21T20:29:06.512Z",
  "type": "newsletter",
  "audience": "only_paid",
  "isPaid": true,
  "reactionCount": 21,
  "topReaction": "❤",
  "commentCount": 9,
  "restacks": 0,
  "wordCount": 2049,
  "sectionName": null
}

A RUN_SUMMARY record reports per-publication counts and any that could not be found.

Who this is for

• Newsletter operators benchmarking their engagement against comparable publications
• Media researchers tracking what gets read, restacked and paywalled
• Content teams mining a niche's archive for topics that landed
• Investors and analysts sizing a publication before a deal

Notes

• Reads Substack's public archive API. No login, no cookies, no CAPTCHA solving, no proxies.
• Post bodies are not included; this returns metadata and engagement.
• maxItemsPerPublication defaults to 100. Set it to 0 for the entire archive, which on a long-running publication can be thousands of posts.