Substack Scraper: Posts, Full Text, Comments & Podcasts

Pull posts from any Substack publication, including ones on custom domains. For each post you get the title, subtitle, authors, date, type (newsletter, podcast, thread, video), paywall status, likes and reactions, comment count, restacks, cover image, podcast audio link, tags, the full body as HTML and plain text, and optionally the whole comment thread.

Uses Substack's public JSON endpoints, so it is fast and does not need a browser.

Use cases

• Archive or migrate a newsletter.
• Research a topic across many newsletters and feed the text to an LLM or RAG pipeline.
• Track engagement (likes, comments, restacks) over time for competitive analysis.
• Monitor an author or industry for new posts on a schedule.

Input

Field	Description
publications	URLs or subdomains: https://www.lennysnewsletter.com, lenny, stratechery.substack.com.
maxPosts	Per publication. Default 50, maximum 10000.
sort	new (the default) or top.
search	Substack's own full-text search filter. Results come in Substack's relevance order, and paging ends after two empty result windows in a row.
postedAfter	ISO date. With sort: new and no search, paging stops as soon as older posts appear, so daily runs stay cheap.
includeBody	Full HTML and text per post. On by default. Paywalled posts return the free preview and isBodyTruncated: true.
includeComments	Flattened comment thread per post with depth and parent id. Off by default.
types	Limit to newsletter, podcast, thread, video.

Output

{
  "publication": "Lenny's Newsletter",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "postId": "215694124",
  "slug": "all-of-the-lenny-and-friends-summit",
  "title": "All of the Lenny & Friends Summit talks are now online!",
  "subtitle": "Plus, some reflections and takeaways from the day",
  "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "type": "newsletter",
  "audience": "everyone",
  "isPaywalled": false,
  "postDate": "2026-09-29T13:15:57.512Z",
  "authors": [{ "name": "Lenny Rachitsky", "handle": "lenny", "id": 1849774 }],
  "wordCount": 1276,
  "reactionCount": 288,
  "commentCount": 7,
  "restacks": 12,
  "bodyHtml": "<p>...</p>",
  "bodyText": "...",
  "isBodyTruncated": false,
  "comments": [{ "id": 1, "depth": 0, "author": "...", "body": "..." }]
}

A post whose body or comments were requested but could not be fetched is still returned. Its bodyHtml, bodyText and isBodyTruncated (or comments) are null and a bodyError (or commentsError) field says why, for example HTTP 503.

The OUTPUT record lists each publication with posts found, posts pushed, and any body or comment fetch failures. It contains:

• publications: one entry per input entry, with input, status and, except for unrecognized entries, base (the publication's URL after redirects), postsFound, postsPushed, bodyFailures, commentFailures, detailsRecovered (bodies or comments that failed once and succeeded on the automatic retry), bodiesStopped, commentsStopped (see Pricing) and, for error and partial, error. A duplicate entry that was skipped before any request carries only input, base, status and duplicateOf.
• postsPushed: posts saved to the dataset. postsCharged: posts billed under pay-per-event pricing, otherwise null.
• chargeLimitReached: true if the run stopped at your maximum charge per run.

Each publication's status is one of:

• ok: the archive was read.
• partial: a later archive page failed after retries. Earlier posts are kept and error says which offset failed and how many posts were kept.
• not_found: the first archive request returned 404.
• error: the first archive request failed after retries; error has the reason.
• unrecognized: the entry is not a Substack URL or subdomain.
• duplicate: the entry is the same publication as an earlier one, so nothing was scraped from it. duplicateOf is the first input for that publication.

Pricing

You pay per post returned. Posts with full text cost more than metadata-only posts; comments are free. A publication entered more than once is scraped and charged once; the repeat entries are marked duplicate and cost nothing. Set a maximum charge per run to cap spend; the Actor stops cleanly when it is reached (chargeLimitReached is true in OUTPUT).

A post whose body or comments could not be fetched is not charged. Failed requests are retried (rate limits and server errors get one more try after a short pause); if the body or comments are still missing, the post is returned with bodyError or commentsError set and counted in bodyFailures or commentFailures. It still counts toward maxPosts. If body or comment requests keep failing (3 of one kind in a row, then a failed retry), the Actor stops requesting that kind for the rest of that publication: bodiesStopped or commentsStopped is true in the publication's OUTPUT entry, and the remaining posts are returned uncharged with an error starting "not requested".

Notes

• Only content visible to a logged-out reader is returned. Paid posts give the free preview.
• The Actor requests 20 posts per archive page; large archives take a few seconds per hundred posts.
• If a later archive page fails after retries, the publication is marked partial and keeps its earlier posts. The status message then reads, for example, "Done: 20 posts from 1 publications (1 incomplete, see OUTPUT)". A failed first page is error or not_found instead.
• Publications are compared by host, ignoring case, a trailing slash and any path, so lenny, lenny.substack.com and https://LENNY.substack.com/p/some-post are one publication. An entry that redirects to a host already scraped (for example the lenny subdomain and https://www.lennysnewsletter.com) is marked duplicate too; when the redirect only shows on its first archive request, that one request is all it costs.
• Archive paging stops when a page holds only posts already seen (the publication stays ok and a warning is logged). With search, such a page counts as an empty result window, so two in a row end the search. The Actor never requests an archive page past offset 50,000.
• When posts were saved without their body or comments, the status message adds "N without body or comments (not charged), see bodyError/commentsError".
• Rate limits (HTTP 429), server errors (5xx) and network errors are tried up to 3 times. The Actor waits for the Retry-After the server sends (at most 60 s), otherwise 1.5 s, then 3 s.