Substack Scraper: Export Newsletter Posts, Comments, and Author Leads

This Substack scraper reads public newsletters and saves posts, comments, and publication details to an Apify dataset. You can download JSON, CSV, or Excel, or hook the dataset into Google Sheets, Make, Zapier, or n8n. There is no login and no Substack API key, because Substack does not offer a public API for this.

It follows redirects, so a name.substack.com address and a custom domain like https://www.lennysnewsletter.com land on the same publication. Paywalled posts are flagged. The body you get for those is whatever Substack returns without a session, which may be a preview rather than the full paid text.

Use cases

• Sponsorship outreach: open a category leaderboard, then sort publications by the subscriber range Substack prints and by the author's Twitter handle.
• Newsletter directory: search a keyword, or walk categories such as business, technology, finance, and crypto, and keep the publication URL plus author handle.
• Reading and research: export free articles as Markdown, plain text, or HTML when you want a pile of posts instead of clicking through the archive.
• Comment review: flatten replies on a post, with parent id and depth, when you care what readers actually said.
• Competitor watch: rerun the same publication URL on a schedule and keep new posts from the archive.

How to run it

1. Choose a mode: posts, post details, comments, publication and author info, or discover.
2. Paste publication URLs, post URLs, or @handles. In discover mode, add category slugs and/or search phrases instead.
3. Set a max result count. The free cap is 1,000 rows per run.
4. Start the run and export the dataset as JSON, CSV, or Excel.

A bad URL does not stop the run. The actor writes an error row and moves on.

Input

Parameter	Type	Default	Description
mode	string	posts	posts, post_details, comments, publication_info, or discover.
startUrls	string[]	https://www.lennysnewsletter.com	Publication URLs, custom domains, https://substack.com/@handle, @handle, or /p/{slug} post URLs.
sort	string	new	Posts mode. new or top.
searchInArchive	string	empty	Posts mode. Keyword passed to Substack's archive search.
includeFullContent	boolean	false	Fetch each post body. One extra request per post.
contentFormat	string	markdown	markdown, text, or html.
dateFrom	string	empty	YYYY-MM-DD. With sort=new, paging stops once posts are older. With sort=top, older posts are skipped and paging continues.
includeAuthorProfile	boolean	true	In publication and discover modes, add followers, subscriber count, bio, and social links from the public profile.
categories	string[]	business	Discover mode. Category slug or numeric id, such as technology or 4.
leaderboardType	string	all	all or paid.
searchQueries	string[]	empty	Discover mode. Global Substack search.
maxItems	integer	100	Cap for the whole run. Maximum 1000.
maxItemsPerSource	integer	50	Cap per publication, category, or search phrase.
maxConcurrency	integer	5	Simultaneous requests. Maximum 5.
timeoutSecs	integer	300	Stop the run after this many seconds and keep rows already saved.
requestTimeoutSecs	integer	30	Per-request timeout.
proxyConfiguration	object	No proxy	Optional Apify proxy. Leave it off unless you hit 429 or 403.

Posts

{
    "mode": "posts",
    "startUrls": ["https://www.lennysnewsletter.com", "https://garymarcus.substack.com"],
    "sort": "new",
    "includeFullContent": false,
    "maxItems": 100
}

Single post

{
    "mode": "post_details",
    "startUrls": ["https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit"],
    "contentFormat": "markdown"
}

Comments

{
    "mode": "comments",
    "startUrls": ["https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit"],
    "maxItems": 200
}

Publication and author leads

{
    "mode": "publication_info",
    "startUrls": ["https://www.lennysnewsletter.com", "@lenny"],
    "includeAuthorProfile": true
}

Discover

{
    "mode": "discover",
    "categories": ["business", "technology"],
    "leaderboardType": "all",
    "searchQueries": ["product management"],
    "includeAuthorProfile": true,
    "maxItems": 100,
    "maxItemsPerSource": 25
}

What data does this actor extract?

Every row has mode, scrapedAt, and sourceInput. Failures use error and statusCode instead of a result. Codes: NOT_SUBSTACK, NOT_FOUND, HTTP_<code>, PARSE_ERROR, INVALID_INPUT.

The dataset table uses postTitle, postType, summary, and heroText because Apify's dataset schema rejects the field names title, type, and description. The JSON row still includes title, type, and description with the same values.

Posts and post details

{
    "mode": "posts",
    "postId": 215694124,
    "publicationId": 10845,
    "publicationHost": "www.lennysnewsletter.com",
    "title": "All of the Lenny & Friends Summit talks are now online!",
    "postTitle": "All of the Lenny & Friends Summit talks are now online!",
    "slug": "all-of-the-lenny-and-friends-summit",
    "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
    "publishedAt": "2026-09-29T13:15:57.512Z",
    "type": "newsletter",
    "postType": "newsletter",
    "audience": "everyone",
    "isPaywalled": false,
    "bodyMayBeTruncated": false,
    "wordCount": 1276,
    "reactionCount": 287,
    "commentCount": 7,
    "authors": [{"id": 1849774, "name": "Lenny Rachitsky", "handle": "lenny"}],
    "scrapedAt": "2026-10-12T00:00:00Z",
    "sourceInput": "https://www.lennysnewsletter.com"
}

Field	Type	Description
postId, publicationId, publicationHost	integer, integer, string	Ids, plus the host after redirects.
title, postTitle, subtitle, description, summary	string	Title and deck. postTitle and summary mirror title and description.
slug, url, publishedAt	string	Slug, canonical URL, publish time.
type, postType, audience	string	Post type (newsletter, and others such as podcast when Substack sends them) and audience (everyone or only_paid).
isPaywalled, bodyMayBeTruncated	boolean	Paywalled when audience is not everyone. The stored body may be incomplete.
wordCount, reactionCount, commentCount, restackCount	integer	Counts from the archive or post payload.
coverImage, section, tags, authors	string, string, string[], object[]	Cover, section name, tag names, and bylines (id, name, handle).
podcastUrl, podcastDurationSec	string, number	Present when the post has podcast fields.
preview	string	Short plain-text preview.
content	string	Body in the requested format. Omitted unless you asked for full content, or the mode is post_details.

Comments

{
    "mode": "comments",
    "commentId": 349085941,
    "postId": 215694124,
    "parentId": null,
    "depth": 0,
    "authorName": "Product Mindset",
    "authorHandle": "productmindset",
    "authorUserId": 8822099,
    "body": "Thank you Lenny for putting this together...",
    "date": "2026-09-29T23:34:47.245Z",
    "reactionCount": 3,
    "replyCount": 0
}

Field	Type	Description
commentId, postId, postUrl	integer, integer, string	Comment, post, and the post URL.
parentId, depth	integer	Parent comment. Depth 0 is a top-level comment. Nested replies are flattened.
authorName, authorHandle, authorUserId	string, string, integer	Who wrote it.
body, date, editedAt	string	Text and timestamps.
reactionCount, restackCount, replyCount	integer	Reactions, restacks, and Substack's reply count.
isDeleted, isPinned	boolean	Deleted flag, and pinned when Substack set pinned_by_user_id.

A post with zero comments produces no comment rows. That is not an error.

Publication info and leaderboard

{
    "mode": "publication_info",
    "publicationId": 10845,
    "name": "Lenny's Newsletter",
    "url": "https://www.lennysnewsletter.com",
    "subdomain": "lenny",
    "customDomain": "www.lennysnewsletter.com",
    "language": "en",
    "createdAt": "2019-06-01T15:35:37.885Z",
    "isPaid": true,
    "paidSubscribersRange": "Tens of thousands of paid subscribers",
    "freeSubscriberCount": 1200000,
    "freeSubscriberCountRaw": "1,200,000",
    "authorName": "Lenny Rachitsky",
    "authorHandle": "lenny",
    "authorFollowers": 1383847,
    "authorSubscriberCount": 1200000,
    "twitter": "lennysan",
    "socialLinks": [
        {"type": "twitter", "url": "https://x.com/lennysan"},
        {"type": "youtube", "url": "https://www.youtube.com/c/LennysPodcast"}
    ]
}

Field	Type	Description
publicationId, name, url, subdomain, customDomain	mixed	Identity and the public URL (custom domain when there is one).
description, heroText, summary	string	Hero text. The same string is stored under all three names.
language, createdAt, firstPostDate	string	Language and dates. First-post date is filled from the leaderboard only.
isPaid	boolean	True when payments_state is enabled.
paidSubscribersRange, totalSubscribersRange	string	Labels such as "Tens of thousands of paid subscribers". The total range comes from the leaderboard only.
freeSubscriberCount, freeSubscriberCountRaw	integer, string	Parsed count, plus the original string (1,200,000). Null when Substack does not send one.
logoUrl, hasPodcast, podcastFeedUrl, supportEmail	mixed	Podcast and support email are leaderboard fields. They stay empty on a plain publication lookup.
category, leaderboardRank	string, integer	Set in discover mode. Rank is the position while paging that leaderboard.
authorName, authorHandle, authorBio	string	Author fields.
authorFollowers, authorSubscriberCount	integer	From the public profile when enrichment is on.
twitter, socialLinks	string, object[]	Screen name and links (type, label, url).

Search rows add resultType: profile, post, or comment.

Errors

Field	Type	Description
sourceInput	string	The URL, handle, category, or query that failed.
error	string	NOT_SUBSTACK, NOT_FOUND, HTTP_<code>, PARSE_ERROR, or INVALID_INPUT.
statusCode	integer	HTTP status when there was one.

NOT_SUBSTACK is for hosts that are not Substack publications. platformer.news, for example, is on Ghost now, and its archive route returns 404.

Limits and notes

• 1,000 results per run. maxItems stops the run mid-page once that many rows are stored. Duplicate post, comment, and publication ids are skipped and do not count.
• Archive requests never ask for more than 50 posts. Substack returns 400 above that.
• Paid posts (audience only_paid) set isPaywalled and bodyMayBeTruncated. Do not treat content as the full paid article.
• Subscriber numbers are whatever Substack shows publicly: a range label, and sometimes a free-subscriber count. This actor does not estimate a number Substack did not send.
• This is not an official Substack product and it is not affiliated with Substack. Use it within Substack's terms and privacy rules. It does not log in, and it does not try to unlock paid posts.

FAQ

Is there an official Substack API?

No public API covers archive posts, comments, and leaderboards. This actor calls the same public JSON routes the site uses in the browser.

Can this Substack scraper read custom-domain newsletters?

Yes. https://www.lennysnewsletter.com and https://lenny.substack.com both work. Publications that moved to a custom domain redirect, and the row records the final host.

Can it export paid Substack posts?

It requests the post without a login. If Substack marks the audience as only_paid, the row is flagged and the body may be only the public preview.

How do I get a Substack subscriber count?

Publication rows include paidSubscribersRange (text such as "Tens of thousands of paid subscribers") and freeSubscriberCount when Substack sends a figure like 1,200,000. Leaderboard rows can also include totalSubscribersRange. There is no exact paid-subscriber integer in these public responses.

How many posts, comments, or newsletters can one run collect?

maxItems caps the run at 1,000. maxItemsPerSource caps each publication, category, or search. Comments are a separate row per comment, including nested replies.

Related tools

If you are cleaning the export afterward, a contact extractor and an HTML-to-Markdown pass are the usual next steps. BigiByte also builds scrapers for other publishing platforms, including Medium and Ghost, when you need them as a separate job.

Need bulk Substack data, a scheduled monitor, or a custom pipeline? BigiByte builds custom scrapers and APIs: https://bigibyte.com

Trademark Notice

Substack and related names, logos, and trademarks belong to their respective owners. Their use in this documentation is solely to identify the source of the data and does not imply affiliation, sponsorship, endorsement, or partnership.

Disclaimer

This Actor is an independent tool and is not affiliated with, endorsed by, sponsored by, or otherwise officially associated with Substack.

Is web scraping legal?

Collecting information that is already public is generally legal. Scraping is not the same thing as hacking, and there is no single law that bans it. The limits are the usual ones: do not break into accounts, do not take private or confidential data, and be careful with personal data and copyrighted works. A site's terms can also restrict how you access it. Facts such as a public name, price, or rating are not the same thing as republishing someone's article or photo.

This actor reads public pages or official public APIs. It does not log in and it does not bypass access controls.

This is not legal advice. For the longer explanation, see Apify's article Is web scraping legal?.