Substack Scraper: Extract Posts, Full Text, Likes and Comments from Any Substack Newsletter

Substack Scraper pulls posts from any Substack publication, whether it lives on name.substack.com or on its own custom domain, and returns clean, structured data: title, subtitle, URL, publish date, authors, post type, paywall status, word count, likes, comment count, restacks, cover image, the full text of free posts (plain text and HTML) and, optionally, the comments.

It can also discover newsletters for you, by keyword or by Substack category, and output a profile of each publication: name, URL, author, description, language and approximate subscriber count.

It reads the same public JSON API that Substack's own web pages use, so it needs no login, no browser and no proxy. You only pay for the posts you get.

What can this Substack scraper do?

• 📰 Scrape any Substack newsletter. Paste a subdomain (lenny), a substack.com address, a custom domain (www.lennysnewsletter.com) or a single post URL. Many at once is fine.
• 📄 Full text of free posts. Each free post comes with bodyText (clean plain text, with subscribe buttons and widgets removed) and bodyHtml. On the 4 free posts in the sample, the plain text's word count was within 3% of Substack's own count.
• 🔒 Honest paywall handling. Paid posts are flagged with isPaywalled: true and the raw audience value. You still get their metadata, word count, likes and a short preview; the full text is only available to paying subscribers, so it isn't output.
• ❤️ Engagement data. Likes (reactionCount), comment count, restacks and word count for every post.
• 💬 Comments, if you want them. Public comments as a flat, thread-ordered list with parentId, author display name, date and likes.
• 🔎 Newsletter discovery. Find publications by keyword (from Substack's post search) or take the top publications of a Substack category such as Technology, Business or Finance. Optionally output a profile item for each publication.
• 🔁 Incremental monitoring. Turn on Only new posts and schedule the Actor. Each run outputs, and charges for, only posts published since the first run. A run that finds nothing new costs two requests per publication.
• 🛡️ Built for reliability. One bad input never stops the run: a misspelled subdomain or a non-Substack domain is reported with a clear reason and the rest continue. Requests are paced politely and retried with backoff on 429/5xx errors. A per-publication summary is saved after every run.
• 💸 Pay per post. Full text and comments are included in the post price. There is no start fee.

Why this scraper instead of the alternatives?

	Substack Scraper	Typical Substack scrapers
Custom domains (e.g. lennysnewsletter.com)	✅ Resolved automatically	Often *.substack.com only
Full text of free posts	✅ Plain text and HTML, widgets stripped	Often HTML only, or an extra charge
Paywalled posts	✅ Flagged (isPaywalled, audience), never output as if they were complete	Preview text often returned as the "content"
Comments	✅ Included in the post price	Extra per-comment charge
Discovery by keyword and by category	✅	Rarely
Incremental "only new posts" mode	✅ Built in, per publication, per monitor	❌
Start fee	❌ None	Often $0.005–$0.09 per run
One bad input fails the whole run	❌ Isolated per publication	Often

Which inputs work?

Any of these, one per line:

lenny
https://lenny.substack.com
https://www.lennysnewsletter.com
climatemoney.substack.com
https://climatemoney.substack.com/p/the-climate-stabilization-era-is
https://open.substack.com/pub/lenny/p/some-post

• A name.substack.com address that moved to a custom domain is followed automatically.
• A post URL (/p/...) scrapes just that post.
• Profile pages (substack.com/@name) are not supported. Open the profile and copy the publication's address instead.

How to scrape Substack

1. Click Try for free or Start.
2. Add one or more publications under Publications, or a keyword under Discover publications by keyword.
3. Optionally set Max posts per publication, Sort, Only posts published after, and turn comments or publication info on.
4. Click Start. Then download the posts as JSON, CSV, Excel or HTML, or get them through the API.

Input example

{
    "publications": ["https://www.lennysnewsletter.com", "climatemoney"],
    "searchKeywords": ["climate tech"],
    "categories": ["technology"],
    "maxPublicationsPerSearch": 5,
    "maxPostsPerPublication": 20,
    "sort": "new",
    "postedAfter": "30 days",
    "includeFullText": true,
    "includeComments": false,
    "includePublicationInfo": true,
    "onlyNewPosts": false
}

Field	Description
publications	Subdomains, substack.com URLs, custom domains or post URLs.
searchKeywords	Find publications that wrote posts matching each keyword, then scrape them.
categories	Take the top publications of a Substack category (technology, business, finance, culture, us-politics, science, climate, health, ...). Subcategories work too.
maxPublicationsPerSearch	Publications per keyword or category (default 10).
maxPostsPerPublication	Limit per publication. 0 = the whole archive.
sort	new (newest first) or top (Substack's "most popular" ranking).
postedAfter	A date (2026-10-01) or a period (7 days, 3 months).
includeFullText	Add bodyText and bodyHtml for free posts (default true).
includeComments	Add public comments to each post (default false).
maxCommentsPerPost	Limit comments per post. 0 = all.
includePosts	Output posts (default true). Turn off to get only publication profiles.
includePublicationInfo	Also output one profile item per publication (default false).
onlyNewPosts	Incremental mode: only output posts published since the first run.
stateKey	Name of the monitor (separate monitors keep separate memories).
maxConcurrency, requestIntervalMs	Politeness settings. The defaults (3 parallel, at most 4 requests per second) rarely need changing.
proxyConfiguration	Off by default. Enable it only if Substack blocks or rate-limits the requests.

Output example

Every post is one dataset item (a real item, with the long text fields shortened):

{
    "itemType": "post",
    "postId": 221405593,
    "title": "After 6 years, I’m launching paid subscriptions on Substack. Here’s why now.",
    "subtitle": "Climate Money is growing!",
    "url": "https://climatemoney.substack.com/p/after-6-years-im-launching-paid-subscriptions",
    "slug": "after-6-years-im-launching-paid-subscriptions",
    "postDate": "2026-10-21T15:03:00.053Z",
    "type": "newsletter",
    "audience": "everyone",
    "isPaywalled": false,
    "publicationId": 219426,
    "publicationName": "Climate Money",
    "publicationSubdomain": "climatemoney",
    "publicationUrl": "https://climatemoney.substack.com",
    "authors": ["Climate Money by Susan Su"],
    "authorHandles": ["climatemoney"],
    "sectionName": null,
    "tags": ["climate finance", "substack", "climate change"],
    "wordcount": 791,
    "reactionCount": 2,
    "commentCount": 3,
    "restacks": 0,
    "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/544b1b96-3316-4905-890e-76f129fcfee6_2400x1260.png",
    "previewText": "I’m excited to share that Climate Money is launching paid subscriptions tomorrow, October 22. ...",
    "podcastUrl": null,
    "podcastDurationSec": null,
    "bodyText": "I’m excited to share that Climate Money is launching paid subscriptions tomorrow, October 22. The deep dives will always ...",
    "bodyHtml": "<p>I’m excited to share that Climate Money is launching paid subscriptions tomorrow, October 22. ...</p>...",
    "comments": [
        {
            "commentId": 367620777,
            "parentId": null,
            "depth": 0,
            "authorName": "(commenter's display name)",
            "authorHandle": "(commenter's handle)",
            "isPostAuthor": false,
            "body": "The free and paid boundary is clearest when readers can tell what remains dependable on each side. ...",
            "date": "2026-10-22T17:34:07.205Z",
            "editedAt": null,
            "reactionCount": 1,
            "restacks": 0
        }
    ],
    "discoveredBy": "input",
    "scrapedAt": "2026-11-02T11:11:16.316Z"
}

See samples/sample-output.json for 8 real items (2 publications, 4 free and 2 paywalled posts; commenter names are replaced with placeholders in that file).

Notes on the fields:

• postId is stable. Use it to de-duplicate.
• isPaywalled is true when the full text is not public, i.e. audience is anything other than everyone (in testing: only_paid; Substack also has founding-member and subscribers-only posts). For those posts bodyText, bodyHtml and comments are null, because Substack only shows them to subscribers. previewText, wordcount and the engagement counts are still there.
• type is Substack's post type: newsletter, podcast, thread and so on. Podcast posts also have podcastUrl and podcastDurationSec.
• bodyText and bodyHtml are only present when Include full text is on, and comments only when Include comments is on.
• Restacks of other publications' posts that appear in an archive are skipped (and counted in the run summary). They belong to the other publication.
• discoveredBy says where a publication came from: input, keyword: ... or category: ....

With Include publication info on, each publication also gets one item with itemType: "publication" and type: "publication":

{
    "itemType": "publication",
    "type": "publication",
    "publicationId": 10845,
    "name": "Lenny's Newsletter",
    "subdomain": "lenny",
    "customDomain": "www.lennysnewsletter.com",
    "url": "https://www.lennysnewsletter.com",
    "description": "Deeply researched product, growth, and career advice for product leaders, founders, and ambitious builders.",
    "language": "en",
    "authorName": "Lenny Rachitsky",
    "authorHandle": "lenny",
    "logoUrl": "https://substackcdn.com/image/fetch/...",
    "createdAt": "2019-06-01T15:35:37.885Z",
    "firstPostDate": "2019-06-14T15:00:48.833Z",
    "subscriberCountApprox": "1.2M+",
    "subscriberCountText": "Millions of subscribers",
    "paidSubscriberCountText": "Tens of thousands of paid subscribers",
    "paidSubscriptionsEnabled": true,
    "hasPodcast": true,
    "twitterHandle": "lennysan",
    "discoveredBy": "input",
    "sourceInput": "https://www.lennysnewsletter.com",
    "scrapedAt": "2026-11-02T11:11:07.509Z"
}

Subscriber counts are the rounded figures Substack shows publicly (e.g. "1.2M+"). Exact counts are not public.

A RUN_SUMMARY record in the run's key-value store lists, for every publication: posts output, posts listed, paywalled posts, posts skipped (too old, not new, restacks), failed requests, errors and duration, plus the results of each keyword and category search.

How much does it cost to scrape Substack?

This Actor uses pay-per-event pricing. You pay $1.00 per 1,000 posts ($0.001 per post) on the Free plan, and less on paid Apify plans. Full text and comments are included. Publication profile items, which are optional, cost $0.002 each. There is no start fee.

Posts	Price (Free plan)
100	$0.10
1,000	$1.00
10,000	$10.00

Posts removed by your filters, posts already output in incremental mode, skipped restacks and failed publications cost nothing. To cap spending, set Maximum cost per run in the run options. The scraper stops cleanly when the budget is reached and says so in the status message.

Scheduling and incremental monitoring of new posts

Use this to get alerts for new posts from a list of newsletters, or to keep a database in sync:

1. Turn on Only new posts and give the monitor a name in Monitor name (e.g. ai-newsletters).
2. Save the configuration as a Task and add a Schedule (e.g. every morning).
3. The first run outputs the current posts (up to Max posts per publication) and records its start time as the baseline. If the first run is cut short (by Maximum cost per run, an error or an abort), the baseline is not set yet: the next run continues the first run and skips the posts already output.
4. Every later run outputs only posts published after the baseline that haven't been output yet. Posts are listed newest-first and the scraper stops at the first archive page older than the baseline, so a run that finds nothing new costs two requests per publication.

Older posts never show up in later runs, even if the first run stopped at Max posts per publication. The memory is kept per publication in a named key-value store (substack-scraper-state by default), under the key you choose. Different monitor names are fully independent. Don't run two runs with the same monitor name at the same time.

Discovering newsletters

• By keyword. For each keyword the scraper runs Substack's post search and collects the publications behind the matching posts, in search order, up to Max publications per keyword or category. This finds publications that write about the topic, which is usually what you want for research, outreach or sponsorships.
• By category. The scraper takes the publications Substack ranks highest in the category (the same list as the category pages on substack.com).
• To get only a list of newsletters, turn Include posts off and Include publication info on.

Substack's dedicated publication-name search returns no results without a login, so it isn't used.

Integrations and API

• API. Start runs and fetch results with the Apify API, or the JavaScript and Python clients:

  curl -X POST "https://api.apify.com/v2/acts/rjmacarthy~substack-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"publications":["https://www.lennysnewsletter.com"],"maxPostsPerPublication":20}'
  

• Webhooks. Get notified (or trigger your pipeline) when a run finishes.

• Zapier, Make, n8n. Send new posts to Google Sheets, Airtable, Slack, email or your CRM with no code.

• Google Sheets, Slack, Airbyte, Keboola and more through Apify integrations.

• AI agents. Use the Actor as a tool through the Apify MCP server, e.g. to summarise this week's posts from a list of newsletters.

FAQ

Do I need a Substack account, login, cookies or a proxy? No. The scraper uses the public endpoints that Substack's own pages use. A proxy option exists in case Substack blocks or rate-limits the requests.

Can it get the full text of paid posts? No. Substack only serves a preview of paid posts to non-subscribers, and this Actor doesn't log in. Paid posts are output with their metadata and isPaywalled: true, and bodyText is null, so you never mistake a preview for the full article.

Why are comments null on paywalled posts? Substack only shows comments on paid posts to subscribers; the API returns none to everyone else. The commentCount is still correct.

Does it collect subscriber lists or email addresses? No. It only outputs what anyone can see on a publication's public pages: posts, public bylines, public comments and the publication profile. It never outputs subscriber lists or emails.

Are there any limits on the free plan? No. Free-plan users get the same features. You pay only per item, and your Apify free monthly credit covers it.

How fast is it? A metadata-only scrape needs one request per 23–50 posts, so whole archives take seconds. Full text needs one extra request per free post, and comments one per post that has comments. At the default pacing of at most 4 requests per second, that is roughly 200 full-text posts per minute. The scraper slows down automatically if Substack answers with HTTP 429.

What happens with an invalid input? That publication is reported as failed with a clear reason, for example "publication not found", "the domain does not exist" or "does not look like a Substack publication", and the others continue. You're not charged for it.

Is scraping Substack legal? The scraper only collects publicly available content that publishers make available to everyone. It doesn't collect personal data beyond public bylines and public comment display names. Always check that your use complies with the applicable terms and laws, including copyright: the full text of posts belongs to their authors.

Limits

• Paid and subscriber-only posts: metadata, preview text and counts only (see above).
• Keyword discovery finds publications through matching posts, not by publication name. Each keyword reads up to 10 search pages.
• Profile URLs (substack.com/@name), Notes and Chat are not supported.
• Restacks of other publications' posts are skipped.
• Substack's "top" ranking is Substack's own and can change between runs.
• Incremental state lives in a named key-value store in your account, created by the Actor itself. Deleting that store resets the monitor. If you already have an unrelated store with the same name, set a different State key-value store name.

Ready-to-run examples

• Find AI newsletters on Substack and scrape their posts
• Export every post from a Substack newsletter

More scrapers by the same developer

• Telegram Channel Scraper
• Google Ads Transparency Scraper
• ATS Jobs Scraper: Greenhouse, Lever, Ashby & Workday
• Dice Jobs Scraper
• SEEK Jobs Scraper
• Workday Jobs Scraper

Browse all my Actors on my Apify Store profile.

Support

Found a publication that doesn't work, or need an extra field? Open an issue on the Issues tab with the URL. Fixes are usually quick.