Substack Scraper

Extract posts, optional full post text and public comments from Substack publications. Give it a publication name, URL or custom domain and get one structured row per post, plus one row per comment when you switch comments on. Results are available as JSON, CSV or Excel from the dataset, or through the API.

What you can do with it

• Build a content archive of a newsletter: titles, dates, authors, tags, reactions and comment counts for every post.
• Collect post text for research, summaries or training sets (paid posts return the public preview only).
• Monitor which posts get the most reactions and discussion in a niche by scraping several publications at once.
• Pull the public comment threads of a post for sentiment or community analysis.

Input

Field	Type	Default	Description
publications	list	sample	Publication URLs, post URLs, hosts or bare names such as lenny. Custom domains work. One value per line or comma separated.
maxPostsPerPublication	number	20	Newest posts per publication. 0 means all posts.
maxResults	number	100	Cap on all rows together (posts and comments). 0 means no limit.
includeContent	true/false	false	Add body_html and body_text.
includeComments	true/false	false	Add one row per public comment.
maxCommentsPerPost	number	20	Comments per post, replies included.

An empty input {} runs a small sample: up to 10 posts from one publication. The run status says so.

Example, posts only:

{
  "publications": ["lennysnewsletter.com", "thezvi.substack.com"],
  "maxPostsPerPublication": 24
}

Example, with post text:

{
  "publications": ["https://www.lennysnewsletter.com/"],
  "maxPostsPerPublication": 5,
  "includeContent": true
}

Example, with comments:

{
  "publications": ["thezvi"],
  "maxPostsPerPublication": 3,
  "includeComments": true,
  "maxCommentsPerPost": 20
}

Accepted publication formats

You enter	Scraped as
lenny	lenny.substack.com
thezvi.substack.com	thezvi.substack.com
lennysnewsletter.com or https://www.lennysnewsletter.com/	that custom domain
https://www.lennysnewsletter.com/p/some-post	the publication of that post
https://open.substack.com/pub/lenny/p/some-post	lenny.substack.com

If a .substack.com name belongs to a publication with its own domain, the results use that domain. The same publication entered in several forms is scraped once, and a post is never returned twice.

Output

Every row has a type field: post or comment.

Sample post row (text shortened):

{
  "type": "post",
  "post_id": 221622350,
  "slug": "the-hackening",
  "title": "The Hackening",
  "subtitle": null,
  "description": "Last week at a small conference, there was talk about a future event referred to as The Hackening.",
  "url": "https://thezvi.substack.com/p/the-hackening",
  "post_date": "2026-10-23T21:13:08.768Z",
  "post_type": "newsletter",
  "audience": "everyone",
  "is_paid": false,
  "wordcount": 7448,
  "reaction_count": 180,
  "reactions": {"❤": 180},
  "comment_count": 44,
  "restacks": 16,
  "tags": [],
  "section_name": null,
  "authors": [{"name": "Zvi Mowshowitz", "handle": "thezvi", "id": 10446622}],
  "podcast_url": null,
  "podcast_duration": null,
  "publication_host": "thezvi.substack.com",
  "publication_id": 573100,
  "truncated_body_text": "Last week at a small conference, there was talk about a future event referred to as The Hackening. ...",
  "scraped_at": "2026-10-28T09:15:02.331Z"
}

Sample comment row (text shortened):

{
  "type": "comment",
  "comment_id": 371329472,
  "post_id": 220367925,
  "post_url": "https://thezvi.substack.com/p/gemini-4-argon-the-model-and-safety",
  "parent_comment_id": 370775851,
  "body": "The Gemini family have long felt a generation behind in factuality and hallucinations. ...",
  "date": "2026-10-27T04:08:22.201Z",
  "author_name": "Coagulopath",
  "author_handle": "coagulopath632244",
  "reaction_count": 0,
  "depth": 1,
  "pinned": false,
  "deleted": false,
  "scraped_at": "2026-10-28T09:15:02.512Z"
}

Post fields

Fill rates come from a test run of 48 posts across two publications (one free, one mostly paid), 3 posts with content, and 54 comments on 3 posts. They depend on the publication.

Field	Description	Fill rate in test
post_id, slug, title, url, post_date, post_type, audience, is_paid	Identity, date and access level (is_paid is true when the audience is only_paid)	100%
subtitle	Subtitle	52%
description	Short description	100%
wordcount	Word count of the full post	100%
reaction_count, reactions, comment_count, restacks	Engagement	100%
cover_image	Cover image URL	100%
tags	Post tags	50%
section_name	Section the post belongs to	0% (none of the tested publications use sections)
authors	Name, handle and id of each author	100%
podcast_url, podcast_duration	Audio file and length for podcast posts	0% (no podcast posts in the test)
publication_host, publication_id	Publication the post belongs to	100%
scraped_at	Time the row was built (UTC, ISO 8601)	100%
truncated_body_text	Opening text of the post	100%
body_html, body_text	Post body. Only with includeContent	100% of posts with content
content_is_preview	True when only the public preview of a paid post is returned. Only with includeContent	100% of posts with content

Comment fields

Field	Description	Fill rate in test
comment_id, post_id, post_url, date, reaction_count, depth, pinned, deleted	Identity and position	100%
parent_comment_id	Parent comment, null for top-level comments	61% (the rest are top-level)
body	Comment text, null for deleted comments	100%
author_name, author_handle	Name and handle shown on the comment	100%
scraped_at	Time the row was built (UTC, ISO 8601)	100%

Comments come in thread order: a comment is followed by its replies. depth is 0 for top-level comments.

Paid posts

For posts with audience set to only_paid, the actor returns the metadata and the part of the text Substack shows publicly. With includeContent, content_is_preview is true on these rows. The full text of paid posts is not available.

Pricing

The actor uses pay per event. See the Pricing tab for the current prices. Events:

• actor-start: once per run, charged with the first delivered row.
• post: one post row without content, a paid-only post with its free preview, or a post whose body could not be loaded.
• post-with-content: one post row where the body was added with includeContent. It is charged instead of post, not on top. Only posts with the full body use this event. Paid-only posts return the free preview and are charged as a regular post, and so is a post whose body could not be loaded.
• comment: one comment row.

Nothing is charged when a run delivers no rows. Set a maximum cost per run in the run options and the actor stops at that limit with the status "Stopped at the max charge limit. N items delivered."

Use with AI agents (MCP)

Add the actor to an MCP client such as Claude Desktop:

{
  "mcpServers": {
    "substack-scraper": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://mcp.apify.com/?tools=santamaria-automations/substack-scraper",
        "--header",
        "Authorization: Bearer <YOUR_API_TOKEN>"
      ]
    }
  }
}

Then ask for example: "Get the 10 newest posts of thezvi.substack.com with their reaction counts."

FAQ

How do I find the publication name? Use the address of the newsletter. For https://lenny.substack.com use lenny; for a custom domain use the domain.

Why is a field empty? subtitle, tags and section_name are only set when the author filled them in. Podcast fields are only set on podcast posts.

Can I get the full text of paid posts? No. Only the public preview is returned and content_is_preview is true. Paid-only posts return the free preview and are charged as a regular post.

How do I get only new posts? Posts are returned newest first. Run with a small maxPostsPerPublication on a schedule and compare post_id with your earlier results.

How do I stop at a budget? Set the maximum cost per run, or use maxResults and maxPostsPerPublication. Comments add rows, so check maxCommentsPerPost when comments are on.

What happens when a publication does not exist? It is skipped and named in the run status. If nothing was found, nothing is charged.

Are subscriber lists or private data included? No. Only what is shown publicly on posts and comments is returned.

Limits

• Only publicly visible data is returned.
• Very large comment threads are cut at maxCommentsPerPost.
• Runs on large archives take longer.

Related actors

• RSS Feed Reader
• Google News Scraper
• Reddit Scraper

Support

Found a bug or missing field? Open an issue from the Issues tab of this actor.