# Substack Scraper: Posts, Content and Comments (`santamaria-automations/substack-scraper`) Actor

Extract posts from any Substack publication: title, subtitle, date, author, reactions, comment count, tags and cover image. Optionally add the full post text and the comments. Pay per result.

- **URL**: https://apify.com/santamaria-automations/substack-scraper.md
- **Developed by:** [NanoScrape](https://apify.com/santamaria-automations) (community)
- **Categories:** Social media, News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper

Extract posts, optional full post text and public comments from [Substack](https://substack.com) publications. Give it a publication name, URL or custom domain and get one structured row per post, plus one row per comment when you switch comments on. Results are available as JSON, CSV or Excel from the dataset, or through the API.

### What you can do with it

- Build a content archive of a newsletter: titles, dates, authors, tags, reactions and comment counts for every post.
- Collect post text for research, summaries or training sets (paid posts return the public preview only).
- Monitor which posts get the most reactions and discussion in a niche by scraping several publications at once.
- Pull the public comment threads of a post for sentiment or community analysis.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `publications` | list | sample | Publication URLs, post URLs, hosts or bare names such as `lenny`. Custom domains work. One value per line or comma separated. |
| `maxPostsPerPublication` | number | 20 | Newest posts per publication. 0 means all posts. |
| `maxResults` | number | 100 | Cap on all rows together (posts and comments). 0 means no limit. |
| `includeContent` | true/false | false | Add `body_html` and `body_text`. |
| `includeComments` | true/false | false | Add one row per public comment. |
| `maxCommentsPerPost` | number | 20 | Comments per post, replies included. |

An empty input `{}` runs a small sample: up to 10 posts from one publication. The run status says so.

Example, posts only:

```json
{
  "publications": ["lennysnewsletter.com", "thezvi.substack.com"],
  "maxPostsPerPublication": 24
}
```

Example, with post text:

```json
{
  "publications": ["https://www.lennysnewsletter.com/"],
  "maxPostsPerPublication": 5,
  "includeContent": true
}
```

Example, with comments:

```json
{
  "publications": ["thezvi"],
  "maxPostsPerPublication": 3,
  "includeComments": true,
  "maxCommentsPerPost": 20
}
```

#### Accepted publication formats

| You enter | Scraped as |
|---|---|
| `lenny` | lenny.substack.com |
| `thezvi.substack.com` | thezvi.substack.com |
| `lennysnewsletter.com` or `https://www.lennysnewsletter.com/` | that custom domain |
| `https://www.lennysnewsletter.com/p/some-post` | the publication of that post |
| `https://open.substack.com/pub/lenny/p/some-post` | lenny.substack.com |

If a `.substack.com` name belongs to a publication with its own domain, the results use that domain. The same publication entered in several forms is scraped once, and a post is never returned twice.

### Output

Every row has a `type` field: `post` or `comment`.

Sample post row (text shortened):

```json
{
  "type": "post",
  "post_id": 221622350,
  "slug": "the-hackening",
  "title": "The Hackening",
  "subtitle": null,
  "description": "Last week at a small conference, there was talk about a future event referred to as The Hackening.",
  "url": "https://thezvi.substack.com/p/the-hackening",
  "post_date": "2026-10-23T21:13:08.768Z",
  "post_type": "newsletter",
  "audience": "everyone",
  "is_paid": false,
  "wordcount": 7448,
  "reaction_count": 180,
  "reactions": {"❤": 180},
  "comment_count": 44,
  "restacks": 16,
  "tags": [],
  "section_name": null,
  "authors": [{"name": "Zvi Mowshowitz", "handle": "thezvi", "id": 10446622}],
  "podcast_url": null,
  "podcast_duration": null,
  "publication_host": "thezvi.substack.com",
  "publication_id": 573100,
  "truncated_body_text": "Last week at a small conference, there was talk about a future event referred to as The Hackening. ...",
  "scraped_at": "2026-10-28T09:15:02.331Z"
}
```

Sample comment row (text shortened):

```json
{
  "type": "comment",
  "comment_id": 371329472,
  "post_id": 220367925,
  "post_url": "https://thezvi.substack.com/p/gemini-4-argon-the-model-and-safety",
  "parent_comment_id": 370775851,
  "body": "The Gemini family have long felt a generation behind in factuality and hallucinations. ...",
  "date": "2026-10-27T04:08:22.201Z",
  "author_name": "Coagulopath",
  "author_handle": "coagulopath632244",
  "reaction_count": 0,
  "depth": 1,
  "pinned": false,
  "deleted": false,
  "scraped_at": "2026-10-28T09:15:02.512Z"
}
```

#### Post fields

Fill rates come from a test run of 48 posts across two publications (one free, one mostly paid), 3 posts with content, and 54 comments on 3 posts. They depend on the publication.

| Field | Description | Fill rate in test |
|---|---|---|
| `post_id`, `slug`, `title`, `url`, `post_date`, `post_type`, `audience`, `is_paid` | Identity, date and access level (`is_paid` is true when the audience is `only_paid`) | 100% |
| `subtitle` | Subtitle | 52% |
| `description` | Short description | 100% |
| `wordcount` | Word count of the full post | 100% |
| `reaction_count`, `reactions`, `comment_count`, `restacks` | Engagement | 100% |
| `cover_image` | Cover image URL | 100% |
| `tags` | Post tags | 50% |
| `section_name` | Section the post belongs to | 0% (none of the tested publications use sections) |
| `authors` | Name, handle and id of each author | 100% |
| `podcast_url`, `podcast_duration` | Audio file and length for podcast posts | 0% (no podcast posts in the test) |
| `publication_host`, `publication_id` | Publication the post belongs to | 100% |
| `scraped_at` | Time the row was built (UTC, ISO 8601) | 100% |
| `truncated_body_text` | Opening text of the post | 100% |
| `body_html`, `body_text` | Post body. Only with `includeContent` | 100% of posts with content |
| `content_is_preview` | True when only the public preview of a paid post is returned. Only with `includeContent` | 100% of posts with content |

#### Comment fields

| Field | Description | Fill rate in test |
|---|---|---|
| `comment_id`, `post_id`, `post_url`, `date`, `reaction_count`, `depth`, `pinned`, `deleted` | Identity and position | 100% |
| `parent_comment_id` | Parent comment, null for top-level comments | 61% (the rest are top-level) |
| `body` | Comment text, null for deleted comments | 100% |
| `author_name`, `author_handle` | Name and handle shown on the comment | 100% |
| `scraped_at` | Time the row was built (UTC, ISO 8601) | 100% |

Comments come in thread order: a comment is followed by its replies. `depth` is 0 for top-level comments.

### Paid posts

For posts with `audience` set to `only_paid`, the actor returns the metadata and the part of the text Substack shows publicly. With `includeContent`, `content_is_preview` is true on these rows. The full text of paid posts is not available.

### Pricing

The actor uses pay per event. See the Pricing tab for the current prices. Events:

- `actor-start`: once per run, charged with the first delivered row.
- `post`: one post row without content, a paid-only post with its free preview, or a post whose body could not be loaded.
- `post-with-content`: one post row where the body was added with `includeContent`. It is charged instead of `post`, not on top. Only posts with the full body use this event. Paid-only posts return the free preview and are charged as a regular `post`, and so is a post whose body could not be loaded.
- `comment`: one comment row.

Nothing is charged when a run delivers no rows. Set a maximum cost per run in the run options and the actor stops at that limit with the status "Stopped at the max charge limit. N items delivered."

### Use with AI agents (MCP)

Add the actor to an MCP client such as Claude Desktop:

```json
{
  "mcpServers": {
    "substack-scraper": {
      "command": "npx",
      "args": [
        "-y",
        "mcp-remote",
        "https://mcp.apify.com/?tools=santamaria-automations/substack-scraper",
        "--header",
        "Authorization: Bearer <YOUR_API_TOKEN>"
      ]
    }
  }
}
```

Then ask for example: "Get the 10 newest posts of thezvi.substack.com with their reaction counts."

### FAQ

**How do I find the publication name?** Use the address of the newsletter. For `https://lenny.substack.com` use `lenny`; for a custom domain use the domain.

**Why is a field empty?** `subtitle`, `tags` and `section_name` are only set when the author filled them in. Podcast fields are only set on podcast posts.

**Can I get the full text of paid posts?** No. Only the public preview is returned and `content_is_preview` is true. Paid-only posts return the free preview and are charged as a regular post.

**How do I get only new posts?** Posts are returned newest first. Run with a small `maxPostsPerPublication` on a schedule and compare `post_id` with your earlier results.

**How do I stop at a budget?** Set the maximum cost per run, or use `maxResults` and `maxPostsPerPublication`. Comments add rows, so check `maxCommentsPerPost` when comments are on.

**What happens when a publication does not exist?** It is skipped and named in the run status. If nothing was found, nothing is charged.

**Are subscriber lists or private data included?** No. Only what is shown publicly on posts and comments is returned.

### Limits

- Only publicly visible data is returned.
- Very large comment threads are cut at `maxCommentsPerPost`.
- Runs on large archives take longer.

### Related actors

- [RSS Feed Reader](https://apify.com/santamaria-automations/rss-feed-reader)
- [Google News Scraper](https://apify.com/santamaria-automations/google-news-scraper)
- [Reddit Scraper](https://apify.com/santamaria-automations/reddit-scraper)

### Support

Found a bug or missing field? Open an issue from the Issues tab of this actor.

# Actor input Schema

## `publications` (type: `array`):

Substack publications to scrape. Accepts a publication URL (https://www.lennysnewsletter.com), a post URL, a host (thezvi.substack.com) or just the subdomain name (lenny). Custom domains work. The same publication listed in several forms is scraped once. Leave empty to run a small sample.

## `maxPostsPerPublication` (type: `integer`):

Maximum number of posts to return for each publication, newest first. Set to 0 for all posts.

## `maxResults` (type: `integer`):

Maximum number of rows (posts and comments together) across all publications. Set to 0 for no limit.

## `includeContent` (type: `boolean`):

Add the post body as HTML (body_html) and plain text (body_text). Paid posts return only the preview that is public (content_is_preview is true). Rows with the full body use the post-with-content price instead of the post price. Paid-only posts return the free preview and are charged as a regular post.

## `includeComments` (type: `boolean`):

Add the public comments of each post as separate rows with type comment. Each comment row is billed as a comment result.

## `maxCommentsPerPost` (type: `integer`):

Maximum number of comments (replies included) returned for each post when comments are included.

## `proxyConfiguration` (type: `object`):

Proxy settings for the requests. The default works for most runs.

## Actor input object example

```json
{
  "publications": [
    "lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 20,
  "maxResults": 100,
  "includeContent": false,
  "includeComments": false,
  "maxCommentsPerPost": 20,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `defaultDataset` (type: `string`):

Posts and optional comments from Substack publications, one row per post or comment, told apart by the type field.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "lennysnewsletter.com"
    ],
    "maxPostsPerPublication": 20,
    "maxResults": 100,
    "maxCommentsPerPost": 20,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("santamaria-automations/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["lennysnewsletter.com"],
    "maxPostsPerPublication": 20,
    "maxResults": 100,
    "maxCommentsPerPost": 20,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("santamaria-automations/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 20,
  "maxResults": 100,
  "maxCommentsPerPost": 20,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call santamaria-automations/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,santamaria-automations/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/RtX6aqUSkI5EWf949/builds/fdU6JYBngIzGIlGvo/openapi.json