# Substack Scraper: Posts, Comments & Full Text (`eugene_krabs/substack-scraper`) Actor

Scrape posts from any Substack newsletter without login: titles, dates, authors, tags, likes, restacks, the full post body and public comments. Export to JSON, CSV or Excel, or use the API.

- **URL**: https://apify.com/eugene_krabs/substack-scraper.md
- **Developed by:** [Eugene](https://apify.com/eugene_krabs) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 post with contents

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper

### What does Substack Scraper do?

Substack Scraper extracts posts from any [Substack](https://substack.com) publication. Give it a list of newsletters and it returns one clean record per post: title, subtitle, publication date, authors, tags, likes, restacks, comment count, the full post body in HTML and, optionally, the public comments with their reply structure.

- **No login, cookies or API key.** The Actor reads only what Substack shows to any visitor.
- **Works with any publication**, on `*.substack.com` or on a custom domain such as `www.lennysnewsletter.com`.
- **Several publications in one run**, with a separate post limit for each.
- **Newest or most popular posts first**, so you can take the latest issues or the all-time top ones.
- **You pay per result**, not for run time, and you can cap the cost of every run.

Results can be downloaded as JSON, CSV, Excel or HTML, or read through the Apify API.

### What can you use Substack data for?

- **Competitor and market research.** See what other newsletters in your niche publish, how often, and which topics get the most likes, restacks and comments.
- **Content analysis.** Compare titles, post length and posting frequency against engagement to find out what works.
- **AI and RAG pipelines.** Load the full text of a newsletter archive into a vector database or use it as context for an LLM.
- **Archiving and monitoring.** Keep a copy of a publication's archive, or schedule the Actor and collect new posts as they come out.
- **Audience research.** Read what subscribers say in the comments and which comments other readers react to.

### What data can you extract from Substack?

| Data | Fields |
| --- | --- |
| Post | `postId`, `url`, `slug`, `title`, `subtitle`, `description`, `type`, `postDate`, `wordCount` |
| Access | `audience`, `isPaid` |
| Engagement | `reactionCount`, `commentCount`, `restackCount` |
| Content | `bodyHtml`, `previewText`, `coverImage`, `podcastUrl` |
| Classification | `section`, `tags`, `authors` (name and handle) |
| Comments | `id`, `parentId`, `body`, `date`, `authorName`, `authorHandle`, `reactionCount` |
| Source | `publicationUrl` |

### How to scrape Substack posts

1. Open the Actor in Apify Console and go to the **Input** tab.
2. Add one or more publications. A full URL, a domain or just the handle all work.
3. Set how many posts you want from each publication and whether to take the newest or the most popular ones.
4. Choose whether you need the post body and the comments. Each of them adds one request per post, so turn off what you do not need.
5. Click **Start**. When the run finishes, open the **Output** tab and export the data in the format you need.

### How much does it cost to scrape Substack?

The Actor uses pay-per-event pricing. You pay only for the data that lands in the dataset:

| Event | Charged for | Price | Per 1,000 |
| --- | --- | --- | --- |
| `post` | A post without the body (`includeBody` off, or the body could not be loaded) | $0.001 | $1.00 |
| `post-with-content` | A post with `bodyHtml` | $0.002 | $2.00 |
| `comment` | Each comment returned in `comments` | $0.0005 | $0.50 |
| `apify-actor-start` | Each run, per GB of memory ($0.0002 at the default 4 GB) | $0.00005 | — |

A post is charged as either `post` or `post-with-content`, never both.

Examples:

| Run | Cost |
| --- | --- |
| 50 newest posts of one publication, with the body | about $0.10 |
| 1,000 posts, metadata only | about $1.00 |
| 1,000 posts with the body | about $2.00 |
| 100 posts with the body and 2,000 comments in total | about $1.20 |

To keep a run within a budget, set the maximum cost per run in the run options. When the limit is reached, the Actor stops and keeps everything it has already saved.

### Input

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `publications` | array | required | Publications to scrape. A full URL (`https://www.lennysnewsletter.com`), a domain (`example.substack.com`) or a handle (`example`). |
| `maxPostsPerPublication` | integer | `50` | Maximum number of posts returned for each publication. |
| `sort` | string | `new` | `new` for newest first, `top` for most popular first. |
| `includeBody` | boolean | `true` | Fetch the HTML body of each post. Turn off for faster runs. |
| `includeComments` | boolean | `false` | Fetch public comments for each post. |
| `maxCommentsPerPost` | integer | `100` | Maximum number of comments per post. |
| `proxyConfiguration` | object | no proxy | Proxy settings. Substack usually works without a proxy. |

Example:

```json
{
    "publications": ["https://www.lennysnewsletter.com", "platformer"],
    "maxPostsPerPublication": 20,
    "sort": "new",
    "includeBody": true,
    "includeComments": false
}
```

### Output

One dataset item per post:

```json
{
    "publicationUrl": "https://www.lennysnewsletter.com",
    "postId": 215694124,
    "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
    "slug": "all-of-the-lenny-and-friends-summit",
    "title": "All of the Lenny & Friends Summit talks are now online!",
    "subtitle": "Plus, some reflections and takeaways from the day",
    "description": "Plus, some reflections and takeaways from the day",
    "type": "newsletter",
    "audience": "everyone",
    "isPaid": false,
    "postDate": "2026-09-29T13:15:57.512Z",
    "wordCount": 1276,
    "reactionCount": 263,
    "commentCount": 5,
    "restackCount": 4,
    "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/example.jpeg",
    "section": null,
    "tags": [],
    "authors": [{ "name": "Lenny Rachitsky", "handle": "lenny" }],
    "podcastUrl": null,
    "previewText": "Hey there, I'm Lenny...",
    "bodyHtml": "<p>...</p>",
    "comments": [
        {
            "id": 348735847,
            "parentId": null,
            "body": "thank you for existing",
            "date": "2026-09-29T15:30:12.240Z",
            "authorName": "Justin Wormley",
            "authorHandle": "justinwormley",
            "reactionCount": 3
        }
    ]
}
```

`comments` is present only when `includeComments` is on. Comments come as a flat list, and each reply points to the comment it answers through `parentId`. `bodyHtml` is `null` when the body was not requested or could not be loaded.

### Tips

- **Start small.** Run one publication with 10–20 posts first to check that the output has what you need.
- **Turn off the body for metadata-only tasks.** Engagement tracking and posting-frequency analysis do not need `bodyHtml`; without it posts cost half as much and the run is faster.
- **Use `top` to find the best content.** With `sort` set to `top`, a small `maxPostsPerPublication` gives you the most popular posts of a publication without going through the whole archive.
- **Watch the comment count on popular newsletters.** Comments are charged individually, so set `maxCommentsPerPost` to what you really need.
- **Schedule it.** To follow a newsletter, schedule a run with `sort` set to `new` and a small post limit.

### Integrations and API

Substack Scraper runs on the Apify platform, so you can start it from the [Apify API](https://docs.apify.com/api/v2), the JavaScript and Python clients or the CLI, run it on a [schedule](https://docs.apify.com/platform/schedules), and send the results on with [webhooks](https://docs.apify.com/platform/integrations/webhooks) or the ready-made [integrations](https://docs.apify.com/platform/integrations) for Make, Zapier, n8n, Google Sheets and others. AI agents can call it through the [Apify MCP server](https://mcp.apify.com).

### FAQ

#### Do I need a Substack account or an API key?

No. The Actor does not log in and needs no cookies or keys.

#### Can it scrape paid posts?

Only the part that Substack shows without a subscription. Paid posts are returned with all their metadata and `isPaid: true`, `bodyHtml` holds the free preview, and their comments come back empty.

#### Does it work with publications on a custom domain?

Yes. You can enter the custom domain directly, and a `*.substack.com` address that redirects to a custom domain is followed automatically.

#### What happens if a publication does not exist?

It is skipped with a warning in the log, and the other publications in the run are scraped as usual.

#### What if the body or the comments of a post fail to load?

The post is saved anyway with the data that was loaded, and it is charged as a `post` without content.

#### Do I need a proxy?

Usually not. If Substack starts to block requests on large runs, turn on Apify Proxy in the `proxyConfiguration` field.

#### Is it legal to scrape Substack?

The Actor collects only publicly available content. Posts are protected by copyright, and comments contain personal data (the names and handles of the people who wrote them), which is protected by GDPR and similar laws. Make sure you have a legitimate reason to collect and use this data, and consult a lawyer if you are not sure.

### Support

Found a bug or miss a field? Open an issue in the **Issues** tab of the Actor and describe the input you used.

# Actor input Schema

## `publications` (type: `array`):

Substack publications to scrape. Use a full URL (https://www.lennysnewsletter.com), a domain (example.substack.com) or just the handle (example).

## `maxPostsPerPublication` (type: `integer`):

Maximum number of posts to return for each publication.

## `sort` (type: `string`):

Order in which posts are taken from the publication archive.

## `includeBody` (type: `boolean`):

Fetch the HTML body of each post. Paid posts return only the part Substack shows publicly. Turning this off makes the run faster.

## `includeComments` (type: `boolean`):

Fetch public comments for each post, including the names of the people who wrote them.

## `maxCommentsPerPost` (type: `integer`):

Maximum number of comments to return for each post. Used only when comments are included.

## `proxyConfiguration` (type: `object`):

Proxy servers to use. Substack usually works without a proxy.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 50,
  "sort": "new",
  "includeBody": true,
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com"
    ],
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("eugene_krabs/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["https://www.lennysnewsletter.com"],
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("eugene_krabs/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call eugene_krabs/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,eugene_krabs/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/PzxDXnHgzkPqBFywK/builds/r5sUmQdCGcmWCKWGe/openapi.json