# Substack Newsletter Scraper (`pulselab/substack-newsletter-scraper`) Actor

Substack newsletters in, clean rows out: posts (title, date, paid/free, reactions, restacks, authors, optional full body), comments (author, reactions, replies), and a per-publication summary with subscriber count, posts/week, paid share and top posts. Custom domains supported. Public API, no login.

- **URL**: https://apify.com/pulselab/substack-newsletter-scraper.md
- **Developed by:** [Pulse Lab](https://apify.com/pulselab) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 post rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Newsletter Scraper – Posts, Comments & Publication Stats

Scrape any public Substack newsletter into clean rows: **one row per post**, optionally **one row per comment**, and **one summary row per publication** with **subscriber count**, posting cadence, paid share, average engagement and top posts.

Works with `name.substack.com`, **custom domains** (`lennysnewsletter.com`, `noahpinion.blog`, …) and single post URLs. Uses Substack's public JSON API – no login, no API key, no proxies, no browser. Three publications with 15 posts and 258 comments take about 3 seconds.

### What you get

**Posts** (`type: "post"`) – title, subtitle, URL, post type (newsletter / podcast / video / thread), free or paid, post date, authors, word count, reactions, restacks, comment count, cover image, tags, podcast audio URL, public preview text, and optionally the **full body** (HTML + plain text).

**Comments** (`type: "comment"`, with `includeComments`) – comment text, author name and handle, date, reactions, restacks, reply count, nesting depth and parent, **whether the comment is by the post author**, and the commenter's membership (free / paid).

**Publication summary** (`type: "publication"`) – **subscribers** (free count, e.g. 1,200,000) and **paid-subscriber tier** ("Tens of thousands of paid subscribers"), bestseller tier, leaderboard ranking, description, logo, created date, paid plan enabled, author name / bio / followers / social links, posts per week, posts in the last 30 days, paid share, average reactions / comments / restacks / word count, post-type mix, top tags, latest and oldest post, **top 3 posts by reactions**.

### Use cases

- **Sponsorship / PR prospecting** – filter newsletters by subscribers, cadence and engagement in one run.
- **Newsletter research & benchmarking** – compare dozens of publications side by side.
- **Audience research** – who comments, what they say, which posts spark discussion.
- **Content monitoring** – run on a schedule with `onlyNewPosts` to get only new posts each time.
- **LLM / RAG datasets** – `includeBody` returns clean plain text per post.

### Input

| Field | Description |
|---|---|
| `publications` (required) | `lenny`, `on.substack.com`, `https://www.lennysnewsletter.com`, or a post URL `…/p/slug`. |
| `maxPostsPerPublication` | Max post rows per publication (default 50, max 5000). |
| `sortBy` | `new` (default) or `top`. |
| `audience` | `all` (default), `free`, `paid`. |
| `keywords` | Search terms – uses Substack's own search (title + body), one query per keyword. |
| `dateFrom` | Keep posts on/after this date. |
| `onlyNewPosts` | Incremental mode for scheduled runs – only posts newer than the last run. |
| `includeBody` | Fetch the full body of each post (extra request per post, separate event). |
| `includeComments` | Fetch comments with replies (extra request per post with comments, per-comment event). |
| `maxCommentsPerPost` | Max comment rows per post (default 100). |
| `includePublicationInfo` | Save the summary row (default true). |

```json
{
  "publications": ["https://www.lennysnewsletter.com", "https://www.noahpinion.blog", "on.substack.com"],
  "maxPostsPerPublication": 50,
  "includeComments": true
}
```

### Output examples

```json
{
  "type": "publication",
  "publication": "Lenny's Newsletter",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "status": "OK",
  "subscribers": 1200000,
  "subscribersLabel": "1,200,000 subscribers",
  "paidSubscribersLabel": "Tens of thousands of paid subscribers",
  "bestsellerTier": 10000,
  "authorName": "Lenny Rachitsky",
  "createdAt": "2019-06-01T15:35:37.885Z",
  "postsPerWeek": 1.9,
  "paidShare": 0.52,
  "avgReactions": 412.6,
  "avgComments": 9.4,
  "topPosts": [{ "title": "How to turn your AI into a world-class designer", "url": "…", "reactions": 1113 }]
}
```

```json
{
  "type": "comment",
  "publication": "Noahpinion",
  "postTitle": "Maybe the Rapture is already here",
  "author": "Reader Name",
  "authorHandle": "readername",
  "isPostAuthor": false,
  "membership": "paid",
  "date": "2026-10-09T10:12:00.000Z",
  "body": "…",
  "reactions": 64,
  "replies": 3,
  "depth": 0
}
```

### Pricing (pay per event)

- **post** – one post row
- **post-body** – full body fetched for a post (only with `includeBody`)
- **comment** – one comment row (only with `includeComments`)
- **publication** – one publication summary row

Publications that cannot be read (`NOT_FOUND`, `ERROR`) and bodies that fail to download are not charged. Platform usage is included.

### Limitations

- Only **public** data: paywalled posts return metadata and the public preview, never the paid body.
- Subscriber counts are the public figures Substack shows on the publication page / author profile; publications that hide them (e.g. Substack's own company newsletters) return `null`.
- Substack Notes are not included.
- A few publications rate-limit body downloads aggressively; the scraper backs off and retries, and the rare failed body is left empty and not charged.

# Actor input Schema

## `publications` (type: `array`):

Substack newsletters: subdomain (lenny), substack.com URL, custom-domain URL (https://www.lennysnewsletter.com) or a single post URL (…/p/slug).

## `maxPostsPerPublication` (type: `integer`):

Upper limit of post rows saved (and charged) per publication.

## `sortBy` (type: `string`):

Order of the archive. Use 'new' for monitoring, 'top' for best-of lists.

## `audience` (type: `string`):

Filter by paywall.

## `keywords` (type: `array`):

Search terms. Uses Substack's own search (title + body), one query per keyword; results are merged. Leave empty for the full archive.

## `dateFrom` (type: `string`):

Keep only posts published on/after this date (YYYY-MM-DD). With 'Newest first' the scraper stops early once older posts appear.

## `onlyNewPosts` (type: `boolean`):

Remember the newest post per publication and, on the next run, return only posts published after it. For scheduled monitoring. State is stored in the run's key-value store as SUBSTACK_STATE.

## `includeBody` (type: `boolean`):

Fetch each post's full body (HTML + text). One extra request per post; charged as a separate event. Paywalled posts return only the public preview.

## `includeComments` (type: `boolean`):

Fetch comments (with replies) for each saved post that has comments. One extra request per post; each comment is a separate row and event.

## `maxCommentsPerPost` (type: `integer`):

Upper limit of comment rows per post.

## `includePublicationInfo` (type: `boolean`):

Save one summary row per publication: subscribers (free count + paid-subscriber tier), bestseller tier, author, created date, posts/week, paid share, average reactions, top posts.

## `proxyConfiguration` (type: `object`):

Optional. Not needed normally.

## Actor input object example

```json
{
  "publications": [
    "on.substack.com",
    "lenny",
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 50,
  "sortBy": "new",
  "audience": "all",
  "keywords": [],
  "onlyNewPosts": false,
  "includeBody": false,
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "includePublicationInfo": true
}
```

# Actor output Schema

## `posts` (type: `string`):

No description

## `comments` (type: `string`):

No description

## `publications` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "on.substack.com",
        "lenny",
        "https://www.lennysnewsletter.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("pulselab/substack-newsletter-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": [
        "on.substack.com",
        "lenny",
        "https://www.lennysnewsletter.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("pulselab/substack-newsletter-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "on.substack.com",
    "lenny",
    "https://www.lennysnewsletter.com"
  ]
}' |
apify call pulselab/substack-newsletter-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pulselab/substack-newsletter-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/oAudP4VHyJGayXEiW/builds/rRwz3cOz2AzCAvsgO/openapi.json