# Substack Scraper: Newsletter Posts, Likes & Full Text (`ksvan/substack-scraper`) Actor

Scrape posts from any Substack newsletter (substack.com or custom domain), single post URLs or newsletters found by keyword: title, authors, date, likes, comments, restacks, word count, paywall flag, tags and full text of free posts. No login.

- **URL**: https://apify.com/ksvan/substack-scraper.md
- **Developed by:** [K.S. Van](https://apify.com/ksvan) (community)
- **Categories:** News, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper: Newsletter Posts, Likes & Full Text

Scrape posts from **any Substack newsletter**, on `name.substack.com` or a **custom domain**, or from single post URLs. You can also **find newsletters by keyword** and scrape their latest posts. You get the title, subtitle, authors, date, likes, comments, restacks, word count, paywall flag and tags, and optionally the **full text** of free posts.

No login, no browser. Pay only per post delivered.

### What you can paste

- Newsletter URLs: `https://www.lennysnewsletter.com`, `https://astralcodexten.substack.com`
- Bare subdomain names: `lenny`, `astralcodexten`
- Single post URLs: `https://www.lennysnewsletter.com/p/how-to-tell-if-a-startup-has-a-moat`, `open.substack.com/pub/name/p/slug`
- Keywords in **Find newsletters by keyword**, for example `productivity` or `AI startups`

### What you get

| Field | Example |
|---|---|
| `postId`, `title`, `subtitle`, `url`, `slug` | post identity |
| `publicationUrl` | https://www.lennysnewsletter.com |
| `publishedAt` | 2026-09-29T13:15:57.512Z |
| `type` | newsletter / podcast / thread |
| `audience`, `isPaywalled` | everyone / only_paid, false |
| `authors`, `authorHandles` | \["Lenny Rachitsky"], \["lenny"] |
| `wordCount`, `likeCount`, `commentCount`, `restackCount` | 1276, 291, 7, 5 |
| `tags`, `section`, `description`, `coverImage`, `language` | metadata |
| `podcastUrl`, `podcastDurationSeconds` | for podcast episodes |
| `contentText`, `contentHtml`, `contentIsPreview` | with **Include full text** |

### Use cases

- **Newsletter research and competitive analysis**: which topics and formats get the most likes, comments and restacks.
- **Content curation and monitoring**: track new posts from the newsletters you follow (use **Published after**).
- **AI and RAG**: clean full text of free posts for summaries, search and LLM pipelines.
- **Lead and partner discovery**: find newsletters in a niche by keyword.

### How to use

1. Add newsletters or post URLs, and/or keywords.
2. Set **Max posts per newsletter** (newest first) and optionally **Published after**.
3. Turn on **Include full text** if you need the post body.
4. Run. Each post is one dataset record.

#### Example input

```json
{
  "startUrls": ["https://www.lennysnewsletter.com", "astralcodexten"],
  "searchQueries": ["productivity"],
  "maxPublicationsPerQuery": 5,
  "maxPostsPerPublication": 50,
  "includeContent": true,
  "publishedAfter": "2026-06-01"
}
```

### Pricing

- **Post**: charged per post with metadata.
- **Post full text**: charged additionally only when **Include full text** is on and the post has text.

Proxies and compute are included.

### Notes

- Paid-subscriber posts are listed with all metadata, but only the public preview of their text is available (`contentIsPreview: true`). This actor does not bypass paywalls.
- Use **Only free posts** to skip paid posts entirely.

### FAQ

**Do I need a Substack account?** No.

**Something broke?** Open an issue on the Issues tab and it will be fixed quickly.

# Actor input Schema

## `startUrls` (type: `array`):

Newsletter URLs (name.substack.com or a custom domain), the bare subdomain name (lenny), or single post URLs (.../p/slug).

## `searchQueries` (type: `array`):

Optional. Searches Substack for newsletters on a topic and scrapes their latest posts.

## `maxPublicationsPerQuery` (type: `integer`):

Only used with keywords.

## `maxPostsPerPublication` (type: `integer`):

Newest posts first.

## `publishedAfter` (type: `string`):

Optional YYYY-MM-DD. Stop at older posts.

## `onlyFreePosts` (type: `boolean`):

Skip paid-subscriber posts.

## `includeContent` (type: `boolean`):

Add the post text and HTML. Free posts come in full; for paid posts only the public preview is available (contentIsPreview = true).

## `maxItems` (type: `integer`):

Stop after this many posts in total. 0 = no limit.

## `proxyConfiguration` (type: `object`):

Apify datacenter proxy works well.

## `maxRetries` (type: `integer`):

Retries with a fresh proxy IP before a source is reported as failed.

## Actor input object example

```json
{
  "startUrls": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPublicationsPerQuery": 10,
  "maxPostsPerPublication": 50,
  "publishedAfter": "",
  "onlyFreePosts": false,
  "includeContent": false,
  "maxItems": 0,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxRetries": 6
}
```

# Actor output Schema

## `results` (type: `string`):

One record per post.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        "https://www.lennysnewsletter.com"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("ksvan/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": ["https://www.lennysnewsletter.com"],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("ksvan/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    "https://www.lennysnewsletter.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call ksvan/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ksvan/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/xfgCSkTAaYodhgxBJ/builds/4178sSbeRbaAJ4hfn/openapi.json