# Substack Scraper: Posts, Full Text, Comments & Newsletters (`leanpilar/substack-scraper`) Actor

Scrape Substack posts with full text as clean Markdown, comments, search across all of Substack, and top newsletters per category with subscriber counts and prices. No login.

- **URL**: https://apify.com/leanpilar/substack-scraper.md
- **Developed by:** [Lean Pilar](https://apify.com/leanpilar) (community)
- **Categories:** Social media, AI, Lead generation
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper: Posts, Full Text, Comments & Newsletters

Get Substack posts with the **full article as clean Markdown**, their comments, **search results from all of Substack**, and **top newsletters per category** with subscriber counts and subscription prices. Works with custom domains (like `www.lennysnewsletter.com`) and profile links (`substack.com/@lenny`). No login, no cookies, no proxy needed.

### What you can do

- **Read a newsletter's archive**: newest or most popular posts, filtered by date ("last 30 days"), free/paid, post type, likes, or a keyword.
- **Get full text for AI**: every free post comes as Markdown and plain text, without Subscribe buttons and share widgets. Ready for RAG, summaries and LLM pipelines. Paid posts include the public preview.
- **Collect comments**: best comments first, with replies, author, likes and date.
- **Search all of Substack**: find posts and newsletters about any topic, e.g. "ai marketing".
- **Find newsletters to sponsor or pitch**: Substack's leaderboards for 32 categories (Technology, Business, Finance, Culture...), top overall or top paid, with approximate subscribers, paid subscriber band, monthly and yearly price, author, X/Twitter handle and launch date. Filter by minimum subscribers.

### Input examples

Latest posts of two newsletters, last 30 days, with full text:

```json
{
  "newsletters": ["lenny", "www.astralcodexten.com"],
  "publishedAfter": "30 days",
  "maxPostsPerNewsletter": 50
}
```

Top 100 paid Business newsletters with at least 10,000 subscribers:

```json
{
  "categories": ["business"],
  "leaderboard": "paid",
  "maxNewslettersPerCategory": 100,
  "minSubscribers": 10000
}
```

Search Substack for a topic and get posts plus comments:

```json
{
  "searchQueries": ["ai marketing"],
  "maxPostsPerSearch": 30,
  "includeComments": true,
  "maxCommentsPerPost": 50
}
```

### Output

Every row has a `type`: `post`, `comment` or `newsletter`. The dataset has a separate table view for each.

**Post**

```json
{
  "type": "post",
  "url": "https://www.noahpinion.blog/p/for-the-democrats-now-comes-the-hard",
  "title": "For the Democrats, now comes the hard part",
  "subtitle": "The Dems have to figure out what they stand for, and what they want to do.",
  "newsletter": "Noahpinion",
  "authors": ["Noah Smith"],
  "publishedAt": "2026-11-06T06:46:24.428Z",
  "isPaid": false,
  "postType": "newsletter",
  "wordCount": 3544,
  "likes": 374,
  "comments": 114,
  "restacks": 41,
  "markdown": "Democrats won a historic victory in Tuesday's election. ...",
  "text": "Democrats won a historic victory ...",
  "isFullText": true
}
```

**Newsletter**

```json
{
  "type": "newsletter",
  "name": "Lenny's Newsletter",
  "url": "https://www.lennysnewsletter.com",
  "author": "Lenny Rachitsky",
  "subscribers": 1200000,
  "subscribersText": "Over 1,200,000 subscribers",
  "paidSubscribersText": "Tens of thousands of paid subscribers",
  "isPaid": true,
  "monthlyPrice": 20,
  "yearlyPrice": 200,
  "currency": "USD",
  "twitter": "lennysan",
  "category": "business",
  "rank": 1
}
```

**Comment**: `postUrl`, `postTitle`, `author`, `authorHandle`, `text`, `likes`, `replies`, `depth`, `parentId`, `publishedAt`.

### Pricing

Pay only for rows you get. No start fee.

| Row | Price |
|---|---|
| Post (full text included) | $1.00 per 1,000 |
| Newsletter | $2.00 per 1,000 |
| Comment | $0.30 per 1,000 |

Set a maximum cost per run in Apify and the scraper stops when it is reached.

### Notes

- Subscriber numbers are the rounded figures Substack shows publicly (for example "271K+"). Substack does not publish exact counts.
- Paid posts: only the free preview text is public, so `isFullText` is `false` for them.
- Only public data is collected. Respect authors' copyright when you reuse full text.

### More tools from the same developer

- [Google Trends Scraper](https://apify.com/leanpilar/google-trends): interest over time, by region, related queries and trending searches.
- [Google Ads Transparency Scraper](https://apify.com/leanpilar/google-ads-transparency): every Google ad a competitor runs, with text, landing page, regions and dates.
- [Google AI Mode & AI Overview Tracker](https://apify.com/leanpilar/google-ai-answer-tracker): see if Google's AI answers mention and cite your brand.
- [YouTube Transcript & Subtitles Scraper](https://apify.com/leanpilar/youtube-transcripts): transcripts of videos, channels, playlists and Shorts.
- [Google Hotels Scraper](https://apify.com/leanpilar/google-hotels): hotel prices and every booking site's price for your dates.
- [Funded Startups Feed (SEC Form D)](https://apify.com/leanpilar/funded-startups): companies that just raised money, with executives and phone.

# Actor input Schema

## `newsletters` (type: `array`):

Newsletters to read posts from. Any form works: "lenny", "lenny.substack.com", a custom domain like "www.lennysnewsletter.com", or a profile link "substack.com/@lenny". Each post is one charged result.

## `postUrls` (type: `array`):

Single posts, e.g. https://name.substack.com/p/post-title.

## `searchQueries` (type: `array`):

Keywords to search across every Substack newsletter. Finds matching posts and (if enabled below) matching newsletters.

## `categories` (type: `array`):

Substack's top newsletters per category, with subscriber counts and subscription prices. Each newsletter is one charged result.

## `sort` (type: `string`):

Newest posts first, or the newsletter's most liked posts first.

## `maxPostsPerNewsletter` (type: `integer`):

0 = the whole archive.

## `searchInNewsletter` (type: `string`):

Optional keyword. Uses each newsletter's own archive search.

## `publishedAfter` (type: `string`):

Date like 2026-01-31, or relative like "30 days", "2 weeks", "6 months".

## `publishedBefore` (type: `string`):

Date like 2026-06-30, or relative like "7 days".

## `audience` (type: `string`):

Paid posts only include the free preview text.

## `postTypes` (type: `array`):

Leave empty for all.

## `minLikes` (type: `integer`):

Skip posts with fewer likes.

## `includeFullText` (type: `boolean`):

Full article as clean Markdown, ready for AI tools. Paid posts only include the free preview. No extra charge.

## `includeComments` (type: `boolean`):

Adds comment rows (with replies) for each post. Each comment is one charged result.

## `maxCommentsPerPost` (type: `integer`):

Best comments first, replies included.

## `maxPostsPerSearch` (type: `integer`):

Matching posts from all of Substack. 0 = newsletters only.

## `findNewsletters` (type: `boolean`):

Newsletters whose name or author matches, with subscriber counts.

## `maxNewslettersPerSearch` (type: `integer`):

Substack search usually returns up to about 10-30 newsletters per term.

## `leaderboard` (type: `string`):

Substack's Top overall list, or the bestseller list ranked by paid subscribers.

## `maxNewslettersPerCategory` (type: `integer`):

Ranked from the top of the leaderboard.

## `minSubscribers` (type: `integer`):

Skip newsletters with fewer subscribers (Substack shows rounded numbers like 271K+).

## `maxConcurrency` (type: `integer`):

How many requests run at the same time.

## `useApifyProxy` (type: `boolean`):

Not needed normally. Turn on only if Substack starts refusing requests.

## Actor input object example

```json
{
  "newsletters": [
    "lenny"
  ],
  "sort": "new",
  "maxPostsPerNewsletter": 20,
  "audience": "all",
  "minLikes": 0,
  "includeFullText": true,
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "maxPostsPerSearch": 20,
  "findNewsletters": true,
  "maxNewslettersPerSearch": 10,
  "leaderboard": "all",
  "maxNewslettersPerCategory": 50,
  "minSubscribers": 0,
  "maxConcurrency": 4,
  "useApifyProxy": false
}
```

# Actor output Schema

## `results` (type: `string`):

One row per post, comment or newsletter. The type column tells them apart.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "newsletters": [
        "lenny"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("leanpilar/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "newsletters": ["lenny"] }

# Run the Actor and wait for it to finish
run = client.actor("leanpilar/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "newsletters": [
    "lenny"
  ]
}' |
apify call leanpilar/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,leanpilar/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/WKsb3aJ5jB6Cl4eQ8/builds/JMediUqNxjo4NddUR/openapi.json
