# Substack Scraper - Posts, Full Text, Comments & Likes (`wakachobi/substack-scraper`) Actor

Scrape any Substack newsletter (custom domains too): full archive, post text, likes, comments, restacks, word count, authors, plus post and publication search and category leaderboards.

- **URL**: https://apify.com/wakachobi/substack-scraper.md
- **Developed by:** [Cobi Hilliard](https://apify.com/wakachobi) (community)
- **Categories:** News, Social media
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper: posts, full text, likes, comments and newsletters

Scrape any **Substack** newsletter, including ones on **custom domains**. Get a publication's **full archive**,
single posts with their **full text**, the **comments** under each post, and search all of Substack for **posts**
or **newsletters**, or pull the **top newsletters in a category**. Every post comes with likes, comment count,
restacks, word count, authors and paywall status. No login and no API key.

### What data you get

- **Posts:** title, subtitle, URL, slug and ID; publish and update dates (ISO 8601); post type (article, podcast,
  thread); audience (`everyone`, `only_paid`, `only_subscribers`)
- Author name plus every byline (name, handle, profile link, guest flag)
- Publication name, URL and ID
- **Likes**, reactions by emoji, **comment count**, **restacks** and **word count**
- Tags, language, cover image, podcast duration, the short public preview text
- **Full post text** as clean plain text (optional HTML). Paid posts give only the free public preview that Substack
  shows to everyone, flagged with `textIsPreviewOnly`
- **Comments** (optional) as separate results: text, date, author name and handle, likes, restacks, number of
  replies, `parentId` and `depth` for threads, and whether the post's author wrote it
- **Publications** (from search or category leaderboards): name, URL, subdomain, custom domain, description,
  author, subscriber count as shown by Substack (e.g. "Over 58,000 subscribers"), paid subscriber band, language,
  launch date and logo
- `type` (`post`, `comment` or `publication`) and `source` (which input found the result) on every item

#### Sample result

```json
{
  "type": "post",
  "id": 215694124,
  "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "title": "All of the Lenny & Friends Summit talks are now online!",
  "subtitle": "Plus, some reflections and takeaways from the day",
  "postType": "newsletter",
  "audience": "everyone",
  "isPaid": false,
  "publishedAt": "2026-09-29T13:15:57.512Z",
  "authorName": "Lenny Rachitsky",
  "authors": [{ "name": "Lenny Rachitsky", "handle": "lenny", "profileUrl": "https://substack.com/@lenny", "isGuest": false }],
  "publicationName": "Lenny's Newsletter",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "likes": 291,
  "comments": 7,
  "restacks": 5,
  "wordCount": 1276,
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/9fed347c-..._9213x6142.jpeg",
  "text": "👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice… I’m excited to share that every main-stage talk from the recent Lenny & Friends Summit is now online…",
  "textIsPreviewOnly": false,
  "source": "publication:www.lennysnewsletter.com"
}
```

```json
{
  "type": "comment",
  "id": 349085941,
  "postId": 215694124,
  "postUrl": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "parentId": null,
  "depth": 0,
  "text": "Thank you Lenny for putting this together…",
  "publishedAt": "2026-09-29T23:34:47.245Z",
  "authorName": "Product Mindset",
  "authorHandle": "productmindset",
  "likes": 4,
  "replies": 0
}
```

### How to use

| Field | What it does |
|---|---|
| `publications` | Newsletters to scrape in full: `garymarcus`, `garymarcus.substack.com` or a custom domain like `https://www.lennysnewsletter.com` |
| `postUrls` | Specific posts, e.g. `https://www.astralcodexten.com/p/why-does-ozempic-cure-all-diseases` or `https://substack.com/home/post/p-221631819` |
| `searchQueries` | Keywords searched across all of Substack |
| `searchType` | `posts` (default) or `publications` |
| `categories` | Category leaderboards, e.g. `technology`, `business`, `finance`, `culture`, `politics` |
| `categoryRanking` | `all` (top overall) or `paid` (top paid) |
| `sort` | Archive order: `new` (default) or `top` (most popular first) |
| `maxItems` | Stop after this many results in total (default 50) |
| `maxPostsPerPublication` | Per-newsletter cap so one big archive does not use the whole budget (0 = no cap) |
| `includeFullText` | Post body as plain text (default on). Turn off for a fast metadata-only run |
| `includeHtml` | Also return the body HTML |
| `includeComments` | Save each post's comments and replies as `comment` results |
| `maxCommentsPerPost` | Most-liked comments first (default 50, 0 = all) |
| `onlyFree` | Skip paid and subscriber-only posts |
| `postType` | `all`, `newsletter`, `podcast` or `thread` |
| `startDate` / `endDate` | Only posts published in this range (YYYY-MM-DD) |

Example inputs:

```json
{ "publications": ["garymarcus", "https://www.lennysnewsletter.com"], "maxPostsPerPublication": 100, "maxItems": 200 }
```

```json
{ "postUrls": [{ "url": "https://www.astralcodexten.com/p/why-does-ozempic-cure-all-diseases" }], "includeComments": true, "maxCommentsPerPost": 0 }
```

```json
{ "searchQueries": ["ai agents"], "searchType": "publications", "categories": ["technology"], "maxItems": 100 }
```

A full archive with text costs one request per post, so 1,000 posts take a few minutes. Turn off
`includeFullText` for a fast run: the archive then gives about 20 posts per request with all stats.

### Pricing

**$2 per 1,000 results**, charged only for results saved to your dataset (a post, a comment or a publication each
count as one result). The full 500-post archive of a newsletter costs $1.00. 100 posts with 20 comments each cost
about $4.20.

### Use cases

- Archive and analyse a newsletter's whole back catalogue: which topics and headlines get the most likes,
  comments and restacks
- Competitor and creator research: compare posting frequency, engagement and paywall strategy
- Find newsletters in a niche through search and category leaderboards, with subscriber counts, for sponsorships
  and partnerships
- Collect free post text for RAG pipelines, summarisation and AI training sets
- Audience research: read what commenters say and which comments get the most likes
- Monitor newsletters on a schedule with `startDate` for new posts

### FAQ

**Is it legal?** The scraper only reads public data that any visitor sees without logging in. It never gets around
the paywall: paid posts return the free preview Substack shows to everyone. It does not collect emails or other
contact details. You are responsible for how you use the data, including personal data and copyright.

**Does it work with custom domains?** Yes. Pass the domain (`www.lennysnewsletter.com`, `astralcodexten.com`); the
scraper finds the right address itself.

**Why are subscriber counts text?** Substack only publishes rounded figures ("Over 1,200,000 subscribers").
`subscribersOrderOfMagnitude` gives a number you can sort by.

**Why did I get fewer results than `maxItems`?** The archive, search or leaderboard ran out, or your filters
(`onlyFree`, `postType`, dates) removed posts. Comments count toward `maxItems` too.

**What is not included?** Paid post bodies, Substack Notes and chat, and subscriber lists.

**Can I run it on a schedule?** Yes. Use Apify Schedules with `startDate` to collect new posts daily and send them to
Google Sheets, Slack, Make, Zapier or a webhook with Apify integrations.

### API

Call it from any language with the Apify API or client libraries (`apify-client` for JavaScript and Python), or
connect it to an AI agent through the Apify MCP server.

# Actor input Schema

## `publications` (type: `array`):

Newsletters to scrape, as a subdomain ('garymarcus'), a substack.com address ('garymarcus.substack.com') or a custom domain ('https://www.lennysnewsletter.com'). Returns every post in the archive, newest first, up to the limits below.

## `postUrls` (type: `array`):

Direct links to Substack posts, e.g. 'https://www.lennysnewsletter.com/p/some-post' or 'https://substack.com/home/post/p-123456'.

## `searchQueries` (type: `array`):

Keywords to search across all of Substack. See 'Search for' to choose posts or publications.

## `searchType` (type: `string`):

What the search queries return.

## `categories` (type: `array`):

Substack categories, e.g. 'technology', 'business', 'culture', 'politics', 'finance'. Returns the top newsletters in each category as publication items.

## `categoryRanking` (type: `string`):

Which category leaderboard to read.

## `sort` (type: `string`):

Order in which a publication's archive is read.

## `maxItems` (type: `integer`):

Stop after this many results in total (posts, comments and publications all count).

## `maxPostsPerPublication` (type: `integer`):

Cap per publication so one big archive does not use the whole budget. 0 = no cap.

## `includeFullText` (type: `boolean`):

Fetch each post and include its body as plain text. Paid posts return only the free public preview.

## `includeHtml` (type: `boolean`):

Also include the body as HTML (needs 'Include post text').

## `includeComments` (type: `boolean`):

Also save the public comments of each post (with replies) as separate 'comment' results.

## `maxCommentsPerPost` (type: `integer`):

Most-liked comments first. 0 = all.

## `onlyFree` (type: `boolean`):

Skip posts for paid subscribers.

## `postType` (type: `string`):

Only keep posts of this type.

## `startDate` (type: `string`):

Only posts published on or after this date (YYYY-MM-DD).

## `endDate` (type: `string`):

Only posts published on or before this date (YYYY-MM-DD).

## `proxyConfiguration` (type: `object`):

Datacenter proxy works for most runs.

## Actor input object example

```json
{
  "publications": [
    "garymarcus"
  ],
  "searchType": "posts",
  "categoryRanking": "all",
  "sort": "new",
  "maxItems": 20,
  "maxPostsPerPublication": 0,
  "includeFullText": true,
  "includeHtml": false,
  "includeComments": false,
  "maxCommentsPerPost": 50,
  "onlyFree": false,
  "postType": "all",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "garymarcus"
    ],
    "maxItems": 20,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("wakachobi/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["garymarcus"],
    "maxItems": 20,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("wakachobi/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "garymarcus"
  ],
  "maxItems": 20,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call wakachobi/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,wakachobi/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/gYaDRx4BNTL2PPgud/builds/XR4bHBrh7VbqmNM1S/openapi.json