# Substack Scraper – Newsletter Posts, Full Text & Comments (`openharvest/substack-scraper`) Actor

Substack scraper without login: newsletter posts with full text of free posts (paywalled posts flagged, free preview only), likes, comment counts, tags and cover images, full comment threads, publication info (subscribers, author) and category leaderboards. Works with custom domains.

- **URL**: https://apify.com/openharvest/substack-scraper.md
- **Developed by:** [Enes Bulanık](https://apify.com/openharvest) (community)
- **Categories:** Social media, News, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 post scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper – Newsletter Posts, Full Text & Comments

Substack scraper without login: newsletter posts with full text of free posts (paywalled posts flagged, free preview only), likes, comment counts, tags and cover images, full comment threads, publication info (subscribers, author) and category leaderboards. Works with custom domains.

**Price: $1.00 per 1,000 posts, $0.30 per 1,000 comments.** No login, no API key, no browser. Inputs that fail produce a free error row.

### What data do you get?

- **Posts:** title, subtitle, authors, publish/update date, URL, slug, audience (`everyone` / `only_paid` / `founding`), `paywalled` flag, likes (reactions), restacks, comment count, word count, tags, section, cover image, language, podcast audio URL and duration for podcast posts
- **Full text:** the post body as clean plain text (`bodyText`), optionally the original HTML (`bodyHtml`). Free posts come in full. Paid posts include only the free preview Substack shows to non-subscribers (`paywalled: true`, `fullTextIncluded: false`).
- **Comments:** comment ID, post ID, author display name, text, date, edit date, likes, `parentId`, `depth`, reply count, a direct comment link and an `isPostAuthor` flag
- **Publication info:** name, URL, subdomain, custom domain, description, author name, language, logo, subscriber count (when Substack shows it publicly), paid-subscriber label, bestseller tier, first post date
- **Discovery:** the top publications of any Substack category from the **Bestseller (paid)** and **Rising (trending)** leaderboards, or the full category list, with rank

### Use cases

- **Newsletter and creator research:** find the top newsletters in a niche and compare audience size, posting frequency and engagement.
- **Content monitoring:** follow competitors, analysts or journalists and pull every new post on a schedule (`onlyNewerThan: "1 day"`).
- **AI, RAG and summarization:** feed full post text into LLM pipelines, knowledge bases or daily digests.
- **Audience and sentiment analysis:** export comment threads with parent/child structure.
- **Sponsorship and PR outreach lists:** shortlist publications per category by rank and subscriber count (no personal emails are collected).

### How to use it

1. Click **Try for free** or **Start**.
2. Choose a **Mode**: `posts` (default), `postDetails`, `comments` or `discover`.
3. Add newsletters to **Publications** (`lenny`, `lenny.substack.com`, `https://www.astralcodexten.com` or `@handle`) or post links to **Post URLs**.
4. Set **Max posts per publication**, **Only posts newer than** and **Max items**, then click **Start**. Export as JSON, CSV or Excel, or connect via API, Make, Zapier, n8n or Google Sheets.

### Input

| Field | Mode | Description |
|---|---|---|
| `mode` | all | `posts` (default), `postDetails`, `comments`, `discover` |
| `publications` | posts, comments | Subdomains, URLs, custom domains or `@handles` |
| `postUrls` | postDetails, comments | `https://name.substack.com/p/slug`, custom-domain post links, `open.substack.com/pub/name/p/slug`, `substack.com/home/post/p-123` |
| `category` | discover | Category slug, name or ID: `technology`, `business`, `finance`, `culture`, `us-politics`, `science`, `health`, `crypto`, `history`, `sports`, `music`, `food`, … (subcategories too) |
| `ranking` | discover | `paid` (Bestsellers), `trending` (Rising) or `all` |
| `onlyNewerThan` | posts, comments | `2026-01-01` or `7 days`, `1 month` … |
| `maxPostsPerPublication` | posts, comments | Newest N posts per publication. Default `20`, `0` = whole archive |
| `includeFullText` | posts | Load `bodyText` for each post. Default `true` |
| `includeHtml` | posts, postDetails | Also save `bodyHtml`. Default `false` |
| `includeComments` | posts, postDetails | Also return each post's comments. Default `false` |
| `maxCommentsPerPost` | all | Default `100`, `0` = all |
| `includePublicationInfo` | posts | One publication row per newsletter. Default `true` |
| `maxItems` | all | Total item limit (= cost cap). Default `100` |

Example inputs:

```json
{ "publications": ["lenny", "https://www.astralcodexten.com", "@noahpinion"], "maxPostsPerPublication": 10, "onlyNewerThan": "30 days" }
```

```json
{ "mode": "postDetails", "postUrls": ["https://www.astralcodexten.com/p/open-thread-454"], "includeComments": true, "maxCommentsPerPost": 200 }
```

```json
{ "mode": "discover", "category": "finance", "ranking": "paid", "maxItems": 100 }
```

### Output

Every item has a `type`: `post`, `comment`, `publication` or `error` (free). Real output (shortened):

**Post** (paid post, so only the free preview is included)

```json
{
    "type": "post",
    "publication": "Lenny's Newsletter",
    "publicationId": 10845,
    "publicationUrl": "https://www.lennysnewsletter.com",
    "postId": 216168140,
    "url": "https://www.lennysnewsletter.com/p/advanced-evals-how-to-find-and-fix",
    "slug": "advanced-evals-how-to-find-and-fix",
    "title": "Advanced evals: How to find (and fix) hidden AI failures in your product",
    "subtitle": "Why you should never skip error discovery",
    "authors": ["Hamel Husain", "Shreya Shankar"],
    "publishedAt": "2026-09-22T12:45:14.998Z",
    "updatedAt": "2026-09-22T19:35:47.938Z",
    "postType": "newsletter",
    "audience": "only_paid",
    "paywalled": true,
    "likes": 350,
    "restacks": 20,
    "commentCount": 4,
    "wordcount": 3807,
    "description": "Why you should never skip error discovery",
    "bodyText": "👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice. …",
    "fullTextIncluded": false,
    "coverImage": "https://substackcdn.com/image/fetch/…/6f6018cf-fa53-4f40-99d9-b1c9980dd031_1456x970.png",
    "tags": ["AI"],
    "section": null,
    "language": "en",
    "podcastUrl": null,
    "podcastDurationSec": null
}
```

Free posts look the same with `"paywalled": false`, `"fullTextIncluded": true` and the whole article in `bodyText`. For example, "An Open Letter To Steven Pinker On AI" on Astral Codex Ten returns 64,000 characters.

**Comment**

```json
{
    "type": "comment",
    "id": 345437906,
    "postId": 215593613,
    "postUrl": "https://www.astralcodexten.com/p/fall-meetups-2026-call-for-meetup",
    "postTitle": "Fall Meetups 2026 - Call For Meetup Organizers",
    "publication": "Astral Codex Ten",
    "author": "Laniakea",
    "isPostAuthor": false,
    "body": "I am hosting a Petrov Day ceremony with friends (i.e. engineering students) next monday evening in Leuven, Belgium. …",
    "date": "2026-09-25T10:02:50.827Z",
    "editedAt": "2026-09-25T10:03:34.615Z",
    "likes": 1,
    "parentId": null,
    "depth": 1,
    "replyCount": 0,
    "url": "https://www.astralcodexten.com/p/fall-meetups-2026-call-for-meetup/comment/345437906"
}
```

**Publication** (mode `discover`, Finance / Rising)

```json
{
    "type": "publication",
    "rank": 1,
    "category": "Finance",
    "ranking": "trending",
    "publicationId": 630791,
    "name": "Lord Fed's Gazette",
    "url": "https://www.lordfed.co.uk",
    "subdomain": "lordfed",
    "customDomain": "www.lordfed.co.uk",
    "description": "Market analysis, idea generation and education from a London-based portfolio manager. …",
    "authorName": "Lord Fed",
    "language": "en",
    "freeSubscriberCount": 24000,
    "subscriberCountText": "Tens of thousands of subscribers",
    "paidSubscribersText": "Thousands of paid subscribers",
    "paidSubscriptionsEnabled": true,
    "bestsellerTier": 1000,
    "firstPostDate": "2022-04-03T01:25:01.568Z",
    "archiveUrl": "https://www.lordfed.co.uk/archive"
}
```

**Error (free)**

```json
{ "type": "error", "input": "https://example.com", "errorCode": "NOT_FOUND", "error": "https://example.com does not look like a Substack publication (no Substack data on the page)." }
```

Possible `errorCode` values: `INVALID_INPUT`, `NOT_FOUND`, `FETCH_FAILED`, `NO_COMMENTS`, `NO_RESULTS`.

Dataset views in Apify Console: **Overview**, **Full text**, **Comments**, **Publications & errors**.

### Pricing

Pay per event, with platform usage included.

| Event | Price | When it is charged |
|---|---|---|
| `post` | **$0.001** per post ($1.00 / 1,000) | Each post saved (with or without full text) |
| `comment` | **$0.0003** per comment ($0.30 / 1,000) | Each comment saved |
| `publication` | **$0.001** per publication | Each publication-info or leaderboard row |
| Errors / no results | **Free** | Invalid inputs, missing newsletters or posts |

| Job | Cost |
|---|---|
| 20 latest posts of 1 newsletter with full text + publication info | **$0.021** |
| 1,000 posts | **$1.00** |
| A post with 800 comments | **$0.24** |
| Top 100 publications in a category | **$0.10** |

Set **Max items** and Apify's **Max cost per run**. The Actor stops cleanly at the limit.

### Use with AI agents (MCP)

Add the Actor as a tool in the [Apify MCP server](https://mcp.apify.com) (`https://mcp.apify.com?tools=openharvest/substack-scraper`), or let agents find it with `search-actors` ("substack").

Example `call-actor` input for "Summarize what Noahpinion wrote this week":

```json
{ "mode": "posts", "publications": ["@noahpinion"], "onlyNewerThan": "7 days", "includeFullText": true, "includePublicationInfo": false, "maxItems": 10 }
```

Example input for "Find the top 25 rising tech newsletters":

```json
{ "mode": "discover", "category": "technology", "ranking": "trending", "maxItems": 25 }
```

### FAQ

**Can it get the full text of paid posts?**
No. The Actor only reads what Substack shows publicly. For paid posts you get the free preview (`paywalled: true`, `fullTextIncluded: false`) and all metadata (title, likes, comment count, word count …).

**Does it work with custom domains?**
Yes. `www.astralcodexten.com`, `www.lennysnewsletter.com` and other custom domains work like `name.substack.com`. Author handles such as `@noahpinion` are resolved to their main publication.

**How many posts can I get?**
The whole public archive. Set `maxPostsPerPublication` to `0` and raise `maxItems`.

**Why do some posts with comments return none?**
Some publications limit their discussion to paid subscribers. Those comments are not public, so they're skipped and you aren't charged.

**Where does the subscriber count come from?**
It's the number Substack displays publicly on the newsletter (for example "1,200,000 subscribers"). Some publications hide it, and then it is `null`.

**How fresh is the data?**
It is live: every run reads Substack's public endpoints directly.

### Is it legal to scrape Substack?

The Actor reads only **publicly available data**, the same posts, previews and comments anyone can see in a browser without logging in. It does not bypass paywalls and does not access subscriber-only content. It does not collect emails, user IDs, profile photos or other contact details. Comment authors appear only by their public display name. Posts and comments are copyrighted by their authors, so respect their rights, Substack's Terms of Use, and laws such as GDPR when you store or republish data. **Do not use the data for spam or unsolicited outreach.**

### Related actors

- Podcast Scraper – Apple Podcasts Search, Charts, Episodes & RSS Feeds *(coming soon)*
- Hacker News Scraper – Stories, Comments, Search & Who Is Hiring *(coming soon)*
- Telegram Channel Scraper – Public Posts, Views & Media *(coming soon)*
- RSS Feed Reader *(coming soon)*

### Support & feedback

Found a bug or a missing field? Open an issue in the **Issues** tab with the publication or post URL.

# Actor input Schema

## `mode` (type: `string`):

<b>posts</b> = latest posts of each publication (newest first). <b>postDetails</b> = specific posts by URL. <b>comments</b> = comment threads of posts (by URL, or of the latest posts of each publication). <b>discover</b> = top publications of a Substack category (leaderboards).

## `publications` (type: `array`):

Modes <b>posts</b> and <b>comments</b>: Substack newsletters. Accepted: subdomain (<code>lenny</code>), <code>lenny.substack.com</code>, any URL of the newsletter, custom domains (<code>www.astralcodexten.com</code>) or an author handle (<code>@lenny</code>).

## `postUrls` (type: `array`):

Modes <b>postDetails</b> and <b>comments</b>: post links such as <code>https://name.substack.com/p/post-slug</code>, custom-domain post links, <code>open.substack.com/pub/name/p/slug</code> or <code>substack.com/home/post/p-123456</code>.

## `category` (type: `string`):

Mode <b>discover</b>: Substack category slug, name or ID, e.g. <code>technology</code>, <code>business</code>, <code>finance</code>, <code>culture</code>, <code>us-politics</code>, <code>science</code>, <code>health</code>, <code>crypto</code>, <code>history</code>, <code>sports</code>, <code>music</code>, <code>food</code>. Subcategories work too.

## `ranking` (type: `string`):

Mode <b>discover</b>: which list to read. <b>paid</b> = Bestseller leaderboard (most paid subscribers). <b>trending</b> = Rising leaderboard. <b>all</b> = all publications Substack lists for the category.

## `onlyNewerThan` (type: `string`):

Modes <b>posts</b> and <b>comments</b>: skip posts older than this. Absolute date (YYYY-MM-DD) or relative period such as <code>7 days</code>, <code>1 month</code>.

## `maxPostsPerPublication` (type: `integer`):

Modes <b>posts</b> and <b>comments</b>: how many of the newest posts to read from each publication. 0 = the whole archive.

## `includeFullText` (type: `boolean`):

Mode <b>posts</b>: load the body of each post as plain text (<code>bodyText</code>). Free posts get the full article; paid posts only the free preview (<code>paywalled: true</code>). Mode <b>postDetails</b> always includes it.

## `includeHtml` (type: `boolean`):

Also save the original HTML of the post body in <code>bodyHtml</code> (only when the full text is loaded).

## `includeComments` (type: `boolean`):

Modes <b>posts</b> and <b>postDetails</b>: after each post, also return its public comments (each comment is charged as a 'comment' event).

## `maxCommentsPerPost` (type: `integer`):

Limit comments per post (threads are returned depth-first in Substack's 'best first' order). 0 = all comments.

## `includePublicationInfo` (type: `boolean`):

Mode <b>posts</b>: add one item per publication with name, author, description, subscriber count, language and logo (charged as a 'publication' event).

## `maxItems` (type: `integer`):

Maximum number of items in total (posts + comments + publications). You pay per item, so this also caps your cost.

## `proxyConfiguration` (type: `object`):

Usually not needed. Enable Apify Proxy only if you scrape thousands of posts and see HTTP 429 (rate limit) warnings.

## Actor input object example

```json
{
  "mode": "posts",
  "publications": [
    "lenny",
    "https://www.astralcodexten.com",
    "@noahpinion"
  ],
  "postUrls": [
    "https://www.astralcodexten.com/p/open-thread-454"
  ],
  "category": "finance",
  "ranking": "paid",
  "onlyNewerThan": "30 days",
  "maxPostsPerPublication": 50,
  "includeFullText": true,
  "includeHtml": false,
  "includeComments": true,
  "maxCommentsPerPost": 50,
  "includePublicationInfo": true,
  "maxItems": 50,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All scraped items in the default dataset (JSON, CSV, Excel and more via the dataset export).

## `files` (type: `string`):

Records saved in the default key-value store, e.g. the run summary.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "mode": "posts",
    "publications": [
        "astralcodexten",
        "https://www.lennysnewsletter.com"
    ],
    "maxPostsPerPublication": 5,
    "maxItems": 20,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("openharvest/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "mode": "posts",
    "publications": [
        "astralcodexten",
        "https://www.lennysnewsletter.com",
    ],
    "maxPostsPerPublication": 5,
    "maxItems": 20,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("openharvest/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "mode": "posts",
  "publications": [
    "astralcodexten",
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 5,
  "maxItems": 20,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call openharvest/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,openharvest/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ceU38wIubliWPLVsU/builds/ecG1mzer6gPwqZWhE/openapi.json