# Substack Scraper – Posts, Comments, Publications & Authors (`teodor_banea/substack-scraper`) Actor

Scrape Substack newsletters without logging in: posts with full HTML and plain text, likes and restacks, flattened comment threads, paywall flags, category leaderboards and author profiles as clean typed rows.

- **URL**: https://apify.com/teodor_banea/substack-scraper.md
- **Developed by:** [Banea Teodor](https://apify.com/teodor_banea) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper — Posts, Comments, Publications & Authors

Scrape any **Substack newsletter** without logging in. Point the Actor at publications and get
every post as a clean row — title, authors, date, likes, comments, restacks, tags, paywall flag
and the **full post as HTML and plain text**. Add the comment threads, discover publications
through Substack's category leaderboards, or look up author profiles.

### What does Substack Scraper do?

It reads Substack's public JSON endpoints (the same ones the website uses), so it works for
publications on `name.substack.com` and on custom domains alike. Four modes:

| Mode | Input | One row per |
|---|---|---|
| `posts` | publication URLs or subdomains | post (plus comments when enabled) |
| `postUrls` | specific post links | post (plus comments when enabled) |
| `publications` | category slugs (`technology`, `culture`, …) | publication on the leaderboard |
| `authors` | Substack handles | author profile |

### Why use this Actor?

- **Full content, not just metadata.** `bodyHtml` and `bodyText` (subscribe widgets removed)
  for every free post; paid posts come back as the public preview with `isPaywalled: true`.
- **Comment threads, flattened.** Every comment is a row with `parentCommentId` and `depth`,
  so threads survive CSV export.
- **Scheduling-friendly.** `postedAfter` stops pagination at the date you give, so a daily run
  over 50 newsletters costs cents.
- **Discovery built in.** Category leaderboards give you ranked publications with author,
  tagline, tier and subscriber signals — the input for your next posts run.
- **One constant row shape per record type.** Absent values are `null`, never missing.
  Managed proxies, honest per-publication shortfall reporting, no $0.09 start fee.

### How much does it cost?

**$2.50 per 1,000 rows** (posts, comments, publications or authors). No per-run fee; proxies
and compute included.

| Run | Rows | Cost |
|---|---|---|
| 100 posts with full content, no comments | 100 | $0.25 |
| 100 posts with content and ~10 comments each | ~1,100 | ~$2.75 |
| Top 50 technology publications | 50 | $0.13 |
| Daily run, 20 newsletters, `postedAfter` yesterday (~15 new posts) | 15 | $0.04 |

Keep `maxCommentsPerPost` low (default 25) — every comment is a billed row.
Paid (subscriber-only) posts return no comment rows: Substack serves their comment threads only to
subscribers, so `includeComments` adds rows for free posts only.

### Input

| Field | Type | Description |
|---|---|---|
| `mode` | string | `posts` (default), `postUrls`, `publications` or `authors`. |
| `publicationUrls` | array | Publications for `posts` mode: `https://lenny.substack.com`, a custom domain, a bare subdomain (`lenny`) or any post URL. Up to 200. |
| `postUrls` | array | Post links for `postUrls` mode: `https://<publication>/p/<slug>` or `/i/<id>`. Up to 1,000. |
| `categories` | array | Category slugs or names for `publications` mode (e.g. `technology`, `business`, `finance`). |
| `authorHandles` | array | Handles for `authors` mode: `lenny`, `@lenny` or `https://substack.com/@lenny`. |
| `maxPostsPerPublication` | integer | Posts per publication. Default 50, max 5,000. |
| `sort` | string | `new` (default) or `top` (most liked first). |
| `searchTerm` | string | Substack's own archive search (title and body). |
| `postedAfter` / `postedBefore` | string | `YYYY-MM-DD` window (UTC). With `sort: new`, `postedAfter` stops pagination early. |
| `audienceFilter` | string | `all` (default), `free` or `paid`. |
| `includeContent` | boolean | Fetch each post for `bodyHtml`/`bodyText` (one extra request per post). Default `true`. |
| `includeComments` | boolean | Fetch comment threads (one extra request and up to `maxCommentsPerPost` billed rows per post). Default `false`. |
| `maxCommentsPerPost` | integer | Default 25, max 2,000. |
| `maxPublicationsPerCategory` | integer | Default 50, max 1,000. |

Example input:

```json
{
    "publicationUrls": ["https://www.astralcodexten.com", "lenny"],
    "maxPostsPerPublication": 20,
    "postedAfter": "2026-09-01",
    "includeContent": true
}
```

### Output

A post row (`recordType: "post"`):

```json
{
    "recordType": "post",
    "postId": 219204484,
    "url": "https://www.astralcodexten.com/p/top-forecasters-on-the-plague",
    "slug": "top-forecasters-on-the-plague",
    "publicationName": "Astral Codex Ten",
    "publicationHost": "www.astralcodexten.com",
    "publicationSubdomain": "astralcodexten",
    "publicationUrl": "https://www.astralcodexten.com",
    "publicationTagline": "P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary.",
    "publicationFreeSubscriberCount": null,
    "title": "Top Forecasters On The Plague",
    "subtitle": "...",
    "description": "...",
    "authorNames": ["Scott Alexander"],
    "authorHandles": ["astralcodexten"],
    "postDate": "2026-10-07T21:13:16.863Z",
    "type": "newsletter",
    "audience": "everyone",
    "isPaywalled": false,
    "wordCount": 880,
    "likes": 243,
    "commentCount": 85,
    "restacks": 10,
    "coverImageUrl": "https://substack-post-media.s3.amazonaws.com/public/images/26f61ee3_1200x655.jpeg",
    "tags": [],
    "sectionName": null,
    "podcastUrl": null,
    "podcastDurationSeconds": null,
    "bodyHtml": "<p>A technician at a Siberian lab has <a href=\"https://apnews.com/...\">died of yersinia infection aka plague</a>. ...</p>",
    "bodyText": "A technician at a Siberian lab has died of yersinia infection aka plague. ...",
    "contentStatus": "full",
    "source": "https://www.astralcodexten.com",
    "scrapedAt": "2026-10-08T14:30:00.000Z"
}
```

A comment row (`recordType: "comment"`):

```json
{
    "recordType": "comment",
    "commentId": 355440418,
    "postId": 219204484,
    "postUrl": "https://www.astralcodexten.com/p/top-forecasters-on-the-plague",
    "publicationHost": "www.astralcodexten.com",
    "parentCommentId": null,
    "depth": 0,
    "authorName": "Sasha Putilin",
    "authorHandle": "psychotechnology",
    "authorUserId": 74588887,
    "authorPhotoUrl": "https://bucketeer.s3.amazonaws.com/public/images/27b067e3.png",
    "authorBestsellerTier": null,
    "body": "Russian here! If there are secondary cases, ...",
    "date": "2026-10-07T21:22:05.209Z",
    "editedAt": null,
    "likes": 15,
    "childrenCount": 2,
    "scrapedAt": "2026-10-08T14:30:00.000Z"
}
```

Publication rows carry `name`, `host`, `url`, `authorName`, `tagline`, `logoUrl`,
`rankingDetail` (e.g. "Tens of thousands of paid subscribers"), `tier`, `categorySlug`,
`categoryRank`, `language`, `createdAt`; `freeSubscriberCount` is usually `null` because the
leaderboard only publishes the ranking text. Author rows carry `name`, `handle`, `profileUrl`, `bio`, `subscriberCount`,
`followerCount`, `bestsellerTier`, `primaryPublicationHost`, `publicationHosts`, `links`.

`contentStatus` is `full` (free post), `preview` (paid post — public preview only),
`not_fetched` (`includeContent` off) or `failed` (the post page could not be fetched; the
archive summary is kept).

### Tips and tricks

- **Mixing posts and comments:** both land in the same dataset; filter on `recordType`, or use
  the "Comments" view in the Console.
- **Daily monitoring:** `sort: "new"` + `postedAfter` = yesterday. With `sort: "top"` the archive
  is not in date order, so the whole archive (up to the page limit) is scanned and filtered.
- **Paid posts** are flagged, never bypassed: `bodyHtml` holds what Substack shows logged-out
  readers, and their comment threads are not public either (0 comment rows even when
  `commentCount` is higher).
- **Publication metadata** (`publicationName`, tagline, subscriber count) comes from the
  publication homepage; a few publications hide subscriber counts, in which case the field is `null`.
- **Search** uses each publication's own archive search; there is no cross-Substack keyword
  search without a session, and this Actor deliberately stays session-free.
- **Failures** (unknown slug, deleted post, unreachable publication, unknown category with the
  list of valid slugs) are stored under the `ERRORS` key of the run's key-value store, and the
  status message reports every shortfall.

### Integrations

Export the dataset as JSON, CSV or Excel from the Apify Console, or pull it through the API:

```
https://api.apify.com/v2/datasets/<datasetId>/items?format=csv
```

Connect the Actor to Google Sheets, Zapier, Make, Slack, Notion or a webhook with Apify
integrations, or call it from your own code with the Apify API client (JavaScript/Python) and
schedule it to run daily.

# Changelog

This Actor's version history is a separate document: https://apify.com/teodor_banea/substack-scraper/changelog.md

# Actor input Schema

## `mode` (type: `string`):

Select the kind of data to collect. Each mode reads its own target field below.

## `publicationUrls` (type: `array`):

Enter one publication per line for the posts mode: a Substack URL (https://lenny.substack.com), a custom domain (https://www.astralcodexten.com), a bare subdomain (lenny) or any post URL (its publication is used). Up to 200.

## `postUrls` (type: `array`):

Enter post links for the postUrls mode: https://<publication>/p/<slug> or https://<publication>/i/<id>. Up to 1,000.

## `categories` (type: `array`):

Enter Substack category slugs or names for the publications mode, e.g. technology, culture, business, finance, science, health. Unknown categories are reported with the list of valid slugs.

## `authorHandles` (type: `array`):

Enter Substack user handles for the authors mode: handle, @handle or https://substack.com/@handle. Up to 500.

## `maxPostsPerPublication` (type: `integer`):

Stop after this many posts for each publication.

## `sort` (type: `string`):

Select the archive order: newest first, or top (most liked) first.

## `searchTerm` (type: `string`):

Only posts matching this term in the publication's own archive search (title and body).

## `postedAfter` (type: `string`):

Only include posts on or after this date (YYYY-MM-DD, UTC). With the newest-first sort, pagination stops at this date, which keeps scheduled runs cheap.

## `postedBefore` (type: `string`):

Only include posts on or before this date (YYYY-MM-DD, UTC).

## `audienceFilter` (type: `string`):

Select which posts to keep: all, free posts only, or paid (subscriber-only) posts only.

## `includeContent` (type: `boolean`):

Fetch each post (one extra request per post) and add bodyHtml and bodyText. Paid posts return the public preview only.

## `includeComments` (type: `boolean`):

Fetch each post's comment thread (one extra request per post) and add one row per comment, flattened with parentCommentId and depth. Every comment row is a billed result.

## `maxCommentsPerPost` (type: `integer`):

Cap the comment rows per post (best-rated first, replies included).

## `maxPublicationsPerCategory` (type: `integer`):

Stop after this many publications per category leaderboard, best-ranked first.

## Actor input object example

```json
{
  "mode": "posts",
  "publicationUrls": [
    "https://www.astralcodexten.com"
  ],
  "postUrls": [
    {
      "url": "https://www.astralcodexten.com/p/top-forecasters-on-the-plague"
    }
  ],
  "categories": [
    "technology"
  ],
  "authorHandles": [
    "lenny"
  ],
  "maxPostsPerPublication": 20,
  "sort": "new",
  "searchTerm": "forecasting",
  "audienceFilter": "all",
  "includeContent": true,
  "includeComments": false,
  "maxCommentsPerPost": 25,
  "maxPublicationsPerCategory": 25
}
```

# Actor output Schema

## `items` (type: `string`):

Post rows (title, authors, date, paywall flag, likes, comments, restacks, tags, full HTML and plain text) plus comment rows when includeComments is on; publication rows in publications mode; author rows in authors mode. Filter on recordType.

## `itemsCsv` (type: `string`):

The same rows as a CSV download for spreadsheets.

## `errors` (type: `string`):

Publications, posts, categories or profiles that could not be fetched (target, URL, HTTP status, error, timestamp). Only present when something failed; the run's status message reports the shortfall per target.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publicationUrls": [
        "https://www.astralcodexten.com"
    ],
    "categories": [
        "technology"
    ],
    "maxPostsPerPublication": 20,
    "maxPublicationsPerCategory": 25
};

// Run the Actor and wait for it to finish
const run = await client.actor("teodor_banea/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publicationUrls": ["https://www.astralcodexten.com"],
    "categories": ["technology"],
    "maxPostsPerPublication": 20,
    "maxPublicationsPerCategory": 25,
}

# Run the Actor and wait for it to finish
run = client.actor("teodor_banea/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publicationUrls": [
    "https://www.astralcodexten.com"
  ],
  "categories": [
    "technology"
  ],
  "maxPostsPerPublication": 20,
  "maxPublicationsPerCategory": 25
}' |
apify call teodor_banea/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,teodor_banea/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/xeZyF8CszIrsgdE4P/builds/V1EGxxsmAO1sLNs4H/openapi.json