# Substack Scraper: Posts, Full Text, Comments & Podcasts (`nekruz/substack-scraper`) Actor

Export posts from any Substack publication, including custom domains: title, authors, date, paywall status, likes, full HTML and plain text, comments and podcast audio links. Search or filter by date. Pay per post returned.

- **URL**: https://apify.com/nekruz/substack-scraper.md
- **Developed by:** [Nekruz Kasimov](https://apify.com/nekruz) (community)
- **Categories:** News, Social media
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.10 / 1,000 post with full texts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper: Posts, Full Text, Comments & Podcasts

Pull posts from any Substack publication, including ones on custom domains. For each post you get the title, subtitle, authors, date, type (newsletter, podcast, thread, video), paywall status, likes and reactions, comment count, restacks, cover image, podcast audio link, tags, the full body as HTML and plain text, and optionally the whole comment thread.

Uses Substack's public JSON endpoints, so it is fast and does not need a browser.

### Use cases

- Archive or migrate a newsletter.
- Research a topic across many newsletters and feed the text to an LLM or RAG pipeline.
- Track engagement (likes, comments, restacks) over time for competitive analysis.
- Monitor an author or industry for new posts on a schedule.

### Input

| Field | Description |
|---|---|
| `publications` | URLs or subdomains: `https://www.lennysnewsletter.com`, `lenny`, `stratechery.substack.com`. |
| `maxPosts` | Per publication. Default `50`, maximum `10000`. |
| `sort` | `new` (the default) or `top`. |
| `search` | Substack's own full-text search filter. Results come in Substack's relevance order, and paging ends after two empty result windows in a row. |
| `postedAfter` | ISO date. With `sort: new` and no `search`, paging stops as soon as older posts appear, so daily runs stay cheap. |
| `includeBody` | Full HTML and text per post. On by default. Paywalled posts return the free preview and `isBodyTruncated: true`. |
| `includeComments` | Flattened comment thread per post with depth and parent id. Off by default. |
| `types` | Limit to `newsletter`, `podcast`, `thread`, `video`. |

### Output

```json
{
  "publication": "Lenny's Newsletter",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "postId": "215694124",
  "slug": "all-of-the-lenny-and-friends-summit",
  "title": "All of the Lenny & Friends Summit talks are now online!",
  "subtitle": "Plus, some reflections and takeaways from the day",
  "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "type": "newsletter",
  "audience": "everyone",
  "isPaywalled": false,
  "postDate": "2026-09-29T13:15:57.512Z",
  "authors": [{ "name": "Lenny Rachitsky", "handle": "lenny", "id": 1849774 }],
  "wordCount": 1276,
  "reactionCount": 288,
  "commentCount": 7,
  "restacks": 12,
  "bodyHtml": "<p>...</p>",
  "bodyText": "...",
  "isBodyTruncated": false,
  "comments": [{ "id": 1, "depth": 0, "author": "...", "body": "..." }]
}
```

A post whose body or comments were requested but could not be fetched is still returned. Its `bodyHtml`, `bodyText` and `isBodyTruncated` (or `comments`) are `null` and a `bodyError` (or `commentsError`) field says why, for example `HTTP 503`.

The `OUTPUT` record lists each publication with posts found, posts pushed, and any body or comment fetch failures. It contains:

- `publications`: one entry per input entry, with `input`, `status` and, except for `unrecognized` entries, `base` (the publication's URL after redirects), `postsFound`, `postsPushed`, `bodyFailures`, `commentFailures`, `detailsRecovered` (bodies or comments that failed once and succeeded on the automatic retry), `bodiesStopped`, `commentsStopped` (see Pricing) and, for `error` and `partial`, `error`. A `duplicate` entry that was skipped before any request carries only `input`, `base`, `status` and `duplicateOf`.
- `postsPushed`: posts saved to the dataset. `postsCharged`: posts billed under pay-per-event pricing, otherwise `null`.
- `chargeLimitReached`: `true` if the run stopped at your maximum charge per run.

Each publication's `status` is one of:

- `ok`: the archive was read.
- `partial`: a later archive page failed after retries. Earlier posts are kept and `error` says which offset failed and how many posts were kept.
- `not_found`: the first archive request returned 404.
- `error`: the first archive request failed after retries; `error` has the reason.
- `unrecognized`: the entry is not a Substack URL or subdomain.
- `duplicate`: the entry is the same publication as an earlier one, so nothing was scraped from it. `duplicateOf` is the first input for that publication.

### Pricing

You pay per post returned. Posts with full text cost more than metadata-only posts; comments are free. A publication entered more than once is scraped and charged once; the repeat entries are marked `duplicate` and cost nothing. Set a maximum charge per run to cap spend; the Actor stops cleanly when it is reached (`chargeLimitReached` is `true` in `OUTPUT`).

A post whose body or comments could not be fetched is not charged. Failed requests are retried (rate limits and server errors get one more try after a short pause); if the body or comments are still missing, the post is returned with `bodyError` or `commentsError` set and counted in `bodyFailures` or `commentFailures`. It still counts toward `maxPosts`. If body or comment requests keep failing (3 of one kind in a row, then a failed retry), the Actor stops requesting that kind for the rest of that publication: `bodiesStopped` or `commentsStopped` is `true` in the publication's `OUTPUT` entry, and the remaining posts are returned uncharged with an error starting "not requested".

### Notes

- Only content visible to a logged-out reader is returned. Paid posts give the free preview.
- The Actor requests 20 posts per archive page; large archives take a few seconds per hundred posts.
- If a later archive page fails after retries, the publication is marked `partial` and keeps its earlier posts. The status message then reads, for example, "Done: 20 posts from 1 publications (1 incomplete, see OUTPUT)". A failed first page is `error` or `not_found` instead.
- Publications are compared by host, ignoring case, a trailing slash and any path, so `lenny`, `lenny.substack.com` and `https://LENNY.substack.com/p/some-post` are one publication. An entry that redirects to a host already scraped (for example the `lenny` subdomain and `https://www.lennysnewsletter.com`) is marked `duplicate` too; when the redirect only shows on its first archive request, that one request is all it costs.
- Archive paging stops when a page holds only posts already seen (the publication stays `ok` and a warning is logged). With `search`, such a page counts as an empty result window, so two in a row end the search. The Actor never requests an archive page past offset 50,000.
- When posts were saved without their body or comments, the status message adds "N without body or comments (not charged), see bodyError/commentsError".
- Rate limits (HTTP 429), server errors (5xx) and network errors are tried up to 3 times. The Actor waits for the `Retry-After` the server sends (at most 60 s), otherwise 1.5 s, then 3 s.

# Changelog

This Actor's version history is a separate document: https://apify.com/nekruz/substack-scraper/changelog.md

# Actor input Schema

## `publications` (type: `array`):

Substack URLs or subdomains, one per line. Custom domains work too. Examples: https://www.lennysnewsletter.com, lenny, https://stratechery.substack.com

## `maxPosts` (type: `integer`):

Stop after this many posts per publication. Each archive page holds 20 posts.

## `sort` (type: `string`):

Archive order: newest posts first, or Substack's most popular first.

## `search` (type: `string`):

Only posts matching this text (Substack's own search).

## `postedAfter` (type: `string`):

ISO date (YYYY-MM-DD). Keep only posts on or after this date. Stops paging early when sorted by newest.

## `includeBody` (type: `boolean`):

Fetch each post's full HTML and plain text (one extra request per post). Paywalled posts return the free preview only.

## `includeComments` (type: `boolean`):

Fetch the comment thread for each post (one extra request per post).

## `types` (type: `array`):

Limit to these types.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPosts": 50,
  "sort": "new",
  "includeBody": true,
  "includeComments": false,
  "types": []
}
```

# Actor output Schema

## `items` (type: `string`):

One record per Substack post, with body, reactions and optional comments.

## `summary` (type: `string`):

The OUTPUT record: per-target statuses, counts, what was charged, and why items were skipped or missing.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("nekruz/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": ["https://www.lennysnewsletter.com"] }

# Run the Actor and wait for it to finish
run = client.actor("nekruz/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com"
  ]
}' |
apify call nekruz/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nekruz/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/UcDzR4EVh8wEA02Ls/builds/wxAltYCiiHGoYdKhr/openapi.json