# Substack Scraper: Newsletters, Posts, Subscribers, Leaderboards (`dottti/substack-scraper`) Actor

Public Substack data with no login. Posts with likes, comments, restacks and free full text as Markdown. Newsletter profiles with subscriber counts, prices and socials. Find newsletters by keyword or category leaderboard. Only-new alerts. Failed inputs are free.

- **URL**: https://apify.com/dottti/substack-scraper.md
- **Developed by:** [Dottti](https://apify.com/dottti) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 result rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper: Newsletters, Posts, Subscribers, Leaderboards

Get public Substack data with no login and no cookies.

- **Posts** from any newsletter: title, date, link, free or paid, likes, comments, restacks, word count, authors, and the full text of free posts as Markdown.
- **Newsletter profiles**: subscriber count where Substack shows it, paid subscriber range, monthly, yearly and founding prices, leaderboard rank, about text and the writer's social links.
- **Discovery**: find newsletters by keyword, or read a category leaderboard (Top Bestsellers or Rising) with stats for each newsletter.

Every input gets an answer. A newsletter that does not exist, a bad link or an unknown category returns a free status row that says what went wrong. The run still ends as SUCCEEDED.

### Who it is for

- **Marketers and sponsors** who need a list of newsletters to sponsor in a niche, with subscriber counts and prices.
- **Researchers and analysts** who track what newsletters publish and how readers react.
- **AI and RAG builders** who need clean newsletter text. Free posts come as Markdown, ready for a vector store.
- **Writers** who watch other newsletters in their niche.

### Paywalls

This actor reads only what Substack shows to a logged-out visitor. Paid posts are never unlocked. A paid post row has all its public stats, `isPaywalled: true` and `textStatus: "paywalled"`, with no full text.

### Input

Pick a mode, then fill the fields for that mode.

| Mode | Fields | You get |
|---|---|---|
| `posts` | `newsletters`, `postUrls`, `maxPostsPerNewsletter`, `includeFullText`, `audience`, `postedAfter` | One row per post |
| `profiles` | `newsletters`, `includeAuthorSocials` | One row per newsletter |
| `discover` | `searchKeywords`, `categories`, `leaderboard`, `maxPublicationsPerSource` | One row per newsletter found |

A newsletter can be written any of these ways:

- `lenny`
- `lenny.substack.com`
- `https://www.lennysnewsletter.com` (custom domains work)
- `@noahpinion` (a writer's handle; the actor finds their main newsletter)

Example: the 50 newest posts from two newsletters, with full text of free posts.

```json
{
    "mode": "posts",
    "newsletters": ["astralcodexten", "https://www.lennysnewsletter.com"],
    "maxPostsPerNewsletter": 50,
    "includeFullText": true
}
```

Example: sponsor research. The top 100 Technology and Finance bestsellers, plus newsletters about climate.

```json
{
    "mode": "discover",
    "categories": ["technology", "finance"],
    "leaderboard": "bestsellers",
    "searchKeywords": ["climate"],
    "maxPublicationsPerSource": 100
}
```

Valid categories include technology, business, finance, culture, us-politics, world-politics, news, science, health, climate, crypto, sports, food, travel, music, art, fiction, history, education, parenting, philosophy and humor. A wrong name returns a free status row with the full list.

With an empty input, the actor runs a small demo and returns 20 posts from one public newsletter.

### Output

Every row has a `type`: `post`, `publication` or `status`. The dataset has three views: Posts, Newsletters and Input status.

A post row (from a real run, text shortened):

```json
{
    "type": "post",
    "newsletterUrl": "https://www.astralcodexten.com",
    "newsletterName": "Astral Codex Ten",
    "publicationId": 89120,
    "postId": 212349565,
    "title": "Your Book Review: The History And Adventures Of An Atom",
    "subtitle": "Finalist #14 in the Book Review Contest",
    "url": "https://www.astralcodexten.com/p/your-book-review-the-history-and",
    "publishedAt": "2026-10-30T16:30:33.689Z",
    "postType": "newsletter",
    "audience": "free",
    "isPaywalled": false,
    "likes": 83,
    "reactions": { "❤": 83 },
    "commentCount": 61,
    "restacks": 3,
    "wordCount": 5789,
    "authors": ["Scott Alexander"],
    "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/159d85cb-..._777x394.jpeg",
    "textMarkdown": "[*This is one of the finalists in the 2026 book review contest...",
    "textStatus": "included"
}
```

`textStatus` is `included`, `paywalled`, `not_requested`, `empty` or `unavailable`.

A newsletter profile row (from a real run):

```json
{
    "type": "publication",
    "source": "profile",
    "name": "Lenny's Newsletter",
    "url": "https://www.lennysnewsletter.com",
    "about": "Deeply researched product, growth, and career advice for product leaders, founders, and ambitious builders.",
    "authorName": "Lenny Rachitsky",
    "authorHandle": "lenny",
    "subscriberCountText": "Over 1,200,000 subscribers",
    "subscriberCountApprox": "1.2M+",
    "subscriberCountNumber": 1200000,
    "paidSubscriberRange": "Tens of thousands of paid subscribers",
    "authorFollowerCount": 1389381,
    "monthlyPrice": 20,
    "yearlyPrice": 200,
    "foundingPrice": 400,
    "currency": "USD",
    "categoryRanking": { "category": "Business", "rank": 1, "board": "bestsellers" },
    "socials": {
        "twitter": "https://x.com/lennysan",
        "youtube": "https://www.youtube.com/c/LennysPodcast",
        "website": "https://www.lennyrachitsky.com/"
    }
}
```

Leaderboard rows also carry `leaderboardCategory`, `leaderboardBoard` and `leaderboardRank`. Search rows carry `searchQuery`.

About subscriber numbers: Substack shows a rounded public figure ("Over 1,200,000 subscribers", "44K+ subscribers"). `subscriberCountNumber` is that figure as a number. It is a lower bound, not an exact count. Some writers hide their count; then the fields are empty.

A status row (free):

```json
{
    "type": "status",
    "input": "thisdoesnotexist98765xyz",
    "kind": "newsletter",
    "status": "not_found",
    "error": "No Substack newsletter at this address.",
    "charged": false
}
```

Statuses: `not_found`, `invalid_input`, `no_posts`, `no_matching_posts`, `no_new_posts`, `no_new_publications`, `no_results`, `blocked`, `skipped_cost_limit`.

The key-value store record `RUN_SUMMARY` lists every input with its status, rows by type, requests, bytes and the proxy tier used.

### Alerts: only new posts or newsletters

Turn on `onlyNew` and schedule the actor (Schedules in Apify Console, for example daily). The first run delivers everything and saves a baseline. Each later run delivers only posts published since the last run, or newsletters that newly entered a leaderboard or keyword result. You pay only for the new rows. Use a different `stateName` for each schedule.

### Pricing

You pay per result row. Platform usage is included in the price. Status rows are free. If you set a maximum cost per run, the actor stops at that limit and marks the remaining inputs `skipped_cost_limit`.

### Limits

- Full text is for free posts only.
- Keyword discovery uses Substack's writer search. It returns writers who have a newsletter, about 20 per page.
- Substack limits how fast one address can make requests. The actor paces itself to about two requests a second and backs off when Substack asks it to. A post with full text needs one extra request. Expect about 1,000 posts with full text in 10 to 25 minutes.
- Substack can change its public pages. If a field disappears, the row keeps its other fields and the status row explains any input that failed.

# Actor input Schema

## `mode` (type: `string`):

posts = posts from the newsletters you list. profiles = one profile row per newsletter (subscriber count, prices, ranking, socials). discover = find newsletters by keyword or category leaderboard.

## `newsletters` (type: `array`):

For posts and profiles. Any of: a Substack name (lenny), a substack.com address (lenny.substack.com), a custom domain (https://www.lennysnewsletter.com) or a writer's @handle (@noahpinion).

## `postUrls` (type: `array`):

For posts mode. Links to single posts, such as https://www.astralcodexten.com/p/open-thread-458. Each link returns one post row.

## `maxPostsPerNewsletter` (type: `integer`):

Newest posts to fetch per newsletter. The archive goes back to the first post, so you can set a large number for a full archive.

## `includeFullText` (type: `boolean`):

Adds the full article as Markdown (textMarkdown) for free posts. Paid posts are never unlocked: they get textStatus "paywalled" and the short public preview only. Turn off for faster metadata-only runs.

## `audience` (type: `string`):

Keep all posts, only free posts or only paid (subscriber-only) posts.

## `postedAfter` (type: `string`):

Optional. A date (2026-10-01) or a relative time (24 hours, 7 days, 2 weeks, 3 months). Older posts are skipped and paging stops once a whole page is older.

## `includeNewsletterProfile` (type: `boolean`):

In posts mode, also adds one profile row per newsletter, the same row that profiles mode returns.

## `includeAuthorSocials` (type: `boolean`):

For profile rows. Reads the writer's public Substack profile for social links (X, YouTube, LinkedIn, website), follower count and leaderboard rank. One extra request per newsletter.

## `searchKeywords` (type: `array`):

For discover mode. Topics such as climate, AI agents or personal finance. Returns writers' newsletters that match, with subscriber counts where Substack shows them.

## `categories` (type: `array`):

For discover mode. Substack categories by name or slug, such as technology, business, finance, culture, climate, crypto, health, sports. A wrong name returns a free status row that lists every valid one.

## `leaderboard` (type: `string`):

Which category leaderboard to read: Top Bestsellers (ranked by paid subscribers) or Rising.

## `maxPublicationsPerSource` (type: `integer`):

Cap per keyword and per category leaderboard.

## `onlyNew` (type: `boolean`):

Remembers what each newsletter, keyword and leaderboard returned last time and delivers only new posts or newly listed newsletters on the next run. Schedule the actor to get alerts. The first run delivers everything and saves the baseline. You pay only for new rows.

## `stateName` (type: `string`):

Keeps separate memories for different alert setups. Use a different name per schedule.

## `proxyConfiguration` (type: `object`):

Optional. Leave empty: the actor starts with no proxy and moves to Apify datacenter, then residential proxy only if Substack starts refusing requests.

## `autoProxyFallback` (type: `boolean`):

When no proxy is set, climb from no proxy to datacenter to residential if Substack rate-limits the run.

## `allowResidentialFallback` (type: `boolean`):

Residential proxy is the last step of the fallback. Turn off to stay on datacenter.

## `maxConcurrency` (type: `integer`):

How many newsletters, keywords or leaderboards run at the same time. Requests to substack.com itself are paced to avoid rate limits either way.

## `maxRetries` (type: `integer`):

Retries after a rate limit, server error or network error, with growing waits.

## Actor input object example

```json
{
  "mode": "posts",
  "newsletters": [
    "astralcodexten"
  ],
  "maxPostsPerNewsletter": 10,
  "includeFullText": true,
  "audience": "all",
  "includeNewsletterProfile": false,
  "includeAuthorSocials": true,
  "leaderboard": "bestsellers",
  "maxPublicationsPerSource": 25,
  "onlyNew": false,
  "stateName": "default",
  "autoProxyFallback": true,
  "allowResidentialFallback": true,
  "maxConcurrency": 3,
  "maxRetries": 4
}
```

# Actor output Schema

## `results` (type: `string`):

All rows. Filter by the type field.

## `runSummary` (type: `string`):

Status per input, rows by type, requests, bytes and proxy tier used.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "newsletters": [
        "astralcodexten"
    ],
    "maxPostsPerNewsletter": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("dottti/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "newsletters": ["astralcodexten"],
    "maxPostsPerNewsletter": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("dottti/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "newsletters": [
    "astralcodexten"
  ],
  "maxPostsPerNewsletter": 10
}' |
apify call dottti/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,dottti/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/EdFfqD81DHwD1qELn/builds/XJxoZB4AUvZAgBdkc/openapi.json