# Substack Scraper — Posts, Comments, Notes & Search (`innolabs/substack-scraper`) Actor

Substack scraper and API: full posts, comments, Notes, keyword search, leaderboards and subscriber counts. Only-new-posts mode for schedules. JSON, CSV, Excel.

- **URL**: https://apify.com/innolabs/substack-scraper.md
- **Developed by:** [Tristan Leroux](https://apify.com/innolabs) (community)
- **Categories:** News, For creators
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: 5.00 out of 5 stars

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper — Posts, Comments, Notes & Search

Scrape Substack newsletters into clean data: full post text, authors, reactions and
complete comment threads, plus keyword search across all of Substack, category
leaderboards with subscriber counts, and writers' Substack Notes. No login, no API key,
no cookies. Download the results as JSON, CSV, Excel or HTML, or pull them through the API.

### What does Substack Scraper do?

Substack Scraper collects public data from [Substack](https://substack.com) in five ways,
which you can combine in one run:

- **Publications** — give it newsletters (`lenny`, `on.substack.com`,
  `www.astralcodexten.com`) and get their posts from the full archive, not just the
  latest 12: title, subtitle, full body as HTML and plain text, authors, date, word
  count, reactions, comment count, cover image and a paywall flag.
- **Comments** — every comment and nested reply on those posts, with author, date,
  reactions and thread position (`parentId`, `depth`).
- **Search all of Substack** — enter keywords and get matching **posts** (with full text)
  or matching **publications** (newsletter name, URL, author, subscriber count).
- **Leaderboards** — Substack's Bestseller and Rising rankings per category (Technology,
  Business, Culture, Finance, U.S. Politics and 30 more) with rank, author and subscriber
  counts.
- **Substack Notes** — the short-form Notes of any writer: text, date, reactions,
  restacks, replies and attachments (linked posts, links, images, quoted notes).

For scheduled runs, **Only new posts** mode remembers what earlier runs delivered and
outputs just the posts published since then.

### Use cases

- **Newsletter and media monitoring** — track competitors' or partners' newsletters
  daily and get only the new posts in Slack, Google Sheets or your CRM.
- **Market and trend research** — search all of Substack for a topic ("AI agents",
  "climate tech") and read what independent writers publish about it.
- **Sponsorship and partnership prospecting** — pull the top 100 publications of a
  category with subscriber counts and author handles to build an outreach list.
- **Audience and engagement analysis** — compare reactions, comment counts and comment
  threads across posts to see which topics resonate.
- **AI and RAG pipelines** — feed full post text and Notes into embeddings, summaries or
  an LLM knowledge base.
- **Creator research** — follow what influential writers post on Substack Notes, with
  reactions and restacks as engagement signals.

### How much does it cost?

You pay only for the items you receive (pay-per-event). Duplicates and items over your
`maxItems` limit are never charged.

| Event | What you pay for | Free plan | Paid plans (from) |
| --- | --- | --- | --- |
| `post` | One post with full text and metadata | $1.00 per 1,000 | $0.50 per 1,000 |
| `comment` | One comment or reply | $0.30 per 1,000 | $0.15 per 1,000 |
| `note` | One Substack Note | $0.50 per 1,000 | $0.25 per 1,000 |
| `publication` | One publication from search or a leaderboard | $1.00 per 1,000 | $0.50 per 1,000 |

"Paid plans (from)" is the Gold-and-above price; Bronze and Silver plans sit in between.
Apify also charges a tiny actor-start fee ($0.00005 per run).

Worked examples (Free plan prices):

1. **Archive of 10 newsletters, 50 posts each** — 500 posts × $1.00 / 1,000 = **$0.50**.
   With comments on (say 3,000 comments) add 3,000 × $0.30 / 1,000 = $0.90.
2. **Top 100 Technology bestsellers + 200 Notes from 4 writers** — 100 publications
   ($0.10) + 200 notes ($0.10) = **$0.20**.

The Apify Free plan includes $5 monthly credit, enough for about 5,000 posts or 10,000
notes every month.

### How to use Substack Scraper

In Apify Console:

1. Open Substack Scraper and click **Try for free**.
2. Fill in at least one target: **Publications**, **Search keywords**, **Leaderboard
   categories** or **Notes authors**.
3. Optional: turn on **Include comments**, set the limits, or enable **Only new posts**
   for scheduled runs.
4. Click **Start** and wait for the run to finish.
5. Open the **Output** tab and export the dataset as JSON, CSV, Excel or HTML.

Through the API:

1. Get your API token in Apify Console → Settings → Integrations.
2. Call the actor with your input JSON (see [Integrations and API](#integrations-and-api)).
3. Read the dataset items from the run.

### Input

Fill in at least one of `publications`, `searchQueries`, `leaderboardCategories` or
`noteProfiles`.

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `publications` | array | – | Newsletters: subdomains (`on`, `lenny`), hosts (`on.substack.com`, `www.lennysnewsletter.com`) or URLs. Aliases of the same publication are scraped once. |
| `includePosts` | boolean | `true` | Output one item per post with full text. |
| `includeComments` | boolean | `false` | Output every comment and reply of the collected posts (publication and search posts). |
| `maxPostsPerPublication` | integer | `50` | Posts per publication (1–1000). |
| `query` | string | `""` | Keep only posts of the listed publications whose title or text contains this word (scans up to 500 posts per publication). |
| `sort` | string | `new` | Archive order: `new` (newest first) or `top`. |
| `onlyNewPosts` | boolean | `false` | Output only posts not delivered by an earlier run. |
| `stateName` | string | `default` | Separate "already delivered" memory per schedule or task. |
| `searchQueries` | array | – | Keywords searched across all of Substack, each separately. |
| `searchType` | string | `posts` | `posts` (full posts) or `publications` (newsletters). |
| `maxSearchResults` | integer | `50` | Results per keyword (1–1000). |
| `leaderboardCategories` | array | – | Category slugs or names (`technology`, `business`, `culture`, `finance`, `us-politics`, `podcast`, …) or URLs like `https://substack.com/leaderboard/technology/paid`. |
| `leaderboardType` | string | `paid` | `paid` (Bestsellers) or `trending` (Rising). A `/rising` or `/paid` URL overrides it. |
| `maxPublicationsPerLeaderboard` | integer | `100` | Ranked publications per category. |
| `noteProfiles` | array | – | Writers: `@handle`, `handle`, `https://substack.com/@handle` or `https://substack.com/profile/<id>-<name>`. |
| `maxNotesPerProfile` | integer | `50` | Most recent Notes per writer (1–5000). |
| `maxItems` | integer | `1000` | Hard cap on all output items in the run. |
| `proxyConfiguration` | object | Apify Proxy | Proxy settings. |
| `debug` | boolean | `false` | More detailed run log. |

Example — a daily monitoring run that also pulls a leaderboard and a writer's Notes:

```json
{
    "publications": ["on.substack.com", "www.astralcodexten.com"],
    "maxPostsPerPublication": 10,
    "includeComments": true,
    "onlyNewPosts": true,
    "searchQueries": ["climate tech"],
    "searchType": "posts",
    "maxSearchResults": 20,
    "leaderboardCategories": ["technology"],
    "leaderboardType": "paid",
    "noteProfiles": ["@hamish"],
    "maxNotesPerProfile": 30,
    "maxItems": 2000
}
```

### Output

All items go into one dataset; the `type` field tells them apart: `post`, `comment`,
`note` or `publication`. Examples below come from real runs (long text shortened).

#### Post

```json
{
    "type": "post",
    "publication": "on",
    "publicationName": "On Substack",
    "postId": 222802023,
    "title": "New on Substack: chat channels, lists, freewalls, discounts, podcast updates, and more",
    "subtitle": "More ways to connect with your audience, get paid for your work, and be in control of your feed",
    "slug": "new-on-substack-chat-channels-lists",
    "url": "https://on.substack.com/p/new-on-substack-chat-channels-lists",
    "publishedAt": "2026-10-30T17:03:17.633Z",
    "authors": [],
    "audience": "everyone",
    "bodyHtml": "<div class=\"captioned-image-container\"><figure>...",
    "bodyText": "Illustrations by Jac Saik, Designer at Substack\nOur latest product updates give everyone more ownership over what they read...",
    "wordCount": 1115,
    "reactions": 736,
    "commentCount": 0,
    "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/4be31bf5-e657-46bf-b998-79e7cb8f0306_2398x1342.png",
    "bodyTruncated": false,
    "fetchedAt": "2026-11-02T13:55:48Z"
}
```

- `bodyTruncated` is `true` when a paid post is only available as its free preview; the
  scraper never bypasses paywalls.
- `authors` is empty when the publication hides bylines.
- Posts found by keyword search also carry `searchQuery`.

#### Comment

```json
{
    "type": "comment",
    "publication": "astralcodexten",
    "publicationName": "Astral Codex Ten",
    "postId": 223080029,
    "commentId": 376501150,
    "parentId": null,
    "depth": 0,
    "author": "Whenyou",
    "authorHandle": "personfromdenmark123",
    "isPostAuthor": false,
    "body": "An older case of an intellectually disabled man ...",
    "createdAt": "2026-11-02T09:53:02.947Z",
    "reactions": 1,
    "fetchedAt": "2026-11-02T12:16:44Z"
}
```

Top-level comments have `parentId: null` and `depth: 0`; replies point to their parent
comment. Deleted comments are skipped, their visible replies are kept.

#### Note

```json
{
    "type": "note",
    "noteId": 374632951,
    "url": "https://substack.com/@hamish/note/c-374632951",
    "author": "Hamish McKenzie",
    "authorHandle": "hamish",
    "authorId": 3567,
    "body": "Bundle of goodies ",
    "publishedAt": "2026-10-30T22:43:42.413Z",
    "editedAt": null,
    "reactions": 44,
    "restacks": 4,
    "replies": 4,
    "attachments": [
        {
            "type": "post",
            "url": "https://on.substack.com/p/new-on-substack-chat-channels-lists",
            "title": "New on Substack: chat channels, lists, freewalls, discounts, podcast updates, and more",
            "imageUrl": "https://substack-post-media.s3.amazonaws.com/public/images/4be31bf5-e657-46bf-b998-79e7cb8f0306_2398x1342.png"
        }
    ],
    "profile": "@hamish",
    "fetchedAt": "2026-11-02T13:56:28Z"
}
```

`attachments[].type` is `post`, `link`, `image`, `note` (a quoted note) or
`publication_recommendation`. `body` can be empty when a note only shares a post.

#### Publication (leaderboard or search)

```json
{
    "type": "publication",
    "publicationId": 6349492,
    "name": "SemiAnalysis",
    "subdomain": "semianalysis",
    "url": "https://newsletter.semianalysis.com",
    "customDomain": "newsletter.semianalysis.com",
    "description": "Bridging the gap between the world's most important industry, semiconductors, and business.",
    "authorName": "Dylan Patel",
    "authorHandle": "semianalysis",
    "authorId": 21783302,
    "logoUrl": "https://substackcdn.com/image/fetch/$s_!y_cf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1af867ac-65cc-46ad-aa7b-13323ae19142_444x415.png",
    "language": "en",
    "subscriberCount": 326000,
    "subscriberCountText": "Over 326,000 subscribers",
    "paidSubscribersText": "Tens of thousands of paid subscribers",
    "source": "leaderboard",
    "searchQuery": null,
    "category": "Technology",
    "categorySlug": "technology",
    "leaderboardType": "paid",
    "rank": 1,
    "fetchedAt": "2026-11-02T13:56:26Z"
}
```

- `subscriberCount` is the number Substack displays publicly ("Over 326,000", "8.6K+"),
  so it is rounded; `subscriberCountText` keeps the original wording. It is `null` when
  the writer hides the count.
- `paidSubscribersText` is Substack's paid-subscriber range (leaderboards only).
- `rank`, `category` and `leaderboardType` are filled for leaderboard rows; `searchQuery`
  for search rows. Fields Substack does not show for a row are `null`.

### Integrations and API

Run the actor and get the items back in one call:

```bash
curl -X POST "https://api.apify.com/v2/acts/innolabs~substack-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"publications": ["on.substack.com"], "maxPostsPerPublication": 10}'
```

Python with [apify-client](https://docs.apify.com/api/client/python):

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("innolabs/substack-scraper").call(
    run_input={"searchQueries": ["AI agents"], "searchType": "posts", "maxSearchResults": 20}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["type"], item.get("title") or item.get("name"), item.get("url"))
```

- **Make, Zapier, n8n** — use the Apify app/node to start the actor and send new posts to
  Google Sheets, Airtable, Notion, Slack or email.
- **Scheduling** — create a schedule in Apify Console (e.g. every morning) and turn on
  **Only new posts**, so each run delivers just what was published since the last one.
- **Webhooks** — trigger your pipeline when a run finishes.
- **AI agents (MCP)** — connect Claude, Cursor or other MCP clients through the Apify MCP
  server at https://mcp.apify.com and let your agent search Substack or read newsletters on
  demand.

### FAQ

#### Is it legal to scrape Substack?

The scraper only collects publicly visible data: free posts, public previews of paid
posts, public comments, Notes, search results and leaderboards that anyone can see
without logging in. It does not log in or bypass paywalls. Personal data in comments and
Notes may be protected by laws such as GDPR, so make sure you have a legitimate reason to
store it and check that your use complies with Substack's terms.

#### Do I need proxies?

Usually not: Substack's public pages are light on blocking and the scraper retries
throttled requests automatically. Apify Proxy is on by default; keep it on for large or
frequent runs.

#### How many items can I get?

Up to 1,000 posts per publication, 1,000 results per search keyword, the full leaderboard
of each category (Substack lists up to 100 publications) and up to 5,000 Notes per writer.
`maxItems` caps the whole run so you never spend more than you planned.

#### Can I get only new posts?

Yes. Turn on **Only new posts** and run the actor on a schedule. The first run delivers
posts normally and remembers them; later runs output only posts published after that
(for search keywords: results not delivered before). The memory is kept in your Apify
account under the name in **State name**, so use different state names for different
schedules, or a new name to start over.

#### What if a publication, writer or category fails?

Each target is processed on its own. If one newsletter, post, writer or category cannot
be loaded (wrong name, deleted, temporarily unavailable), the run continues with the rest
and the problem is listed in the run log and in the run's `SUMMARY` record. Failed items
are not charged.

#### Can it scrape paid posts?

It returns what Substack shows to non-subscribers. For paid posts that is the free preview,
and `bodyTruncated: true` tells you the text is incomplete.

### Limits

- Paid posts: only the free preview is available (`bodyTruncated: true`).
- Subscriber counts are Substack's rounded public figures; exact numbers are not public.
  Paid-subscriber figures are ranges ("Tens of thousands") and appear only on
  leaderboards.
- Leaderboards exist for Substack's top-level categories; subcategories have no public
  ranking. Each leaderboard lists up to 100 publications.
- Publication search returns the newsletters Substack itself suggests for the keyword
  (typically up to ~100 per keyword).
- Notes: the writer's own Notes are collected (newest first); restacks of other people's
  Notes are skipped. Replies to a Note are counted (`replies`) but not downloaded.
- Only new posts applies to posts (from publications and keyword search); Notes,
  comments of older posts, leaderboards and publication search are always delivered in
  full.
- Comments are collected for the posts scraped in the same run.

# Changelog

This Actor's version history is a separate document: https://apify.com/innolabs/substack-scraper/changelog.md

# Actor input Schema

## `publications` (type: `array`):

Substack newsletters to scrape: subdomains (on, lenny), hosts (on.substack.com, www.lennysnewsletter.com) or full URLs. Duplicates and aliases of the same publication are scraped once.

## `includePosts` (type: `boolean`):

Emit one dataset item per post with full bodyHtml, bodyText, authors, reactions and comment count.

## `includeComments` (type: `boolean`):

Emit one dataset item per comment, including nested replies (parentId and depth preserved). Applies to publication posts and to posts found by search.

## `maxPostsPerPublication` (type: `integer`):

How many posts to collect from each publication. Comments are fetched for these posts only. With a query, matching posts are collected up to this number (the actor scans at most 500 posts looking for matches).

## `query` (type: `string`):

Optional case-insensitive filter on post title or body text inside the publications above. To search all of Substack, use Search keywords instead.

## `sort` (type: `string`):

Order used for the archive: newest first, or the publication's top posts.

## `onlyNewPosts` (type: `boolean`):

Push only posts that previous runs have not delivered. The actor remembers the newest post per publication (and the results seen per search keyword) in a named key-value store, so a daily schedule outputs just the new posts. The first run delivers posts normally and saves the state.

## `stateName` (type: `string`):

Keeps separate 'already seen' memory for different schedules or tasks. Letters, digits, '-' and '\_' only. Change it to start fresh.

## `searchQueries` (type: `array`):

Keywords or phrases searched across all of Substack, e.g. 'climate tech' or 'AI agents'. Each keyword is searched separately.

## `searchType` (type: `string`):

Return matching posts (full post bodies, billed per post) or matching publications (newsletter profiles with subscriber counts, billed per publication).

## `maxSearchResults` (type: `integer`):

How many posts or publications to return for each keyword.

## `leaderboardCategories` (type: `array`):

Category slugs or names (technology, business, culture, finance, us-politics, ...) or leaderboard URLs such as https://substack.com/leaderboard/technology/paid.

## `leaderboardType` (type: `string`):

Bestsellers (ranked by paid subscribers) or Rising (fastest-growing). A leaderboard URL in the list above overrides this.

## `maxPublicationsPerLeaderboard` (type: `integer`):

How many ranked publications to return per category. Substack leaderboards list up to 100.

## `noteProfiles` (type: `array`):

Writers whose Notes to collect: @handle, handle, https://substack.com/@handle or https://substack.com/profile/<id>-<name>.

## `maxNotesPerProfile` (type: `integer`):

How many of each writer's most recent Notes to collect.

## `maxItems` (type: `integer`):

Hard cap on dataset items across all item types. The run stops cleanly when reached.

## `proxyConfiguration` (type: `object`):

Apify Proxy settings. Use a proxy if Substack rate-limits or blocks the actor's IP.

## `debug` (type: `boolean`):

Log per-publication progress and diagnostics.

## Actor input object example

```json
{
  "publications": [
    "on.substack.com"
  ],
  "includePosts": true,
  "includeComments": false,
  "maxPostsPerPublication": 10,
  "query": "",
  "sort": "new",
  "onlyNewPosts": false,
  "stateName": "default",
  "searchType": "posts",
  "maxSearchResults": 50,
  "leaderboardType": "paid",
  "maxPublicationsPerLeaderboard": 100,
  "maxNotesPerProfile": 50,
  "maxItems": 1000,
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "debug": false
}
```

# Actor output Schema

## `dataset` (type: `string`):

All items produced by the run, one JSON object per record.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "on.substack.com"
    ],
    "maxPostsPerPublication": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("innolabs/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["on.substack.com"],
    "maxPostsPerPublication": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("innolabs/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "on.substack.com"
  ],
  "maxPostsPerPublication": 10
}' |
apify call innolabs/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,innolabs/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/jy16248J2wmbCn7FH/builds/TyfE4KhYpcsxJTf9S/openapi.json