# Substack Scraper (Posts, Publications, Comments) (`signalclear/substack-scraper`) Actor

Posts from any Substack newsletter, including custom domains: title, authors, date, paywall flag, reactions, comments, restacks and optional full text. Find newsletters by search or category. Bounded retries; failed publications are never charged.

- **URL**: https://apify.com/signalclear/substack-scraper.md
- **Developed by:** [signal clear](https://apify.com/signalclear) (community)
- **Categories:** News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper (Posts, Publications, Comments)

Get posts from **any Substack newsletter**, including ones on their own custom domain. Paste newsletter names or URLs, or find newsletters by keyword or category, and get back one clean JSON row per post: title, subtitle, authors, publish date, post type, paywall flag, word count, likes, comment count, restacks and cover image. Optionally add the full post text, the comments, and one row of publication info per newsletter.

**You only pay for publications that succeed.** Each publication gets a bounded number of attempts (3 by default, at most 5), each on a fresh proxy session that resumes where the last one stopped. A publication that still fails is listed in the `FAILED_PUBLICATIONS` record and none of its rows are saved or charged. A newsletter that does not exist fails at once, without retries. Every request has a timeout and every publication a post cap, so a run never hangs until the platform timeout.

It uses the same public JSON that Substack's own website loads for logged-out readers. No login, no cookies.

### What you can use it for

- **Newsletter research:** compare posting frequency, length and engagement across the newsletters in a niche.
- **Content monitoring:** schedule a daily run with *Newest* and *Published since* to catch new posts from a list of newsletters.
- **Lead and sponsor lists:** search a keyword or category and get each newsletter's name, author, description and Substack's subscriber tier.
- **Audience analysis:** pull the comments on a newsletter's most-discussed posts.
- **AI and BI pipelines:** a flat, predictable row per post, ready for an LLM, a spreadsheet or a dashboard.

### How to use it

1. Add **publications**, one per line. Any of these work:
   - `example` (the name before `.substack.com`)
   - `example.substack.com` or `https://example.substack.com`
   - `https://www.example-newsletter.com` (a custom domain)
   - any post URL on either, e.g. `https://example.substack.com/p/some-post`
2. Or let the Actor find newsletters: add **search queries** (Substack's publication search) and/or **category ids** (e.g. `4` Technology, `62` Business, `153` Finance, `134` Science, `96` Culture). *Publications per search or category* caps how many each one adds.
3. Choose **sort** (*Newest* or *Top*), the **maximum posts per publication**, and optionally **published since**.
4. Turn on **Fetch full post text**, **Fetch comments** or **Add a publication row** if you need them. Each adds requests, so leave them off when you only need post metadata.
5. Click **Start**.

#### Example input

```json
{
  "publications": ["example", "https://www.example-newsletter.com"],
  "searchQueries": ["climate"],
  "maxPublicationsPerQuery": 5,
  "maxPostsPerPublication": 50,
  "sort": "new",
  "publishedSince": "2026-01-01",
  "includeBody": false,
  "includeComments": false,
  "includePublicationInfo": true
}
```

### Output

Every row has a `type`: `post`, `comment` or `publication`. The dataset has a view for each.

A post (the newsletter here is fictional):

```json
{
  "type": "post",
  "publication": "The Example Gazette",
  "publicationUrl": "https://examplegazette.substack.com",
  "postId": 123456789,
  "slug": "why-small-towns-are-growing-again",
  "title": "Why small towns are growing again",
  "subtitle": "Five charts on the remote-work migration",
  "url": "https://examplegazette.substack.com/p/why-small-towns-are-growing-again",
  "canonicalUrl": "https://examplegazette.substack.com/p/why-small-towns-are-growing-again",
  "authors": ["Jane Example"],
  "publishedAt": "2026-10-20T13:00:00+00:00",
  "postType": "newsletter",
  "audience": "everyone",
  "isPaywalled": false,
  "section": null,
  "wordCount": 1840,
  "reactionCount": 412,
  "commentCount": 57,
  "restacks": 23,
  "coverImageUrl": "https://substack-post-media.s3.amazonaws.com/public/images/example_1456x816.png",
  "description": "Five charts on the remote-work migration",
  "podcastUrl": null,
  "scrapedAt": "2026-10-28T12:00:00+00:00"
}
```

- `postType` is Substack's own type: `newsletter`, `podcast`, `thread` or `video`.
- `audience` is `everyone`, `paid` (paid or founding members only) or `freeSubscribers` (free sign-up required). `isPaywalled` is `true` for anything but `everyone`.
- With **Fetch full post text**, post rows also get `bodyHtml`, `bodyText` and `isPreviewOnly`.

A comment (with **Fetch comments**):

```json
{
  "type": "comment",
  "publication": "The Example Gazette",
  "publicationUrl": "https://examplegazette.substack.com",
  "postId": 123456789,
  "postUrl": "https://examplegazette.substack.com/p/why-small-towns-are-growing-again",
  "commentId": 987654321,
  "parentCommentId": null,
  "author": "Sample Reader",
  "body": "Our town doubled its library hours last year, so this tracks.",
  "date": "2026-10-20T15:42:10+00:00",
  "editedAt": null,
  "likes": 12,
  "replyCount": 2,
  "scrapedAt": "2026-10-28T12:00:00+00:00"
}
```

Replies have `parentCommentId` set to the comment they answer.

A publication row (with **Add a publication row**) has `publication`, `publicationUrl`, `publicationId`, `subdomain`, `customDomain`, `description`, `author`, `subscribersText` (Substack's own wording, e.g. "Tens of thousands of subscribers", or `null` when the newsletter hides it), `rankingDetail`, `language`, `paidSubscriptions`, `hasPodcast`, `createdAt`, `firstPostDate` and `logoUrl`.

Publications that failed are listed in the key-value store record `FAILED_PUBLICATIONS` as `{publication, error, attempts}`, for example `not found (HTTP 404 for https://no-such-name.substack.com/)`.

### Pricing

You pay per result: one result per dataset row (post, comment or publication row). Publications that fail are never pushed to the dataset, so they cost nothing, and a run where every publication fails exits with an error and charges nothing. Use *Maximum rows in total* to cap a run's cost. See the Pricing tab for current rates.

### Limitations

- **Paid posts are previews.** For paid-only posts Substack gives logged-out readers only the free preview, so `bodyText` is the preview, `isPreviewOnly` is `true` and `wordCount` is Substack's count for the whole post.
- **Comments depend on the newsletter's settings.** Many newsletters show comments only to subscribers or only on free posts; those posts return no comment rows even when `commentCount` is above zero.
- **Subscriber counts are tiers, not numbers.** Substack publishes ranges ("Thousands of subscribers"), and some newsletters hide them.
- **Top with a date filter** scans at most 10x the post cap (at least 500 posts) per publication, so very old top posts may be missed.
- **Search and category results** come in Substack's own order and are capped by *Publications per search or category*.
- Substack can change these undocumented endpoints without notice. When that happens, the affected publications land in `FAILED_PUBLICATIONS` (not charged) and a fix follows.

### Tips

- **Proxy:** the default Apify proxy (datacenter) is usually enough. If the log shows many retries, switch to residential.
- **Large lists:** keep *Publications in parallel* at 2-4.
- **Monitoring:** run on a schedule with *Newest* and *Published since*, then deduplicate on `postId`.

### Legal and data

This Actor reads newsletter posts that publishers make public on Substack, the same data any logged-out visitor sees. It stores author and commenter names exactly as Substack displays them next to the post or comment. It does not collect subscriber lists, emails, user ids, profile links, profile pictures or any paid content beyond the free preview. You are responsible for using the data in line with Substack's terms, the authors' copyright and the privacy law that applies to you (for example GDPR).

### FAQ

**Why does a run return fewer posts than I asked for?** The newsletter has fewer posts, or *Published since* cut the list.

**Does it work on custom domains?** Yes. Paste the domain or any post URL on it.

**Something broke?** Open an issue on the Issues tab with your run ID.

# Actor input Schema

## `publications` (type: `array`):

One newsletter per line: its Substack name (the part before .substack.com, e.g. example), a name.substack.com address, a custom-domain URL, or any post URL on it.

## `searchQueries` (type: `array`):

Keywords to find newsletters with Substack's publication search. Each match is scraped like a publication you listed. The number of matches per query is capped by 'Publications per search or category'.

## `categoryIds` (type: `array`):

Substack category ids whose top publications to scrape, e.g. 4 (Technology), 62 (Business), 153 (Finance), 134 (Science), 96 (Culture). Capped by 'Publications per search or category'.

## `maxPublicationsPerQuery` (type: `integer`):

How many publications each search query or category adds to the run.

## `maxPostsPerPublication` (type: `integer`):

Stop after this many posts for each publication. Set 0 with 'Add a publication row' to collect publication info only.

## `sort` (type: `string`):

Newest first, or the publication's own 'Top' ranking (most engagement).

## `publishedSince` (type: `string`):

Only posts published on or after this date (YYYY-MM-DD, UTC). With 'Newest' the run stops paging as soon as it passes the date; with 'Top' it scans at most 10x the post cap (at least 500 posts).

## `includeBody` (type: `boolean`):

One extra request per post. Adds bodyHtml and bodyText. Paid-only posts return just the free preview Substack shows to logged-out readers; those rows have isPreviewOnly set.

## `includeComments` (type: `boolean`):

One extra request per post that has comments. Adds a row per comment (type: comment), replies included, up to 'Maximum comments per post'.

## `maxCommentsPerPost` (type: `integer`):

Cap on comment rows per post, counting replies, in Substack's 'best first' order.

## `includePublicationInfo` (type: `boolean`):

Adds one row per publication (type: publication) with its name, description, author, subscriber tier as Substack displays it, language and dates.

## `maxItems` (type: `integer`):

Hard cap on dataset rows (posts, comments and publication rows together) for the whole run. Leave empty for no cap.

## `maxConcurrency` (type: `integer`):

Publications fetched at the same time. Higher is faster but more likely to be rate-limited.

## `maxAttempts` (type: `integer`):

Each attempt uses a fresh proxy session and resumes where the last one stopped. Publications that still fail are listed in FAILED_PUBLICATIONS and are not charged. Unknown publications fail at once.

## `requestTimeoutSecs` (type: `integer`):

How long to wait for a single HTTP response before the attempt counts as failed and is retried on a fresh session.

## `proxyConfiguration` (type: `object`):

Substack's public JSON endpoints answer datacenter IPs, so the default Apify proxy is usually enough. Switch to residential if you see many retries.

## Actor input object example

```json
{
  "publications": [
    "https://www.astralcodexten.com"
  ],
  "maxPublicationsPerQuery": 10,
  "maxPostsPerPublication": 50,
  "sort": "new",
  "includeBody": false,
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "includePublicationInfo": false,
  "maxConcurrency": 3,
  "maxAttempts": 3,
  "requestTimeoutSecs": 30,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `posts` (type: `string`):

Dataset results in the "Posts" view.

## `comments` (type: `string`):

Dataset results in the "Comments" view.

## `publications` (type: `string`):

Dataset results in the "Publications" view.

## `failed` (type: `string`):

JSON list of publications or queries that failed after every retry, with the error for each. Empty when everything succeeded. Failed inputs are never charged.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.astralcodexten.com"
    ],
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("signalclear/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["https://www.astralcodexten.com"],
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("signalclear/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.astralcodexten.com"
  ],
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call signalclear/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,signalclear/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/IWhIy1sRxHsdL5XqM/builds/B3v3bORY0fp1XGawj/openapi.json