# Substack Scraper - Posts, Full Text, Comments & Rankings (`pipiagent/substack-scraper`) Actor

Scrape Substack newsletters: every post of a publication with full text, comments and replies, keyword search across all of Substack, and category leaderboards with subscriber counts. No login, no API key.

- **URL**: https://apify.com/pipiagent/substack-scraper.md
- **Developed by:** [Frank0306](https://apify.com/pipiagent) (community)
- **Categories:** News, Social media, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper

Get data from [Substack](https://substack.com) newsletters without an API key and without a login. Five modes in one Actor:

- **Posts of publications.** Every post of the newsletters you choose, newest or most popular first, with likes, comments, restacks, word count, authors and tags. Add the full text with one switch.
- **Post details.** Full text and HTML of single posts.
- **Comments.** All comments of a post, with replies, likes and authors.
- **Search.** Posts that match your keywords across all of Substack, like the search box on substack.com.
- **Publications and rankings.** Substack's category leaderboards (top paid newsletters, or all newsletters) with subscriber counts, followers and paid subscriber ranges, or the same profile data for the publications you list.

Use it for media monitoring, competitor and creator research, finding newsletters to sponsor or pitch, tracking what is said about a topic, building reading lists and RSS-like feeds, and feeding articles to AI tools.

- **Type a URL, a subdomain or a handle.** https://www.lennysnewsletter.com, noahpinion and @lenny all work, custom domains included.
- **Clean, flat records.** ISO dates, numbers as numbers, plain text next to the HTML, ready for spreadsheets.
- **Subscriber counts.** The number Substack shows on author profiles ("462K+ subscribers"), plus followers and the paid subscriber range ("Tens of thousands").
- **No login, no API key, no proxy.**
- **Low price.** $1 per 1,000 results.

### What data can you get?

| Mode | One result is | Main fields |
|---|---|---|
| Posts of publications | One post | Title, subtitle, link, publish date, type (newsletter, podcast, thread), free or paid, word count, likes, comments, restacks, authors, tags, preview text, cover image, podcast link and length. Optional: full text and HTML |
| Post details | One post | The same fields, always with full text and HTML, last update time and whether the text is cut by a paywall |
| Comments | One comment or reply | Text, date, likes, number of replies, author name and handle, parent comment, depth, pinned flag, link |
| Search | One post | Post fields plus the publication's name and link and the keyword that found it |
| Publications and rankings | One publication | Rank, category, name, link, custom domain, description, author, author bio, X/Twitter handle, creation date, paid plan benefits, subscriber count, followers, paid subscriber range, leaderboard rank |

### How to use it

1. Click **Try for free**.
2. Choose a mode in **What to scrape**.
3. Add publications, post links, keywords or categories.
4. Click **Start**, then download the results as JSON, CSV or Excel.

### Input examples

Posts of two newsletters published since September, with full text:

```json
{
    "mode": "posts",
    "publications": ["https://www.lennysnewsletter.com", "noahpinion"],
    "dateFrom": "2026-09-01",
    "includeContent": true
}
```

The 20 most popular posts of a newsletter:

```json
{
    "mode": "posts",
    "publications": ["@lenny"],
    "sortBy": "top",
    "maxItems": 20
}
```

Comments and replies of a post:

```json
{
    "mode": "comments",
    "postUrls": ["https://charlesduhigg.substack.com/p/the-robots-are-coming-and-i-feel"]
}
```

Posts about AI agents from this year, across all of Substack:

```json
{
    "mode": "search",
    "searchQueries": ["ai agents"],
    "dateFrom": "2026-01-01",
    "maxItems": 200
}
```

The top 100 paid newsletters in Technology and Finance, with subscriber counts:

```json
{
    "mode": "publications",
    "categories": ["4", "153"],
    "ranking": "paid",
    "maxItemsPerQuery": 100
}
```

### Output examples

A post:

```json
{
    "type": "post",
    "postId": 174637920,
    "title": "Artificial intelligence is taking over the world!",
    "subtitle": "What's the best way to learn about artificial intelligence? Which apps should I be using?",
    "url": "https://charlesduhigg.substack.com/p/the-robots-are-coming-and-i-feel",
    "publishedAt": "2025-09-27T10:57:16Z",
    "postType": "newsletter",
    "audience": "everyone",
    "isPaid": false,
    "wordCount": 782,
    "reactionCount": 91,
    "commentCount": 14,
    "restackCount": 4,
    "authors": ["Charles Duhigg"],
    "authorHandles": ["charlesduhigg"],
    "tags": [],
    "previewText": "I live near Silicon Valley, where conventional wisdom holds there are three ways to become a billionaire:",
    "publicationId": 1810583,
    "publicationName": "The Science of Better",
    "publicationUrl": "https://charlesduhigg.substack.com",
    "isPaywalled": false,
    "text": "I live near Silicon Valley, where conventional wisdom holds there are three ways to become a billionaire:\n- Be born to a billionaire and then wait.\n...",
    "html": "<p>I live near Silicon Valley, ...</p>"
}
```

A publication from a leaderboard:

```json
{
    "type": "publication",
    "rank": 1,
    "category": "Technology",
    "ranking": "paid",
    "publicationId": 6349492,
    "name": "SemiAnalysis",
    "url": "https://newsletter.semianalysis.com",
    "subdomain": "semianalysis",
    "customDomain": "newsletter.semianalysis.com",
    "description": "Bridging the gap between the world's most important industry, semiconductors, and business.",
    "authorName": "Dylan Patel",
    "authorHandle": "semianalysis",
    "twitter": "dylan522p",
    "hasPaidPlan": true,
    "subscriberCount": 325000,
    "subscriberCountText": "325K+ subscribers",
    "paidSubscribersRange": "Tens of thousands",
    "followerCount": 378966,
    "leaderboardRank": 1
}
```

A comment:

```json
{
    "type": "comment",
    "commentId": 160837451,
    "commentUrl": "https://charlesduhigg.substack.com/p/the-robots-are-coming-and-i-feel/comment/160837451",
    "postTitle": "Artificial intelligence is taking over the world!",
    "isReply": true,
    "parentCommentId": 160496861,
    "depth": 1,
    "text": "nuts! it used to be free!",
    "createdAt": "2025-09-28T17:33:23Z",
    "reactionCount": 0,
    "authorName": "Charles Duhigg",
    "authorHandle": "charlesduhigg"
}
```

### How much does it cost?

You pay per saved result: **$1.00 per 1,000 results**. A post, a comment, a reply and a publication each count as one result. There is no start fee and no monthly rental.

| Run | Results | Cost |
|---|---|---|
| The last 100 posts of a newsletter | 100 | $0.10 |
| 1,000 search results with full text | 1,000 | $1.00 |
| Top 500 paid newsletters of a category | 500 | $0.50 |

Set a maximum charge per run in the run options and the Actor stops when it is reached.

### Limits you should know

- **Paid posts give only the free part.** Without a subscription, Substack shows the beginning of a paid post. The Actor saves that part and sets `isPaywalled` to true. It does not log in and cannot read paid content.
- **The Actor is polite, so big runs take time.** It waits a little between requests to stay within Substack's limits: about 30 posts per second from archives, and about 1 to 2 per second with full text, comments or subscriber counts turned on.
- **Search returns what Substack's own search returns**, ordered by relevance, usually a few hundred posts per keyword. Notes and profiles in the search results are skipped.
- **Leaderboards stop at about 500 publications** per category, the same as on substack.com.
- **Subscriber counts are rounded by Substack** ("325K+"), and some authors hide them. Paid subscriber numbers are only given as ranges.
- **"Only posts about" uses the publication's archive search** and always sorts newest first.
- **This is unofficial.** The Actor reads the same public data that Substack's website loads. It is not made by, endorsed by or connected to Substack.

### Related Actors

- [Telegram Channel Scraper](https://apify.com/pipiagent/telegram-channel-scraper): posts, views and subscriber counts of public Telegram channels.
- [YouTube Scraper](https://apify.com/pipiagent/youtube-scraper): videos, Shorts, comments and channel profiles.
- [Google News Scraper](https://apify.com/pipiagent/google-news-scraper): news by keyword, topic or city, with date ranges.
- [Website Contact Scraper](https://apify.com/pipiagent/website-contact-scraper): emails, phone numbers and social links from any website.

### FAQ

**Is it legal?** The Actor collects only data that Substack shows publicly to every visitor, and it does not read paid content. Results can contain personal data such as names and comments. Make sure you have a legitimate reason to process it, as required by the GDPR and similar laws, and respect the authors' copyright if you republish their text.

**Something is broken or missing?** Open a ticket in the Issues tab. Issues are usually answered within a day.

# Actor input Schema

## `mode` (type: `string`):

Posts: every post of the publications you list. Post details: full text of single posts. Comments: comments and replies of posts. Search: posts that match keywords, across all of Substack. Publications: category leaderboards, or profiles of the publications you list, with subscriber counts.

## `publications` (type: `array`):

For Posts and Publications modes. Publication URLs (https://www.lennysnewsletter.com, https://noahpinion.substack.com), subdomains (noahpinion) or author handles (@lenny).

## `postUrls` (type: `array`):

For Post details and Comments modes. Links to posts, for example https://www.lennysnewsletter.com/p/how-to-tell-if-a-startup-has-a-moat .

## `searchQueries` (type: `array`):

For Search mode. Each keyword is searched across all of Substack, like the search box on substack.com.

## `categories` (type: `array`):

For Publications mode. Substack leaderboard categories. Each gives up to about 500 ranked publications.

## `ranking` (type: `string`):

For categories: Substack's bestseller list of paid newsletters, or the list of all newsletters (free and paid).

## `includeSubscriberCounts` (type: `boolean`):

For Publications mode. Adds the subscriber count, follower count and paid subscriber range from the author's profile. One extra request per publication.

## `maxItems` (type: `integer`):

Maximum number of results to save in this run. You are only charged for saved results.

## `maxItemsPerQuery` (type: `integer`):

Optional. Limits the results of each publication, post, keyword or category, so one big input does not use up the whole run.

## `sortBy` (type: `string`):

Posts mode: newest posts first, or the most popular. Comments mode: newest comments first, or the best ones.

## `postKeyword` (type: `string`):

Posts mode, optional. Keeps only posts of the publication that match this word, using the publication's own archive search.

## `dateFrom` (type: `string`):

Optional. Keeps only posts published on or after this date (YYYY-MM-DD). Works in Posts and Search modes.

## `dateTo` (type: `string`):

Optional. Keeps only posts published on or before this date (YYYY-MM-DD).

## `includeContent` (type: `boolean`):

Posts and Search modes: also loads the full text and HTML of every post (one extra request per post). Paid posts give only the free preview part. Post details mode always includes it.

## `includeReplies` (type: `boolean`):

Comments mode: also saves the replies under each comment. Every reply counts as one result.

## `proxyConfiguration` (type: `object`):

Optional. Substack works without a proxy. Set one only if runs keep failing.

## Actor input object example

```json
{
  "mode": "posts",
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "ranking": "paid",
  "includeSubscriberCounts": true,
  "maxItems": 100,
  "sortBy": "new",
  "includeContent": false,
  "includeReplies": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Posts, comments or publications saved by the run, one item each.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("pipiagent/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": ["https://www.lennysnewsletter.com"] }

# Run the Actor and wait for it to finish
run = client.actor("pipiagent/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com"
  ]
}' |
apify call pipiagent/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pipiagent/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/xKoZoH2fwQYpq7uqa/builds/865L4c9HAicdbDdeo/openapi.json