# Substack Scraper - Posts, Full Text, Search & Newsletter Leads (`pantomath/substack-scraper`) Actor

Scrape any Substack newsletter: posts with full text & comments, keyword search across Substack, and top newsletters per category with subscriber counts, author socials, paid prices & sponsors. New-post monitoring.

- **URL**: https://apify.com/pantomath/substack-scraper.md
- **Developed by:** [abdallah alramahi](https://apify.com/pantomath) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 post scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper - Posts, Full Text, Search & Newsletter Leads

**All of Substack in one Actor: scrape any newsletter's posts with full text and comments, search all of Substack by keyword, and export the top newsletters of any category with subscriber counts, author social links, paid plan prices and current sponsors.**

No login, no browser, fast JSON output. From $1 per 1,000 posts.

### What you can scrape

| Input | You get |
|---|---|
| **Publications** (`lennysnewsletter.com`, `name.substack.com`, `substack.com/@handle`) | Every post: title, subtitle, date, URL, likes, restacks, comment count, word count, paywall status, tags, authors, cover image, podcast audio, **full text** |
| **Comments** (optional) | Every comment and reply: author, handle, date, text, likes, thread depth & parent |
| **Search queries** (`ai agents`, `real estate investing`…) | Matching posts from all of Substack **plus matching authors with subscriber & follower counts** |
| **Categories** (`technology`, `business`, `finance`, `health`, `culture`, `crypto`…) | Top newsletters from Substack leaderboards (Top paid / Trending): rank, author, bio, **subscriber count**, followers, **X / LinkedIn / YouTube / website links**, **monthly & annual price**, bestseller tier, **active sponsors**, podcast feed |

### Use cases

- **Newsletter sponsorships & influencer outreach** — build a list of the top newsletters in your niche with subscriber counts, prices and social links. See who already sponsors them.
- **Content & trend research** — what topics get the most likes, restacks and comments.
- **Competitor tracking** — follow competitor newsletters; get only new posts each day (monitoring mode).
- **AI / RAG datasets** — clean full-text articles with metadata.
- **Audience research** — read what readers actually say in comments.

### Monitoring new posts

Enable **"Monitoring: only new posts since last run"** and create a daily **Schedule**. Each run returns only posts published since the previous run — connect Slack, email, Google Sheets or a webhook to get alerts.

### Pricing (pay per event)

| Event | Price |
|---|---|
| Post (incl. full text) | $0.001 |
| Comment | $0.0003 |
| Newsletter / author record (leaderboards & search) | $0.003 |

Examples: 1,000 posts with full text = **$1**. The top 100 newsletters in a category with subscriber counts and socials = **$0.30**.

### Example: newsletter record

```json
{
  "type": "publication",
  "rank": 1,
  "category": "Technology",
  "name": "SemiAnalysis",
  "url": "https://newsletter.semianalysis.com",
  "authorName": "Dylan Patel",
  "subscriberCount": 319000,
  "subscriberCountText": "319K+ subscribers",
  "authorFollowers": 371194,
  "authorSocialLinks": { "twitter": "https://x.com/dylan522p", "website": "https://www.semianalysis.com" },
  "isPaid": true,
  "monthlyPrice": 50.0,
  "annualPrice": 500.0,
  "currency": "USD",
  "bestsellerTier": 1000,
  "activeSponsors": [],
  "firstPostDate": "2020-05-22T21:26:00.000Z"
}
```

### Example: post record

```json
{
  "type": "post",
  "publication": "Lenny's Newsletter",
  "title": "All of the Lenny & Friends Summit talks are now online!",
  "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "postDate": "2026-09-29T13:15:57.512Z",
  "audience": "everyone",
  "isPaywalled": false,
  "wordCount": 1276,
  "likes": 277,
  "restacks": 4,
  "comments": 6,
  "authors": [{ "name": "Lenny Rachitsky", "handle": "lenny" }],
  "content": "Full article text…"
}
```

`enrichAuthors` adds subscriber counts and social links to leaderboard records (one extra request per newsletter).

### FAQ

**Can it scrape paywalled posts?** It returns all metadata and the public preview of paid posts (`contentIsPreview: true`). It does not bypass paywalls.

**Does it work with custom domains?** Yes — any Substack publication, including custom domains like `lennysnewsletter.com`.

**Are subscriber counts always available?** Substack shows them for most authors; a few authors hide them, in which case the field is empty.

**Need another field?** Open an issue — I usually reply within 24 hours.

# Actor input Schema

## `publications` (type: `array`):

Newsletters to scrape posts from. Any format: lennysnewsletter.com, https://example.substack.com, https://substack.com/@handle or just a handle.

## `searchQueries` (type: `array`):

Search all of Substack: returns matching posts and authors (with subscriber & follower counts).

## `categories` (type: `array`):

Top newsletters in a category, e.g. technology, business, finance, health, culture, science, crypto, sports, food. Returns author, social links, paid plan prices, bestseller tier and current sponsors.

## `leaderboard` (type: `string`):

Which Substack leaderboard to read for categories.

## `maxPostsPerPublication` (type: `integer`):

Newest posts first.

## `maxResultsPerQuery` (type: `integer`):

Maximum number of posts returned for each search query.

## `maxNewslettersPerCategory` (type: `integer`):

Leaderboards are paginated by 25.

## `includeContent` (type: `boolean`):

Adds the full article text (free posts) or the public preview (paywalled posts).

## `includeComments` (type: `boolean`):

Adds every comment and reply (with author, date, likes, thread depth) for scraped publication posts.

## `maxCommentsPerPost` (type: `integer`):

Maximum number of comments saved for each post.

## `postedAfter` (type: `string`):

Skip older posts from publications.

## `onlyNewPosts` (type: `boolean`):

Remembers the newest post of every publication and returns only posts published after it. Schedule the Actor to get alerts for new posts.

## `includeAuthorsInSearch` (type: `boolean`):

Search also returns matching author profiles with subscriber and follower counts.

## `enrichAuthors` (type: `boolean`):

Adds follower count, subscriber count (when public) and all social links (X, YouTube, LinkedIn, website...) of each leaderboard author. One extra request per newsletter.

## `proxyConfiguration` (type: `object`):

Usually not needed.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "leaderboard": "paid",
  "maxPostsPerPublication": 50,
  "maxResultsPerQuery": 50,
  "maxNewslettersPerCategory": 25,
  "includeContent": true,
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "onlyNewPosts": false,
  "includeAuthorsInSearch": true,
  "enrichAuthors": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("pantomath/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": ["https://www.lennysnewsletter.com"] }

# Run the Actor and wait for it to finish
run = client.actor("pantomath/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com"
  ]
}' |
apify call pantomath/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pantomath/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/aL92OLKMTNjB2qtKM/builds/X6iBYhKMbow7dteIr/openapi.json