# Substack Scraper: Newsletter Posts, Search & Leaderboards (`datadesk-io/substack-scraper`) Actor

Scrape Substack newsletters: posts from any publication (archive, top posts, full text), Substack-wide post search and category leaderboards (bestseller and rising newsletters with paid tier, subscribers, prices and author). No login, no API key.

- **URL**: https://apify.com/datadesk-io/substack-scraper.md
- **Developed by:** [DataDesk](https://apify.com/datadesk-io) (community)
- **Categories:** Social media, Lead generation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper: Newsletter Posts, Search & Leaderboards

Get **posts from any Substack newsletter** as clean JSON/CSV/Excel: title, subtitle, date, **likes, comments, restacks**, word count, free/paid flag, authors, tags, cover image, podcast audio and (optionally) the **full post text**. Search **all of Substack** by keyword, and list the newsletters on **Substack's category leaderboards** (bestsellers and rising) with paid tier, subscriber count, prices and author. **No login, no API key, no paid proxy.**

🇪🇸 **Español:** extrae publicaciones de cualquier newsletter de Substack: título, fecha, me gusta, comentarios, restacks, palabras, si es de pago, autores, etiquetas y el texto completo (opcional). Busca en todo Substack por palabra clave y lista los newsletters de los rankings por categoría (bestsellers y en ascenso) con nivel de suscriptores de pago, suscriptores, precios y autor. Sin cuenta ni API key.
🇧🇷 **Português:** extrai posts de qualquer newsletter do Substack: título, data, curtidas, comentários, restacks, palavras, se é pago, autores, tags e o texto completo (opcional). Busca em todo o Substack por palavra-chave e lista as newsletters dos rankings por categoria (bestsellers e em alta) com faixa de assinantes pagos, assinantes, preços e autor. Sem login nem API key.

### What you get

- 📰 **Newsletter archives**: any publication by subdomain (`lenny`), custom domain (`www.lennysnewsletter.com`), URL or author `@handle`. Newest first or Substack's **top posts** order, back to the first post.
- 🔎 **Substack-wide search** (`searchQueries`): posts matching a keyword across all newsletters, with their newsletter and subscriber count.
- 🏆 **Leaderboards** (`leaderboardCategories`): Substack's bestseller and rising lists per category (technology, business, finance, culture, politics, ...): rank, newsletter, URL, **paid tier** ("Thousands of paid subscribers"), subscribers when shown, **monthly/yearly price**, author, bio, first post date. Built for **sponsorship prospecting, influencer and competitor research**.
- 📄 **Full text** (`includeContent`): plain text + HTML of every post (free preview only for paid posts). Ideal for **AI/RAG datasets, summaries and content analysis**.
- 📈 **Newsletter info on every post**: name, URL, subscriber count as shown by Substack ("Over 1,200,000 subscribers") and paid tier.
- 📅 **Date cut-off** (`postedAfter`) for daily monitoring; duplicates removed across sources.
- 💸 **$1 per 1,000 posts**, $2 per 1,000 leaderboard newsletters, plus $0.005 per run. Runs with no results cost nothing.

### Pricing

Pay-per-event: **$0.001 per post** and **$0.002 per leaderboard newsletter** (Store discounts for Silver/Gold/Platinum/Diamond plans) + $0.005 once per run, charged only if at least one result is found. Full text costs the same. Unknown newsletters, missing posts and duplicates are free. A spending limit is respected: the run stops cleanly at it.

### Input (all optional)

```json
{
  "publications": ["lenny", "https://www.astralcodexten.com", "@noahpinion"],
  "postUrls": ["https://www.lennysnewsletter.com/p/how-to-tell-if-a-startup-has-a-moat"],
  "searchQueries": ["ai agents"],
  "leaderboardCategories": ["technology", "finance"],
  "leaderboardType": "paid",
  "maxPostsPerSource": 100,
  "sort": "new",
  "includeContent": false,
  "postedAfter": "2026-10-01",
  "maxResults": 1000
}
```

Empty input returns the 25 newest posts of Lenny's Newsletter and The Pragmatic Engineer in a few seconds.

- `maxPostsPerSource` applies to each newsletter, each search query and each leaderboard category.
- `postedAfter` is a UTC date (`YYYY-MM-DD`). With `sort: new` the archive stops at the date, so older posts are never fetched or charged.

### Output example (real run, 2026-10-28, shortened)

```json
{
  "recordType": "post",
  "id": 219482066,
  "title": "How to tell if a startup has a moat",
  "subtitle": "A framework for assessing whether the company you're considering joining is built to last",
  "url": "https://www.lennysnewsletter.com/p/how-to-tell-if-a-startup-has-a-moat",
  "date": "2026-10-13T12:45:41.160Z",
  "type": "newsletter",
  "audience": "only_paid",
  "isPaid": true,
  "wordcount": 4153,
  "reactionCount": 372,
  "commentCount": 2,
  "restacks": 21,
  "authors": [{"name": "Dan Hockenmaier", "handle": "danhock", "profileUrl": "https://substack.com/@danhock"}],
  "publicationName": "Lenny's Newsletter",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "publicationSubscribers": "Over 1,200,000 subscribers",
  "publicationPaidSubscribers": "Tens of thousands of paid subscribers"
}
```

Leaderboard row:

```json
{
  "recordType": "publication",
  "rank": 1,
  "leaderboard": "Bestsellers in Technology (paid)",
  "publicationName": "SemiAnalysis",
  "publicationUrl": "https://newsletter.semianalysis.com",
  "paidSubscribers": "Tens of thousands of paid subscribers",
  "monthlyPrice": 50,
  "currency": "USD",
  "authorName": "Dylan Patel",
  "authorProfileUrl": "https://substack.com/@semianalysis"
}
```

### Honest limits (read this)

- **Public data only.** It reads the same public JSON Substack's website uses. **Paid posts return only the free preview** (`contentIsPreviewOnly: true`); there is no login and no paywall bypass.
- **Subscriber numbers are Substack's rounded labels** ("Over 92,000 subscribers", "Thousands of paid subscribers"), and only when the newsletter shows them. Exact counts and subscriber lists/emails are not public, so they are not returned.
- **Search** is Substack's own search (relevance order, posts only; notes and comments are skipped).
- **No paid proxy.** In our tests (Oct 28, 2026) on Apify, every request worked without a proxy. If Substack answers 429/5xx, the Actor retries with backoff and then uses the Apify datacenter proxy included in your plan (capped per run); results already collected are kept.
- Posts are public but copyrighted and may contain personal data. Check Substack's terms and the laws that apply to your use (copyright, GDPR) before you republish or store them.

### Use cases

Newsletter sponsorship and partnership prospecting · competitor and creator research · content and topic trend analysis · AI/RAG datasets from newsletters you follow · daily monitoring of newsletters with `postedAfter` · lead lists of top creators per category.

### Scheduling and integrations

Run on a schedule with Apify Schedules and send new data through webhooks, Make, Zapier, n8n or the API. Plain REST: `POST https://api.apify.com/v2/acts/datadesk-io~substack-scraper/run-sync-get-dataset-items?token=...`.

### Use with AI agents (MCP)

This Actor works as a tool for AI agents through the [Apify MCP server](https://mcp.apify.com). Add it to Claude, ChatGPT, Cursor, VS Code or any MCP client:

`https://mcp.apify.com/?tools=datadesk-io/substack-scraper`

Sign in with OAuth or `Authorization: Bearer <APIFY_TOKEN>`. Local stdio config:

```json
{"mcpServers": {"substack": {"command": "npx", "args": ["-y", "@apify/actors-mcp-server", "--tools", "datadesk-io/substack-scraper"], "env": {"APIFY_TOKEN": "<token>"}}}}
```

**What goes in → what comes out:** Input: newsletters (subdomain, domain, URL or @handle), post URLs, search keywords and/or leaderboard categories, with optional caps, order, full text and date cut-off. Output: one JSON row per post (title, date, engagement, authors, newsletter, optional full text) or per leaderboard newsletter (rank, paid tier, prices, author). Every input is optional and every output field is documented.

**Example prompt:** "Get the last 30 days of posts from Lenny's Newsletter and The Pragmatic Engineer with full text and summarize the main topics." Use `maxPostsPerSource`, `postedAfter` or `maxResults` to cap the cost of one call.

### FAQ

**Do I need a Substack account or API key?** No.
**Can it read paid posts in full?** No. Only the free preview that Substack shows to everyone.
**Can it export subscriber emails?** No. Substack does not publish them.
**Issues?** Report them on the Issues tab. We usually reply within 48 h.

# Actor input Schema

## `publications` (type: `array`):

Substack newsletters to scrape: subdomain (lenny), substack.com address (lenny.substack.com), custom domain or URL (https://www.lennysnewsletter.com) or an author @handle (@lenny). Leave everything empty to get the latest posts of 2 sample newsletters.

## `postUrls` (type: `array`):

Single posts to scrape with full text, e.g. https://www.lennysnewsletter.com/p/how-to-tell-if-a-startup-has-a-moat. Paid posts return only the free preview.

## `searchQueries` (type: `array`):

Keywords to search across all of Substack (the same search as substack.com). Returns matching posts with their newsletter, author and engagement.

## `leaderboardCategories` (type: `array`):

Substack leaderboard categories to list newsletters from: technology, business, finance, culture, us-politics, world-politics, news, science, health, food, sports, music, art, crypto, education, history, fiction, literature, design, travel, parenting, philosophy, faith, climate, comics, humor, games, film-and-tv, fashionandbeauty, home-garden, international. One row per newsletter (rank, paid tier, subscribers, prices, author).

## `leaderboardType` (type: `string`):

Which Substack leaderboard to read for the categories above.

## `maxPostsPerSource` (type: `integer`):

Cap per newsletter, per search query and per leaderboard category. Newsletter archives are read 50 posts per page until this cap, the date limit or the first post.

## `sort` (type: `string`):

Order of newsletter archive posts: newest first or Substack's 'top' (most popular) order.

## `includeContent` (type: `boolean`):

Add the full post body (plain text and HTML) to every post. Costs one extra request per post (slower), same price. Paid posts only include the free preview part.

## `postedAfter` (type: `string`):

Skip posts older than this date (YYYY-MM-DD, UTC). With 'Newest first' the archive stops at the date, so older posts are never fetched or charged. Great for daily monitoring.

## `maxResults` (type: `integer`):

Hard cap on rows (posts + publications) for the whole run. Never exceeded.

## Actor input object example

```json
{
  "publications": [
    "lenny",
    "https://newsletter.pragmaticengineer.com"
  ],
  "leaderboardType": "paid",
  "maxPostsPerSource": 25,
  "sort": "new",
  "includeContent": false
}
```

# Actor output Schema

## `overview` (type: `string`):

One row per post (see dataset schema for field descriptions).

## `publications` (type: `string`):

Leaderboard rows (recordType = publication).

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "lenny",
        "https://newsletter.pragmaticengineer.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("datadesk-io/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": [
        "lenny",
        "https://newsletter.pragmaticengineer.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("datadesk-io/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "lenny",
    "https://newsletter.pragmaticengineer.com"
  ]
}' |
apify call datadesk-io/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,datadesk-io/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/3O7fRGvfe52Yi5zX0/builds/GgjuHydPGdYdYqHDN/openapi.json