# Substack Scraper: Newsletter Posts, Likes & Comments (`sysco32/substack-newsletter-scraper`) Actor

Scrape posts from any Substack newsletter (subdomain or custom domain): title, subtitle, date, authors, likes, comments, wordcount, paid/free and full text of free posts. Date range, category discovery and a new-posts monitor mode.

- **URL**: https://apify.com/sysco32/substack-newsletter-scraper.md
- **Developed by:** [Gabor Sziladi](https://apify.com/sysco32) (community)
- **Categories:** Social media, News, Automation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 post results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper: Newsletter Posts, Likes, Comments & Full Text

**Substack Scraper** extracts posts from any public **Substack newsletter**, whether it lives on `name.substack.com` or on a custom domain like `www.lennysnewsletter.com`. For every post you get the **title, subtitle, URL, publish date, authors, likes, comments, restacks, word count, free/paid audience** and the **post content** as plain text and HTML. Filter by date range or audience, find newsletters by **category**, and use **monitor mode** to get only new posts on a schedule.

No login, no cookies, no proxies. The Actor only reads the same public JSON and RSS data that Substack serves to every logged-out visitor.

### What can this Substack scraper do?

- 📰 **Scrape any Substack publication**: subdomain (`noahpinion`), Substack URL, **custom domain**, author profile (`@lenny`) or a single post URL
- ❤️ **Engagement metrics**: public likes (reactions), comment count and restacks for every post
- 📝 **Full text of free posts** as clean plain text plus the original HTML
- 🔒 **Paywalled posts handled safely**: metadata plus the **public preview only** (`content_is_preview: true`); paid content is never accessed
- 📅 **Date range filter**: absolute (`2026-01-01`) or relative (`30 days`, `6 months`)
- 🆓 **Free / paid filter** to keep only free posts or only paywalled ones
- 🔎 **Newsletter discovery**: add the top publications from a Substack category (Technology, Business, Finance, Culture, Politics, Science, …)
- 🔔 **Monitor mode (only new posts)**: schedule the Actor and you only get (and only pay for) posts you haven't seen yet
- 📊 **Run summary** with per-publication counts and clear error messages; one broken URL never fails the whole run
- 🐢 **Polite by design**: per-site throttling plus automatic retries with backoff on HTTP 429/5xx

### What data does it extract?

| Field | Description |
|---|---|
| `publication_name`, `publication_url` | Newsletter name and home URL (custom domain if set) |
| `post_id`, `title`, `subtitle`, `slug`, `url` | Post identity and canonical URL |
| `published_at`, `updated_at` | ISO 8601 timestamps (UTC) |
| `authors`, `author_names` | Bylines with name, handle, ID and profile URL |
| `likes`, `comments`, `restacks` | Public engagement counts |
| `wordcount` | Length of the full post (also for paid posts) |
| `audience`, `is_paid` | `everyone`, `only_paid`, `founding`, … and a simple boolean |
| `type`, `section` | newsletter / podcast / thread, and the publication section |
| `description`, `cover_image`, `podcast_url` | SEO description, cover image, podcast audio (if public) |
| `content_text`, `content_html` | Post body: full for free posts, public preview for paid posts |
| `content_is_preview` | `true` when the content is only the public preview |

### How to scrape Substack

1. Click **Try for free**.
2. Paste newsletter URLs or subdomains into **Substack publications or post URLs** (or pick a category under *Newsletter discovery*).
3. Set **Max posts per publication** and, if needed, a **date range**.
4. Click **Start**, then download the data as JSON, CSV, Excel or HTML, or read it through the Apify API.

The default input (3 newsletters × 10 posts with full content) finishes in about a minute.

#### Input example

```json
{
  "publications": ["https://www.lennysnewsletter.com", "astralcodexten.substack.com", "noahpinion", "@lenny"],
  "maxPostsPerPublication": 50,
  "dateFrom": "90 days",
  "audience": "all",
  "includeContent": true,
  "onlyNewPosts": false
}
```

#### Output example

```json
{
  "publication_name": "Lenny's Newsletter",
  "publication_url": "https://www.lennysnewsletter.com",
  "post_id": 215694124,
  "title": "All of the Lenny & Friends Summit talks are now online!",
  "subtitle": "Plus, some reflections and takeaways from the day",
  "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "published_at": "2026-09-29T13:15:57Z",
  "type": "newsletter",
  "audience": "everyone",
  "is_paid": false,
  "authors": [{"name": "Lenny Rachitsky", "handle": "lenny", "id": 1849774, "profile_url": "https://substack.com/@lenny"}],
  "author_names": "Lenny Rachitsky",
  "likes": 291,
  "comments": 7,
  "restacks": 5,
  "wordcount": 1276,
  "content_text": "👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice…",
  "content_html": "<p><em>👋 Hey there, I’m Lenny…",
  "content_is_preview": false
}
```

A paywalled post looks the same, with `"is_paid": true`, `"audience": "only_paid"`, the full `wordcount`, and `"content_is_preview": true`. The content is only the teaser that Substack shows to logged-out readers.

A run summary is saved as `OUTPUT` in the default key-value store: posts scanned and output per publication, discovered publications, and errors (e.g. *"https://example.com does not look like a Substack publication"*).

### Monitor Substack newsletters for new posts

Turn on **Only new posts (monitor mode)** and **schedule** the Actor (e.g. every hour or every day). Each run checks the latest *Max posts per publication* posts of every newsletter and outputs only the ones that earlier runs haven't returned. Seen posts are stored in a named key-value store (`substack-scraper-seen` by default; use a different **State store name** for each independent monitor). Connect the run to **Slack, email, Google Sheets, Zapier, Make or n8n** through Apify integrations or webhooks to get alerted about new posts.

### How much does it cost to scrape Substack?

The Actor uses **pay-per-event** pricing:

| Event | Price |
|---|---|
| Post result (one post saved to the dataset) | **$0.0015** ($1.50 per 1,000 posts) |
| Actor start | $0.002 per run |

Failed publications and posts skipped by filters or by monitor mode are never charged. Example: scraping the latest 50 posts of 20 newsletters (1,000 posts) costs about **$1.50**. Set a *maximum cost per run* in the run options to cap spending. The run stops cleanly when the limit is reached.

### Use cases

- **Content and market research**: track what top newsletters in your niche publish and what gets the most likes and comments
- **Media monitoring and PR**: get alerts when newsletters publish about your company, industry or competitors
- **Lead generation and creator outreach**: build lists of Substack writers by category with their engagement
- **AI and LLM datasets / RAG**: clean plain-text bodies of free posts, with dates and authors
- **Newsletter analytics**: posting frequency, paid vs. free mix, word count and engagement trends

### FAQ

**Does it work with custom domains?** Yes. Paste the custom domain (e.g. `https://www.lennysnewsletter.com`). `name.substack.com` URLs that redirect to a custom domain are followed automatically.

**Can it get paid (paywalled) content?** No, by design. For paid posts you get the metadata (title, date, authors, likes, comments, word count) and only the public preview that anyone can see without subscribing. No login or subscription is ever used.

**How many posts can I get?** Set *Max posts per publication* to `0` to walk the whole public archive (combine it with a date range to limit it).

**How do I find newsletters to scrape?** Use *Newsletter discovery* to add the top publications of a Substack category, or paste author profiles like `@handle`.

**Is scraping Substack legal?** The Actor only collects publicly available data that Substack serves without login, and it throttles its requests. Note that Substack's Terms of Use restrict automated access, and posts can contain personal data and copyrighted text. You are responsible for using the data in line with Substack's terms, copyright and privacy laws (e.g. GDPR). If unsure, consult a lawyer.

**Something doesn't work?** Check the `OUTPUT` record for per-publication errors and open an issue on the Actor's **Issues** tab. We respond quickly.

# Actor input Schema

## `publications` (type: `array`):

One per line. Accepted forms:<br>• A subdomain: <code>noahpinion</code><br>• A Substack URL: <code>https://astralcodexten.substack.com</code><br>• A custom domain: <code>https://www.lennysnewsletter.com</code><br>• A single post URL: <code>https://www.lennysnewsletter.com/p/some-post</code> or <code>https://open.substack.com/pub/name/p/some-post</code><br>• An author profile: <code>@lenny</code> or <code>https://substack.com/@lenny</code> (uses the author's main publication)<br>Only public data is read; no login is used.

## `maxPostsPerPublication` (type: `integer`):

Newest-first. Set 0 to get the whole archive (respects the date range).

## `dateFrom` (type: `string`):

Absolute date <code>2026-01-01</code> or relative like <code>30 days</code>, <code>2 weeks</code>, <code>6 months</code>. Leave empty for no lower limit.

## `dateTo` (type: `string`):

Absolute date <code>2026-06-30</code> or relative like <code>7 days</code>. Leave empty for no upper limit.

## `audience` (type: `string`):

Filter by audience. For paid posts only the public preview text is returned.

## `includeContent` (type: `boolean`):

Fetch each post's body as plain text (and HTML). Free posts: full text. Paywalled posts: only the public preview that Substack shows to logged-out readers. Turn off for faster metadata-only runs (title, date, likes, comments, wordcount, …).

## `includeHtml` (type: `boolean`):

Also store <code>content_html</code> (only when content is included).

## `onlyNewPosts` (type: `boolean`):

Output only posts that previous runs have not output yet. Each run checks the latest posts (up to <b>Max posts per publication</b>) and remembers what it has seen in a named key-value store. Schedule the Actor to get a feed of new posts; you only pay for new ones.

## `stateStoreName` (type: `string`):

Named key-value store that remembers seen posts. Use different names for independent monitors.

## `discoverCategory` (type: `string`):

Optionally add the top publications of a Substack category (from Substack's public directory) to the list above.

## `maxDiscoveredPublications` (type: `integer`):

How many top publications to take from the chosen category.

## `maxConcurrency` (type: `integer`):

How many publications are scraped at the same time. Requests to any single site are still throttled.

## `requestIntervalSecs` (type: `number`):

Politeness delay per host. Requests are also retried with backoff on HTTP 429/5xx.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com",
    "astralcodexten.substack.com",
    "noahpinion"
  ],
  "maxPostsPerPublication": 10,
  "audience": "all",
  "includeContent": true,
  "includeHtml": true,
  "onlyNewPosts": false,
  "stateStoreName": "substack-scraper-seen",
  "discoverCategory": "none",
  "maxDiscoveredPublications": 10,
  "maxConcurrency": 4,
  "requestIntervalSecs": 0.5
}
```

# Actor output Schema

## `results` (type: `string`):

All posts found in this run (publication, title, subtitle, url, date, authors, likes, comments, wordcount, audience, content).

## `summary` (type: `string`):

JSON summary: posts scanned and output per publication, discovered publications, failed inputs with error messages.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com",
        "astralcodexten.substack.com",
        "noahpinion"
    ],
    "maxPostsPerPublication": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("sysco32/substack-newsletter-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": [
        "https://www.lennysnewsletter.com",
        "astralcodexten.substack.com",
        "noahpinion",
    ],
    "maxPostsPerPublication": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("sysco32/substack-newsletter-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com",
    "astralcodexten.substack.com",
    "noahpinion"
  ],
  "maxPostsPerPublication": 10
}' |
apify call sysco32/substack-newsletter-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sysco32/substack-newsletter-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/sBrMfFUF20sQ6Msvf/builds/OfKDxJHUCNVT79oCF/openapi.json