# Substack Scraper & New Post Monitor (`onepierre/substack-newsletter-monitor`) Actor

Scrape Substack newsletters (posts, authors, reactions, comments count, full text as Markdown) and monitor them for new posts. Works with custom domains. Schedule it to get every new post from your favorite newsletters.

- **URL**: https://apify.com/onepierre/substack-newsletter-monitor.md
- **Developed by:** [Pierre Clarou](https://apify.com/onepierre) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper & New Post Monitor

Scrape any **Substack newsletter** and **get every new post automatically**: title, date, authors, likes, comments, word count, and the **full text as clean Markdown**, ready for LLMs, RAG pipelines, summaries or archives.

- ✅ Works with `name.substack.com`, **custom domains** (e.g. lennysnewsletter.com) and `substack.com/@name`
- ✅ **Monitoring mode**: returns only the posts published since the previous run, so you can schedule it daily or hourly
- ✅ **Clean Markdown**: subscribe buttons, share widgets and icons are removed
- ✅ Fast and reliable: uses Substack's public data, with no browser and no login

### How it works

**New posts only** (default, for monitoring):

1. The first run remembers the latest posts of each newsletter (baseline).
2. Every next run returns only posts published since the previous run.
3. Schedule it in Apify and send new posts to Slack, email, Notion, Google Sheets, Make, Zapier, n8n or your own LLM pipeline.

**Latest posts**: returns the most recent posts every time. Use it to export an archive or for one-off research.

### Output example

```json
{
  "publication": "lennysnewsletter.com",
  "postId": 215694124,
  "title": "All of the Lenny & Friends Summit talks are now online!",
  "subtitle": "Plus, some reflections and takeaways from the day",
  "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "postDate": "2026-09-29T13:15:57.512Z",
  "type": "newsletter",
  "audience": "everyone",
  "isPaid": false,
  "authors": ["Lenny Rachitsky"],
  "tags": [],
  "wordCount": 1276,
  "reactionCount": 283,
  "commentCount": 6,
  "restackCount": 5,
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/...",
  "excerpt": "Each week, I share deeply researched product, growth, and career advice...",
  "contentMarkdown": "I'm excited to share that every main-stage talk from the recent Lenny & Friends Summit is now online...",
  "contentHtml": "<p>...</p>",
  "isContentPreview": false
}
```

### Input

| Field | Description |
|---|---|
| `publicationUrls` | Newsletter URLs: `name.substack.com`, a custom domain or `substack.com/@name`. |
| `mode` | **New posts only** (monitoring) or **Latest posts**. |
| `maxPostsPerPublication` | Maximum posts per newsletter and run (0 = no limit). |
| `postedAfter` | Only posts published after this date. |
| `includeContent` | Add the full text as Markdown and HTML. |
| `onlyFreePosts` | Skip subscriber-only posts. |
| `stateStoreName` | Name of the monitor. Use different names for independent monitors. |

> **Tip:** with *New posts only*, the first run saves the baseline and outputs nothing. Use *Latest posts* to get posts right away.

### Paid posts

For subscriber-only posts, the Actor returns only what Substack shows publicly to non-subscribers: the metadata and the free preview chosen by the author. `isContentPreview: true` marks these posts. It never bypasses paywalls.

### Use cases

- **Newsletter digests**: collect new posts from dozens of newsletters and summarize them with an LLM.
- **RAG and knowledge bases**: feed clean Markdown into your vector database.
- **Market and competitor watch**: follow industry newsletters and get alerted on new posts.
- **Research and analytics**: export a newsletter archive with engagement metrics (likes, comments, restacks).

### FAQ

**Does it work with custom domains?** Yes. Paste the newsletter's URL, whatever its domain.

**Can I get comments or subscriber lists?** No. The Actor only collects posts and public engagement counts, never personal data about readers.

**Is it legal?** The Actor reads only content that publications make publicly available. You are responsible for using it in line with applicable laws, Substack's terms and authors' copyright.

# Actor input Schema

## `publicationUrls` (type: `array`):

Newsletter URLs: 'name.substack.com', a custom domain (e.g. https://www.lennysnewsletter.com) or 'substack.com/@name'. A post URL also works: its publication is used.

## `mode` (type: `string`):

'New posts only' returns posts published since the previous run: ideal for scheduled monitoring. The first run saves a baseline and outputs nothing. 'Latest posts' returns the most recent posts every time.

## `maxPostsPerPublication` (type: `integer`):

Maximum posts returned per publication and run. 0 = no limit (the whole archive in 'Latest posts' mode).

## `postedAfter` (type: `string`):

Only return posts published after this date (YYYY-MM-DD).

## `includeContent` (type: `boolean`):

Add the post content as Markdown (great for LLMs, RAG and summaries) and HTML. For paid posts, only the free preview that Substack shows publicly is available.

## `onlyFreePosts` (type: `boolean`):

Skip paid (subscriber-only) posts.

## `stateStoreName` (type: `string`):

Name of the key-value store remembering already-seen posts ('New posts only' mode). Use a different name for each independent monitor.

## `proxyConfiguration` (type: `object`):

Apify Proxy is recommended.

## `maxConcurrency` (type: `integer`):

How many publications are scanned in parallel.

## Actor input object example

```json
{
  "publicationUrls": [
    "https://www.lennysnewsletter.com",
    "https://astralcodexten.substack.com"
  ],
  "mode": "latest",
  "maxPostsPerPublication": 5,
  "includeContent": true,
  "onlyFreePosts": false,
  "stateStoreName": "onepierre-substack-monitor",
  "proxyConfiguration": {
    "useApifyProxy": true
  },
  "maxConcurrency": 3
}
```

# Actor output Schema

## `posts` (type: `string`):

No description

## `content` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `failedPublications` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publicationUrls": [
        "https://www.lennysnewsletter.com",
        "https://astralcodexten.substack.com"
    ],
    "mode": "latest",
    "maxPostsPerPublication": 5,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("onepierre/substack-newsletter-monitor").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publicationUrls": [
        "https://www.lennysnewsletter.com",
        "https://astralcodexten.substack.com",
    ],
    "mode": "latest",
    "maxPostsPerPublication": 5,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("onepierre/substack-newsletter-monitor").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publicationUrls": [
    "https://www.lennysnewsletter.com",
    "https://astralcodexten.substack.com"
  ],
  "mode": "latest",
  "maxPostsPerPublication": 5,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call onepierre/substack-newsletter-monitor --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,onepierre/substack-newsletter-monitor"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/jIDIE9MSq6vGctEGl/builds/e7aSBSyJEmv69ur6B/openapi.json