# Substack Scraper: Newsletter Posts, Reactions & Monitor (`webobsidians/substack-scraper`) Actor

Scrape any public Substack newsletter without a login: titles, dates, authors, tags, reactions, comments, restacks and text. Search a newsletter, filter by date or free posts, and use the built-in monitor to get only new posts.

- **URL**: https://apify.com/webobsidians/substack-scraper.md
- **Developed by:** [Web Obsidians](https://apify.com/webobsidians) (community)
- **Categories:** News, Marketing
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper

Get the posts of **public Substack newsletters** in one clean format: title, subtitle, date, authors, tags, reactions, comments, restacks, word count, free or paid, cover image and text. Paste newsletter names or addresses, optionally search inside a newsletter, set a date range or keep only the free posts, and download the result.

Turn on **Return only new posts** and schedule the Actor to monitor newsletters: each run returns only the posts that appeared since the last run.

### Why this Actor

- **No login, no API keys.** It reads the public post list that every Substack newsletter publishes, on `name.substack.com` or on its own domain. No browser, no proxies.
- **The numbers that matter.** Reactions, comments, restacks, word count, authors, tags, section and podcast link come with every post (`null` or an empty list when a post has none).
- **Free and paid posts.** Every post says whether it is free or for paid subscribers. For a paid post you get its details and the short preview that Substack shows to everyone, never the paywalled text.
- **Full text on request.** Turn on **Include the full text** to get the text of every free post.
- **Search and date range.** Use Substack's own search inside a newsletter, or stop at a date.
- **Monitoring built in.** The monitor remembers the newest post it returned and gives you only newer posts on the next run, oldest first.
- **Cheap and predictable.** $1.00 per 1,000 posts, plus $0.00005 per run start. Platform usage is included.

### Use cases

- **Newsletter research:** see which posts, topics and authors get the most reactions and comments.
- **Competitor and market monitoring:** watch newsletters and send each new post to Slack or a webhook.
- **Finding writers:** collect authors, tags and engagement across the newsletters of a niche.
- **AI agents and analysis:** give your agent the clean text of free posts with their dates and links.

### How to use

1. Add newsletters to **Newsletters**, one per line: `astralcodexten`, `astralcodexten.substack.com`, `https://www.lennysnewsletter.com` or a link to any of their posts.
2. Optionally set a search word, a date range, free posts only, full text, or how many posts you want per newsletter.
3. Click **Start**. Download the results as JSON, CSV or Excel, or send them on with Apify integrations and webhooks (for example to Slack).

#### Input example

This is the input of a real run of this Actor:

```json
{
  "publications": ["astralcodexten", "https://www.lennysnewsletter.com"],
  "maxPostsPerPublication": 3,
  "includeBody": true
}
```

#### Get only new posts every hour

Set **Return only new posts** to on and give the monitor a name (for example `ai-newsletters`). Then create a schedule in Apify Console. The first run returns the latest posts and remembers where it stopped. Each later run returns only posts that are newer than the last post it returned, oldest first. If a run could not return everything (your per-newsletter limit or your maximum cost per run was reached), the rest is returned by the next run. You pay only for the posts that are returned.

### Output

Each post is one item in the dataset. This is a real item from a run of this Actor, shortened (the image link and the text are cut):

```json
{
  "publication": "www.lennysnewsletter.com",
  "postId": 224206488,
  "title": "State of the tech job market: late 2026",
  "subtitle": "PMs are winning, engineers are plateauing, designers are hurting, and the biggest AI hiring boom isn’t in software",
  "url": "https://www.lennysnewsletter.com/p/state-of-the-tech-job-market-late",
  "date": "2026-11-10T15:04:03Z",
  "audience": "paid",
  "type": "newsletter",
  "authors": ["Lenny Rachitsky"],
  "tags": ["Career"],
  "reactions": 404,
  "comments": 42,
  "restacks": 37,
  "wordcount": 2183,
  "description": "PMs are winning, engineers are plateauing, designers are hurting, and the biggest AI hiring boom isn’t in software",
  "coverImage": "https://substackcdn.com/image/fetch/…",
  "podcastUrl": null,
  "sectionName": null,
  "text": "(the post preview text, cut here)",
  "scrapedAt": "2026-11-19T05:25:13Z"
}
```

Field notes:

- `publication` is the host name of the newsletter (`astralcodexten` becomes `astralcodexten.substack.com`; a custom domain stays as you wrote it). `date` and `scrapedAt` are UTC.
- `audience` is `free` or `paid`. Any other value that Substack uses (for example `founding`) is passed on as it is.
- `type` is the kind of post that Substack reports: `newsletter`, `podcast`, `thread` and so on. `podcastUrl` is the podcast link that Substack gives for the post, `null` for other posts.
- `authors` and `tags` are lists of names. `reactions`, `comments`, `restacks` and `wordcount` are whole numbers, or `null` when Substack does not show one.
- `text` is the short preview that Substack's post list shows. Some newsletters show no preview for their free posts; then `text` is `null`. With **Include the full text** on, `text` is the whole post as plain text for every free post: paragraphs are separated by a blank line, list items start with `- `, links keep only their words, and images are left out. A paid post always keeps its preview.
- A per-newsletter summary (posts returned, any error, and with full text on, how many posts could not be read) is saved to the key-value store as `SUMMARY`.

### Pricing

This Actor uses pay-per-event pricing:

- **$1.00 per 1,000 posts** saved to the dataset ($0.001 per post).
- **$0.00005 per run start.**

Platform usage (compute and storage) is included, and the full text costs nothing extra: it is still $0.001 per post. For example, 100 posts cost about $0.10, and an hourly monitor that returns 5 new posts a day costs about $0.19 a month ($0.15 for 150 posts plus $0.04 for 720 run starts). If you set a maximum cost per run, the Actor stops cleanly when it is reached.

### FAQ

**Which newsletters work?**
Any public Substack newsletter, on `name.substack.com` or on its own domain. A name that does not exist, an address that is not a Substack newsletter, and a link to a Substack profile or home page (`substack.com/@name`) are reported in `SUMMARY` with an error and cost nothing.

**Do I need a Substack account?**
No. The Actor reads the same post list that anyone can open without logging in.

**Does it get paid posts?**
It lists them with their title, date, authors, tags and numbers, but it does not collect the paywalled text. For a paid post `text` is only the preview that Substack shows to everyone.

**What does it not get?**
The text of comments, subscriber lists, Notes, and anything that sits behind a paywall or a login.

**How many posts can I get?**
Up to 10,000 posts per newsletter in one run, newest first (1,000 when the full text is on). Substack gives 25 posts per request and the Actor pauses between requests, so a large run takes a few minutes. The full text needs one more request for each free post.

**How do the search and the date range work?**
`dateFrom` and `dateTo` are UTC days (for example `2026-11-01`) and both days are included. Without a search, the Actor reads from the newest post backwards and stops when it reaches `dateFrom`. Substack ranks search results by relevance, not by date, so with a search the Actor reads all the results and keeps the ones inside your date range.

**How does the monitor know what is new?**
It remembers the date and id of the newest post it returned for each newsletter. A post that was written long ago but published today is new, because the publication date decides. Each monitor run looks at the newest 2,000 posts that are newer than the last returned one, so run it more often for a very busy newsletter.

**Why are some numbers different from what I see on Substack?**
Reactions, comments and restacks are the numbers that Substack reports at the moment of the run, and they keep changing.

# Actor input Schema

## `publications` (type: `array`):

One newsletter per line: its name (astralcodexten), its address (astralcodexten.substack.com, or a custom domain such as https://www.lennysnewsletter.com) or a link to any of its posts.

## `maxPostsPerPublication` (type: `integer`):

Stop after this many posts per newsletter, newest first. 0 means no limit, but at most 10,000 posts per newsletter are returned in one run (1,000 when the full text is on).

## `searchQuery` (type: `string`):

Return only posts that match this word or phrase, using Substack's own search for the newsletter. Search results come in Substack's relevance order, not by date.

## `onlyFree` (type: `boolean`):

Skip the posts that are for paid subscribers.

## `dateFrom` (type: `string`):

Skip posts older than this day. Format YYYY-MM-DD, UTC, for example 2026-11-01.

## `dateTo` (type: `string`):

Skip posts newer than this day (this day is included). Format YYYY-MM-DD, UTC.

## `includeBody` (type: `boolean`):

Read the full text of every free post. This takes one extra request per post, so the run is slower, and at most 1,000 posts per newsletter are returned. Paid posts always keep the preview that Substack shows to everyone.

## `onlyNewPosts` (type: `boolean`):

Return only posts that are newer than the ones returned by earlier runs with the same monitor name. The first run returns the latest posts and remembers where it stopped.

## `monitorName` (type: `string`):

Use different names to keep separate histories, for example one per client or per topic.

## Actor input object example

```json
{
  "publications": [
    "astralcodexten",
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 20,
  "onlyFree": false,
  "includeBody": false,
  "onlyNewPosts": false,
  "monitorName": "default"
}
```

# Actor output Schema

## `posts` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "astralcodexten",
        "https://www.lennysnewsletter.com"
    ],
    "maxPostsPerPublication": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("webobsidians/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": [
        "astralcodexten",
        "https://www.lennysnewsletter.com",
    ],
    "maxPostsPerPublication": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("webobsidians/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "astralcodexten",
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 20
}' |
apify call webobsidians/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,webobsidians/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/mslAXypDYzkGqQIaL/builds/2eIUwuDerCKLAJxpb/openapi.json
