# Substack Scraper: Posts & Newsletters (`tidyharvest/substack-scraper`) Actor

Scrape posts from any Substack newsletter (subdomain or custom domain): titles, dates, authors, likes, comments, restacks, tags, paid/free status and full text of free posts. Newest or most popular first. No login. $0.90 per 1,000 posts with or without full text, down to $0.48 on higher Apify plans.

- **URL**: https://apify.com/tidyharvest/substack-scraper.md
- **Developed by:** [J](https://apify.com/tidyharvest) (community)
- **Categories:** News, Social media
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.48 / 1,000 post scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper: Posts & Newsletters

Get the **posts of any Substack newsletter** as clean, structured data: title, subtitle, date, author, **likes,
comment count, restacks**, word count, tags, cover image, podcast link, paid/free status and, if you want it, the
**full text of free posts**. Works with `*.substack.com` publications and custom domains. No login, no browser,
fast. **$0.90 per 1,000 posts** (less on paid plans), the same price with or without full text, and no start fee.

### What you can do with it

- **Track newsletters you follow or compete with:** what they publish, how often, and which posts get the most likes and comments.
- **Research a niche:** pull the most popular posts of a set of publications (`Most popular first`) to see what resonates.
- **Feed an LLM or a search index:** collect the full text of free posts from the newsletters you choose.
- **Monitor topics:** keep only posts that mention your keywords, or use each publication's own archive search.
- **Build a reading list or digest:** newest posts since a date across many publications in one dataset.

### Input

| Field | What it does |
|---|---|
| `publications` | Subdomains (`astralcodexten`), `name.substack.com`, custom domains (`www.lennysnewsletter.com`) or any link to a publication or post. |
| `sort` | `new` (newest first) or `top` (most popular first). |
| `includeContent` | Adds `bodyText` for free posts (default off). `includeHtml` also adds `bodyHtml`. |
| `searchWithinPublication` | Uses each publication's own archive search. |
| `keywords` | Keep only posts that mention one of these. You pay only for posts that match. |
| `dateFrom` | Only posts published on or after this date. |
| `maxPostsPerPublication` | Cap per publication (default 25). |
| `maxItems` | Hard cap for the whole run, and your cost (default 50, 0 = no limit). |

Example:

```json
{ "publications": ["astralcodexten", "www.lennysnewsletter.com"], "sort": "top", "includeContent": true, "maxPostsPerPublication": 50 }
```

### Output

One item per post:

```json
{
  "postId": 210686641,
  "title": "Your Book Review: We Present Television",
  "subtitle": "Finalist #11 in the Book Review Contest",
  "url": "https://www.astralcodexten.com/p/your-book-review-we-present-television",
  "postDate": "2026-10-10T01:05:13.124Z",
  "type": "newsletter",
  "audience": "everyone",
  "isPaywalled": false,
  "wordcount": 8651,
  "likes": 134,
  "commentCount": 91,
  "restacks": 7,
  "tags": [],
  "authors": [{ "name": "Scott Alexander", "handle": "astralcodexten" }],
  "publication": { "name": "Astral Codex Ten", "subdomain": "astralcodexten", "customDomain": "www.astralcodexten.com", "author": "Scott Alexander" },
  "bodyText": "…full text when includeContent is on and the post is free…"
}
```

| Field | Meaning |
|---|---|
| `postId`, `slug`, `url` | Substack's post id, slug and canonical link. |
| `title`, `subtitle`, `description`, `preview` | Headline, subtitle, short description and the public preview text. |
| `postDate`, `type`, `section`, `tags`, `language` | Publication date (UTC), `newsletter` / `podcast` / `thread`, section and tags. |
| `audience`, `isPaywalled` | `everyone` for free posts, `only_paid` / `founding` for paid ones. |
| `likes`, `reactions`, `commentCount`, `restacks`, `wordcount` | Engagement and length. |
| `coverImage`, `podcastUrl`, `podcastDurationSec` | Media links when present. |
| `authors` | Name and Substack handle of each byline. |
| `publication` | Name, subdomain, custom domain, main author, tagline, logo, created date, whether paid plans are on. |
| `bodyText`, `bodyHtml` | Full post body, free posts only, when requested. |
| `source`, `input` | `api` (or `rss` fallback) and the publication you entered. |

### How to use

1. Enter one or more publications (a name like `astralcodexten` or a domain).
2. Pick the order, and turn on full text if you need it.
3. Run, then download the dataset as JSON, CSV or Excel, or read it through the Apify API.

### Pricing

Pay per result: **$0.90 per 1,000 posts** on the Free plan, down to $0.48 on higher plans (`post-scraped`), with or without full text, no start fee. Keyword filters
and caps apply before you are charged. A run that finds nothing fails instead of silently returning an empty dataset.

| Apify plan tier | Free | Bronze | Silver | Gold | Platinum | Diamond |
|---|---|---|---|---|---|---|
| per 1,000 posts | $0.90 | $0.80 | $0.62 | $0.48 | $0.48 | $0.48 |

### Limits and notes

- **Paywalled posts return metadata and the public preview only.** The scraper never tries to get around a paywall.
- Very large archives take a few minutes.
- If a publication's full archive cannot be read, you get its latest ~20 posts with fewer fields.
- Comments themselves, Substack Notes, subscriber lists and Substack-wide search are not included.
- Public data only. Respect the authors' copyright: post text belongs to its authors, so check your use (quotation, analysis, internal research) is allowed before republishing it.
- This actor is not affiliated with, endorsed by or sponsored by Substack Inc.

# Actor input Schema

## `publications` (type: `array`):

One per line: a Substack subdomain (`astralcodexten`), a `*.substack.com` address, a custom domain (`www.lennysnewsletter.com`) or any link to the publication or one of its posts. Up to 200.

## `sort` (type: `string`):

Newest posts first, or the publication's most popular posts first.

## `includeContent` (type: `boolean`):

Adds `bodyText` (plain text) for posts that are free to read. One extra request per free post, so a bit slower. Paywalled posts always return metadata and the public preview only.

## `includeHtml` (type: `boolean`):

With full text on, also return the post body as HTML (`bodyHtml`).

## `searchWithinPublication` (type: `string`):

Only posts matching this search term, using the publication's own archive search. Leave empty for all posts.

## `keywords` (type: `array`):

Keep a post only if its title, subtitle, description, tags or (with full text on) body contains one of these. You pay only for posts that match. Empty = all.

## `dateFrom` (type: `string`):

Only posts published on or after this date (YYYY-MM-DD). Leave empty for no date limit.

## `maxPostsPerPublication` (type: `integer`):

Maximum posts returned for each publication.

## `maxItems` (type: `integer`):

Hard cap on posts for the whole run (also caps your cost). 0 = no limit.

## Actor input object example

```json
{
  "publications": [
    "astralcodexten",
    "lenny"
  ],
  "sort": "new",
  "includeContent": false,
  "includeHtml": false,
  "maxPostsPerPublication": 25,
  "maxItems": 50
}
```

# Actor output Schema

## `posts` (type: `string`):

One item per post.

## `summary` (type: `string`):

Posts per publication and any publications that failed.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "astralcodexten",
        "lenny"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tidyharvest/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": [
        "astralcodexten",
        "lenny",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("tidyharvest/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "astralcodexten",
    "lenny"
  ]
}' |
apify call tidyharvest/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tidyharvest/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ho3axdgj4mvb3pZ8m/builds/oKw2uYbgXRKa579Uy/openapi.json