# Substack Posts Scraper (`literate_universe/substack-posts-scraper`) Actor

Scrape any Substack publication: post titles, subtitles, authors, dates, likes, comments, paywall status, cover images and optionally full HTML body. Give it publication URLs and it walks the archive. No login.

- **URL**: https://apify.com/literate\_universe/substack-posts-scraper.md
- **Developed by:** [John Rutherford](https://apify.com/literate_universe) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 66.7% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 post rows

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Substack Posts Scraper

Scrape **any Substack publication's archive**: titles, subtitles, authors, dates, likes, comments, restacks, paywall status, cover images and, optionally, the full post body. Give it publication URLs and it walks the archive newest-first. No login.

For newsletter research, competitor tracking, content aggregation, trend analysis and building reading lists or datasets.

### What you get

One row per post:

| Field | Meaning |
|---|---|
| `publication`, `post_id`, `slug`, `url` | Identity and link |
| `title`, `subtitle`, `description`, `cover_image` | Headline material |
| `type` | newsletter, podcast, thread |
| `post_date`, `author` | Published time (ISO) and bylines |
| `likes`, `comments`, `restacks` | Engagement |
| `paywalled`, `audience` | `only_paid` means subscribers only |
| `word_count` | Length |
| `body_html`, `body_text` | Full body when **Include full post body** is on (free preview only for paywalled posts) |

Sample row:

```json
{
  "publication": "www.lennysnewsletter.com",
  "title": "How to hire your first PM",
  "post_date": "2026-09-02T12:01:00.000Z",
  "author": "Lenny Rachitsky",
  "likes": 412,
  "comments": 37,
  "paywalled": false,
  "url": "https://www.lennysnewsletter.com/p/how-to-hire-your-first-pm"
}
```

### Input

| Option | Default | What it does |
|---|---|---|
| **Publication URLs** | | One or more Substack sites, custom domains included |
| **Max posts per publication** | 100 | Newest first |
| **Only posts since** | | ISO date; stops at older posts |
| **Post type** | all | newsletter, podcast, thread |
| **Include full post body** | off | Adds `body_html` and `body_text` |
| **Max posts total** | 5000 | Cap across all publications |

### Pricing

Pay per event: one small charge per post row. The body option adds one request per post but no extra charge.

### Notes and limits

- Works with custom domains as long as the site is a Substack (the `/api/v1/archive` endpoint answers).
- Paywalled bodies return only the public preview; this Actor does not bypass paywalls.
- Be polite: a 300 ms pause between archive pages is built in.

# Actor input Schema

## `publications` (type: `array`):

One or more Substack publication URLs, e.g. https://www.lennysnewsletter.com or https://stratechery.substack.com.

## `maxPostsPerPublication` (type: `integer`):

Newest first.

## `sinceDate` (type: `string`):

Optional, e.g. 2026-01-01. Stops walking the archive once older posts are reached.

## `postType` (type: `string`):

Newsletter posts, podcasts, threads, or everything.

## `includeBody` (type: `boolean`):

Fetch each post and add body\_html and body\_text. Paywalled posts return the free preview only. One extra request per post.

## `maxItems` (type: `integer`):

Stop after this many posts across all publications.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 100,
  "postType": "all",
  "includeBody": false,
  "maxItems": 5000
}
```

# Actor output Schema

## `posts` (type: `string`):

One row per post. Append ?format=csv for CSV.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("literate_universe/substack-posts-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": ["https://www.lennysnewsletter.com"] }

# Run the Actor and wait for it to finish
run = client.actor("literate_universe/substack-posts-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com"
  ]
}' |
apify call literate_universe/substack-posts-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,literate_universe/substack-posts-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/U4Q9CpGhEt7EUKlIW/builds/s5GyR4bCjvcmPb0zC/openapi.json