# Substack Posts Scraper (`khadinakbar/substack-posts-scraper`) Actor

Scrape Substack newsletter posts from any publication (subdomain or custom domain). Full post text, engagement, tags, and paywall status via Substack's public JSON API. HTTP-only, MCP-ready.

- **URL**: https://apify.com/khadinakbar/substack-posts-scraper.md
- **Developed by:** [Khadin Akbar](https://apify.com/khadinakbar) (community)
- **Categories:** Social media, News, Marketing
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 post scrapeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Scrape Substack newsletter posts into structured JSON

Turn any Substack publication — `*.substack.com` subdomain or a custom domain like `lennysnewsletter.com` — into clean, flat, LLM-ready post records. Newsletter researchers, content analysts, and AI teams use this Actor to export full post archives with article text, engagement metrics, tags, and paywall status. One dataset item represents one published post, with fields such as `title`, `authors`, `publishedAt`, `bodyText`, `likes`, `restacks`, `commentCount`, `tags`, and `paywallStatus`.

### Best fit for this Actor

- Start with one or more **publication URLs** (the homepage of a Substack newsletter) when you want a full, paginated post archive.
- Use a **single post URL** (`/p/some-post`) when you want to enrich or capture one specific article.
- Give the Actor both `*.substack.com` subdomains and Substack custom domains; it resolves either to the same JSON source.

This Actor works as a focused standalone workflow: it turns publication URLs into post datasets. For keyword discovery across all of Substack, or for Substack Notes and reader profiles, choose a dedicated tool for that job.

### Newsletter competitive-intelligence workflow

A content analyst tracking the AI-and-tech newsletter space starts with a list of publication homepages they already follow (`platformer.substack.com`, `www.lennysnewsletter.com`, `doomberg.substack.com`). They run the Actor with `maxPosts: 50` and `includeBodyText: true` for each. The Actor paginates each archive and returns one record per post with `title`, `publishedAt`, `wordCount`, `likes`, `restacks`, `commentCount`, `tags`, and `paywallStatus`. The analyst compares engagement-by-topic across publications in a spreadsheet, spots which posts perform, and identifies sponsorship or cross-promotion candidates using the `authors` and `commentCount` fields. A `publishedAfter` date filter keeps the dataset to the last quarter for a fresh comparison.

### Quick start input

```json
{
    "startUrls": [
        { "url": "https://platformer.substack.com" },
        { "url": "https://www.lennysnewsletter.com" }
    ],
    "maxPosts": 20,
    "includeBodyHtml": true,
    "includeBodyText": true
}
```

Each entry in `startUrls` becomes one paginated publication scrape, capped at `maxPosts` per publication. The Actor prints the maximum event cost at the start of every run.

### Input reference

| Field | Type | What it controls |
|---|---|---|
| `startUrls` | array (required) | Publication homepages or single `/p/` post URLs. Accepts objects `{ "url": "..." }`. Both `*.substack.com` and custom domains resolve to the same JSON source. |
| `maxPosts` | integer | Hard cap on posts scraped and billed per publication. Default `50`; set lower for cheap tests. |
| `includeBodyHtml` | boolean | When `true` (default), fetches each post's detail page for the complete article HTML. Public posts return full HTML. |
| `includeBodyText` | boolean | When `true` (default), includes a plain-text body derived from the HTML. Disable to reduce payload size. |
| `publishedAfter` | string | ISO date (e.g. `2026-01-01`); keep posts published on or after this date. |
| `publishedBefore` | string | ISO date; keep posts published on or before this date. |
| `audience` | enum | `all` (default), `free`, or `paid` posts only. Use `free` to receive only posts where full body text is available. |

### What data you receive

One dataset item is one published Substack post. The Actor normalizes Substack's archive and post-detail JSON into a flat record:

```json
{
    "title": "Why Platformer is leaving Substack",
    "url": "https://platformer.substack.com/p/why-platformer-is-leaving-substack",
    "publicationUrl": "https://platformer.substack.com",
    "authors": ["Casey Newton"],
    "publishedAt": "2024-01-12T01:00:49.665Z",
    "paywallStatus": "free",
    "hasBody": true,
    "likes": 727,
    "restacks": 210,
    "commentCount": 55,
    "wordCount": 4030,
    "tags": ["Substack"],
    "scrapedAt": "2026-08-05T20:38:00.000Z"
}
```

| Field | Meaning |
|---|---|
| `title`, `subtitle`, `description` | Post headline, dek, and SEO description |
| `url`, `canonicalUrl`, `slug` | Canonical post URL and Substack slug |
| `publicationUrl` | Base URL of the publication this post belongs to |
| `authors` | Byline author names |
| `publishedAt`, `updatedAt` | Publish and update timestamps (ISO 8601) |
| `paywallStatus` | `free` (full body included) or `paid` (metadata and teaser) |
| `hasBody` | Whether the record includes article body content |
| `bodyHtml`, `bodyText` | Full article HTML and plain-text body for public posts |
| `likes` | ❤ reaction count |
| `restacks` | Restack (Substack reshare) count |
| `commentCount` | Total comment count |
| `tags` | Post tags |
| `wordCount` | Reported word count |
| `coverImageUrl` | Cover/lead image URL |
| `podcastUrl`, `podcastDurationSeconds` | Audio attachment for podcast posts |
| `scrapedAt` | Run capture timestamp for freshness |

Paid posts return `paywallStatus: "paid"`, `hasBody: false`, and metadata plus the public teaser rather than the paywalled body. The `audience` field preserves Substack's original value (`everyone` / `only_paid`). Choose `audience: "free"` to receive only posts where the complete body text is available.

### Use through the API

Run the Actor synchronously and receive the dataset items in one response:

```bash
curl -X POST "https://api.apify.com/v2/acts/khadinakbar~substack-posts-scraper/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url":"https://platformer.substack.com"}],
    "maxPosts": 10
  }'
```

For large archives, start an async run and poll the dataset:

```bash
curl -X POST "https://api.apify.com/v2/acts/khadinakbar~substack-posts-scraper/runs" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://platformer.substack.com"}],"maxPosts":100}'
```

Capture the `id` and `defaultDatasetId` from the response, wait for the run to finish, then read the dataset:

```bash
curl -H "Authorization: Bearer $APIFY_TOKEN" \
  "https://api.apify.com/v2/datasets/$DATASET_ID/items?clean=true"
```

### Use with AI agents through Apify MCP

> Scrape the 20 most recent posts from platformer.substack.com. Return title,
> authors, publishedAt, bodyText, likes, restacks, commentCount, and tags.
> Stop at 20 posts and report the terminal outcome.

The Actor accepts publication or post URLs and returns one flat record per post with engagement and provenance fields. Agents should inspect the run's `outcome` (`COMPLETE`, `PARTIAL`, `VALID_EMPTY`, `UPSTREAM_FAILED`, or `CONFIG_ERROR`) in the `OUTPUT` / `RUN_SUMMARY` key-value store records, then read the dataset. Cost is predictable: one `post-scraped` event per returned post plus the platform start fee. Set up the connection through the [Apify MCP](https://mcp.apify.com) configurator for your client.

### Pricing

This Actor uses **Pay per event plus Apify platform usage**. You pay the event charges plus the underlying Apify platform compute and proxy usage for the run. When budgeting, open the live [Pricing tab](https://apify.com/khadinakbar/substack-posts-scraper/pricing) for the current event details, and use Apify's run cost controls to keep the workflow aligned with your budget.

- **Actor start** — a small per-run fee scaled by allocated RAM.
- **Post scraped** — the primary billable event, charged once per post returned.

The Actor prints the maximum event cost at the start of every run, so you can forecast spend before any charge fires.

### Best results

- Start with publication homepage URLs for full-archive exports; the Actor paginates automatically.
- Use a single `/p/` post URL for targeted one-article capture or enrichment.
- Set `audience: "free"` to receive only posts where full body text is available.
- Disable `includeBodyHtml` and `includeBodyText` when you only need metadata and engagement — this roughly halves the runtime by skipping detail-page fetches.
- Use `publishedAfter` / `publishedBefore` to bound large archives to a relevant window.

### Builder's note

While building this Actor I learned that Substack exposes two clean, unauthenticated JSON surfaces: a paginated archive endpoint that returns the post list with engagement, and a `window._preloads` JSON blob embedded in each post page that carries the full `body_html`. This means the Actor reads the same JSON the Substack frontend consumes, with no browser, login, or anti-bot workaround needed. I designed the custom-domain resolution after finding that domains like `lennysnewsletter.com` serve the identical JSON API as `*.substack.com` subdomains, so the Actor resolves both to the same source. Paid posts are handled honestly — the `paywallStatus` field signals when only the public teaser is available rather than the full body, so downstream analysis can filter confidently.

### Responsible use

This Actor reads publicly accessible data from Substack publication pages. It respects paywalls, uses no login, and collects only public content. You are responsible for complying with Substack's Terms of Service, each publication's terms, and applicable copyright laws. Use scraped content for lawful purposes such as research, analysis, and personal use, and seek permission before republishing copyrighted material.

# Actor input Schema

## `startUrls` (type: `array`):

One or more Substack publication URLs. Accepts publication roots (e.g. 'https://platformer.substack.com' or custom domain 'https://www.lennysnewsletter.com') and single post URLs ('https://.../p/some-post'). Each publication is paginated up to maxPosts. NOT a keyword search input — this actor scrapes by publication URL.

## `maxPosts` (type: `integer`):

Hard cap on the number of posts scraped and billed per publication. Each post counts as one billable result. Defaults to 50. Set lower for cheap test runs.

## `includeBodyHtml` (type: `boolean`):

When true (default), fetches each post's detail page to capture the complete article HTML. Public posts return full HTML; paid posts return metadata + paywall signal only. Adds one HTTP request per post.

## `includeBodyText` (type: `boolean`):

When true (default), includes a plain-text version of the article body (derived from the HTML). Public posts return full text; paid posts return the teaser only. Disable to reduce payload size.

## `publishedAfter` (type: `string`):

Optional ISO 8601 date or datetime. Only posts published on or after this date are returned, e.g. '2026-01-01' or '2026-06-01T00:00:00Z'. Leave empty for no lower bound.

## `publishedBefore` (type: `string`):

Optional ISO 8601 date or datetime. Only posts published on or before this date are returned. Leave empty for no upper bound.

## `audience` (type: `string`):

Filter posts by paywall audience. 'all' (default) returns every post; 'free' returns only free posts; 'paid' returns only paid/paywalled posts (metadata + teaser).

## `proxyConfiguration` (type: `object`):

Optional Apify proxy. Substack's public JSON API works without one, so leave this off unless you hit a transient network block.

## Actor input object example

```json
{
  "startUrls": [
    {
      "url": "https://platformer.substack.com"
    },
    {
      "url": "https://www.lennysnewsletter.com"
    }
  ],
  "maxPosts": 20,
  "includeBodyHtml": true,
  "includeBodyText": true,
  "publishedAfter": "2026-01-01",
  "publishedBefore": "2026-12-31",
  "audience": "all",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset items containing Substack post titles, authors, dates, full body HTML/text (public posts), engagement metrics, tags, paywall status, podcast metadata, and source URLs.

## `summary` (type: `string`):

Machine-readable run summary with publications, filters, per-publication counts, charged post count, estimated event cost, and terminal status message.

## `runSummary` (type: `string`):

Alias containing the same terminal outcome at RUN\_SUMMARY.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "startUrls": [
        {
            "url": "https://platformer.substack.com"
        },
        {
            "url": "https://www.lennysnewsletter.com"
        }
    ],
    "maxPosts": 20,
    "includeBodyHtml": true,
    "includeBodyText": true,
    "publishedAfter": "",
    "publishedBefore": "",
    "audience": "all",
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("khadinakbar/substack-posts-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "startUrls": [
        { "url": "https://platformer.substack.com" },
        { "url": "https://www.lennysnewsletter.com" },
    ],
    "maxPosts": 20,
    "includeBodyHtml": True,
    "includeBodyText": True,
    "publishedAfter": "",
    "publishedBefore": "",
    "audience": "all",
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("khadinakbar/substack-posts-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "startUrls": [
    {
      "url": "https://platformer.substack.com"
    },
    {
      "url": "https://www.lennysnewsletter.com"
    }
  ],
  "maxPosts": 20,
  "includeBodyHtml": true,
  "includeBodyText": true,
  "publishedAfter": "",
  "publishedBefore": "",
  "audience": "all",
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call khadinakbar/substack-posts-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,khadinakbar/substack-posts-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/U2vs9AzwTDlMEbbWp/builds/SX3923VdGgbbVq72S/openapi.json