# Substack Posts Scraper (`davidts/substack-posts-scraper`) Actor

Extract posts from any Substack publication: title, subtitle, authors, date, type, paywall status, reactions, comments, restacks, podcast audio and optionally the full body (HTML and text). Works with custom domains, no login, uses Substack's public JSON endpoints. Export to JSON, CSV or Excel.

- **URL**: https://apify.com/davidts/substack-posts-scraper.md
- **Developed by:** [David Tsitsakishvili](https://apify.com/davidts) (community)
- **Categories:** News, Social media
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Posts Scraper

**Substack Posts Scraper** is a Substack scraper that exports every post of any **Substack
newsletter** to JSON, CSV or Excel: title, subtitle, authors, publish date, post type, **paywall
status**, likes (reactions), comments, restacks, podcast audio and, optionally, the **full post
body** (HTML and plain text). Give it a publication URL, a subdomain such as `lenny` or a custom
domain such as `newsletter.pragmaticengineer.com`, and get one row per post - a whole newsletter
archive in a spreadsheet, with no code and no login.

The Actor uses Substack's public JSON endpoints (the same ones the Substack website uses) instead
of parsing HTML pages, so it is fast, light on Substack's servers and does not break when the
page layout changes. No login is required and no private data is accessed.

### What can you do with it

- **Export a Substack archive to CSV or Excel**: every post of a newsletter (`maxPostsPerPublication: 0`)
  in one file, for backups, migrations or a reading list.
- **Monitor competitor newsletters**: track what a set of Substacks publish, how often, and which
  posts get the most likes, comments or restacks.
- **Find what resonates in a niche**: sort a publication's posts by reactions to see which topics
  and formats perform best before you write your own.
- **Feed AI and search tools**: pull the full text of free posts for summaries, topic modelling,
  RAG pipelines or an internal search index.
- **Study paywall strategy**: see which posts a publication paywalls, how long they are and who
  writes them.
- **Collect podcast episodes**: audio URLs and durations from Substack podcasts.

### How to use

1. Open [Substack Posts Scraper](https://apify.com/davidts/substack-posts-scraper) and click
   **Try for free** (or **Start**).
2. Paste one or more publications (`lenny`, `https://www.lennysnewsletter.com`, a custom domain),
   set how many posts you want (`0` = whole archive) and tick **Include post body** if you need
   the text.
3. Click **Start**, then download the results as CSV, Excel or JSON from the **Output** tab, or
   fetch them through the API.

#### Sample output (selected columns)

| title | publication.name | publishedAt | paywalled | reactionCount | commentCount | wordCount |
| --- | --- | --- | --- | --- | --- | --- |
| How to tell if a startup has a moat | Lenny's Newsletter | 2026-10-13 | true | 375 | 2 | 4153 |

The full record (25+ fields) is shown under *Output* below.

### Input

| Field | Type | Default | Description |
| --- | --- | --- | --- |
| `publications` | array of strings | required | Substack publications to scrape: full URLs (`https://www.lennysnewsletter.com`), subdomains (`lenny`, `lenny.substack.com`) or custom domains (`newsletter.pragmaticengineer.com`). Publications that moved to a custom domain are followed automatically. |
| `maxPostsPerPublication` | integer | `100` | Posts to save per publication. `0` = the whole archive. |
| `sort` | `new` or `top` | `new` | `new` = newest first (chronological). `top` = Substack's own popularity ranking (not chronological, so it cannot be combined with `publishedAfter`). |
| `includePostBody` | boolean | `false` | Fetch each post individually and add `bodyHtml` and `bodyText`. Free posts get the full body; paywalled posts get the public preview only. Roughly doubles the number of requests. |
| `publishedAfter` | string | - | Only keep posts published on or after this date and stop paging there. Absolute (`2026-01-31`, `2026-01-31T10:00:00Z`, UTC) or relative (`7 days`, `2 weeks`, `3 months`). Only with `sort: new`. |
| `postTypes` | array | all | Keep only these types: `newsletter`, `podcast`, `video`, `thread`. Empty = all. |
| `proxyConfiguration` | object | Apify datacenter proxies | Proxy settings. Datacenter proxies are enough for normal volumes; switch to residential if you see repeated 403/429 errors in the log. |

Example input:

```json
{
    "publications": ["lenny", "newsletter.pragmaticengineer.com"],
    "maxPostsPerPublication": 50,
    "sort": "new",
    "includePostBody": true,
    "publishedAfter": "30 days",
    "postTypes": ["newsletter", "podcast"]
}
```

### Output

One dataset item per post. The fields are stable and documented in the dataset schema; `bodyHtml`
and `bodyText` appear only when `includePostBody` is enabled.

```json
{
    "publication": {
        "subdomain": "lenny",
        "name": "Lenny's Newsletter",
        "url": "https://www.lennysnewsletter.com",
        "customDomain": "www.lennysnewsletter.com"
    },
    "postId": 219482066,
    "slug": "how-to-tell-if-a-startup-has-a-moat",
    "url": "https://www.lennysnewsletter.com/p/how-to-tell-if-a-startup-has-a-moat",
    "canonicalUrl": "https://www.lennysnewsletter.com/p/how-to-tell-if-a-startup-has-a-moat",
    "title": "How to tell if a startup has a moat",
    "subtitle": "A framework for assessing whether the company you're considering joining is built to last",
    "description": "A framework for assessing whether the company you're considering joining is built to last",
    "type": "newsletter",
    "audience": "only_paid",
    "paywalled": true,
    "publishedAt": "2026-10-13T12:45:41.160Z",
    "authors": [{ "name": "Dan Hockenmaier", "handle": "danhock" }],
    "coverImageUrl": "https://substackcdn.com/image/fetch/...",
    "wordCount": 4153,
    "reactionCount": 375,
    "commentCount": 2,
    "restackCount": 22,
    "podcastUrl": null,
    "podcastDurationSec": null,
    "bodyHtml": "<p><em>Hey there, I'm Lenny. ...</em></p>...",
    "bodyText": "Hey there, I'm Lenny. ...",
    "scrapedAt": "2026-10-29T12:00:00.000Z"
}
```

| Field | Meaning |
| --- | --- |
| `publication.subdomain` | Substack subdomain (`lenny`). `null` when the publication uses a custom domain and Substack did not expose the subdomain. |
| `publication.name`, `publication.url`, `publication.customDomain` | Name and home page (current domain) of the publication. |
| `postId`, `slug`, `url`, `canonicalUrl` | Substack's numeric ID, the URL slug, the post URL on the current domain and the canonical URL reported by Substack (differs for cross-posts). |
| `title`, `subtitle`, `description` | Post texts; `subtitle`/`description` are `null` when empty. |
| `type` | `newsletter`, `podcast`, `video` or `thread`. |
| `audience` | Substack's audience value: `everyone`, `only_paid`, `only_free` (free subscribers) or `founding`. |
| `paywalled` | `true` when the full post needs a paid subscription (`only_paid` or `founding`). |
| `publishedAt` | ISO 8601 (UTC). |
| `authors` | Public bylines, `name` and `handle` only. |
| `coverImageUrl` | Cover image, or `null`. |
| `wordCount` | Length of the full post as reported by Substack (also for paywalled posts). |
| `reactionCount`, `commentCount`, `restackCount` | Engagement counters at scrape time. |
| `podcastUrl`, `podcastDurationSec` | Audio file URL and duration in seconds for podcast posts, otherwise `null`. |
| `bodyHtml`, `bodyText` | Post body (only with `includePostBody`). For paywalled posts this is the public preview. `null` if the post could not be fetched. |
| `scrapedAt` | When the item was collected. |

The **Overview** tab in the run shows title, publication, date, paywall flag, reactions and URL;
the full items are available as JSON, CSV, Excel or via the API like any Apify dataset.

### Pricing

This Actor uses Apify's **pay-per-event** model, so you only pay for what you get:

- **$0.0015 per saved post** (one dataset item), and
- **$0.005 per run start**.

1,000 posts therefore cost about **$1.50 + $0.005**, whichever options you choose; enabling
`includePostBody` does not change the price per post. The exact prices are shown on the Pricing
tab of this page. Platform usage (compute, proxies) is covered by these prices and is **not**
billed separately. Set a maximum cost per run in the run options if you want a hard cap: the
Actor stops requesting more posts as soon as the cap is reached and everything saved until then
stays in the dataset.

### How it works

For every publication the Actor:

1. reads the publication's RSS feed once (`/feed`) to learn its name and current domain
   (publications that moved from `name.substack.com` to a custom domain are followed);
2. pages through the public archive endpoint (`/api/v1/archive`) 50 posts at a time, newest first,
   until the limit, the `publishedAfter` cut-off or the end of the archive;
3. with `includePostBody`, fetches each selected post (`/api/v1/posts/<slug>`) for its body.

Politeness: at most 3 requests in flight, 300-800 ms between requests to the same publication,
exponential backoff on 429 / 5xx responses (up to 5 retries), no login and no cookies. A
publication that fails (typo, removed, blocked) is reported in the log and the run continues with
the next one; the run only fails when every publication failed. If two entries turn out to be the
same publication (for example `lenny` and `https://www.lennysnewsletter.com`), it is scraped once
and the second entry is reported as skipped.

The log ends with a summary: publications processed, posts saved, paywalled count and the error
of every publication that could not be scraped.

### Limitations

- **Paywalled posts** only contain the public preview in `bodyHtml`/`bodyText` (plus all
  metadata, including the full `wordCount`). The Actor never logs in and cannot bypass paywalls.
- Posts restricted to free subscribers (`audience: only_free`) are not marked `paywalled`, but
  their body may also be a preview.
- `sort: top` is Substack's own ranking, which is not strictly by reaction count and cannot be
  combined with `publishedAfter`.
- Comments, subscriber counts, Notes and recommendations are not collected (they are not posts).
- Very large archives (thousands of posts) with `includePostBody` take longer because every post
  is one request; expect roughly 2-4 posts per second.
- Substack can rate-limit an IP that sends too many requests. The Actor backs off automatically;
  if a run still ends with 429/403 errors, run fewer publications at once or choose residential
  proxies in `proxyConfiguration`.

### Legal, privacy and robots.txt

The Actor collects only **publicly available** data from publication pages that Substack's
`robots.txt` allows for all crawlers (`/api/`, `/feed` and `/p/` are not disallowed; the Actor
does not touch `/feed/private`, comment pages or any sign-in flows). It does not use accounts,
cookies or paid content.

Personal data is limited to what Substack publishes on every post: the author's **display name
and handle**. Emails, photos, bios and subscriber status that Substack includes in its responses
are discarded. If you process this data in the EU you remain responsible for having a lawful
basis (for example legitimate interest for research) and for honouring data-subject requests;
see Apify's notes on [GDPR and web scraping](https://blog.apify.com/is-web-scraping-legal/).
Respect the publication's terms when republishing content.

### Integrations

- **Make, Zapier, n8n and other automation tools**: add the Apify integration (the official
  "Apify" app/node), choose *Run an Actor* with this Actor and your input, and use *Get dataset
  items* (or the "Actor run finished" trigger) to pass the records on - for example to a Google Sheet of new posts or a Slack message for every new issue of a newsletter.
  Each tool also accepts Apify webhooks: Console -> Actor -> Integrations -> Webhooks, event
  `Run succeeded`.
- **AI agents (MCP)**: Apify's MCP server (`https://mcp.apify.com`, or the `@apify/actors-mcp-server`
  package) exposes this Actor as a tool to Claude, ChatGPT, Cursor and other MCP clients: the
  agent calls `substack-posts-scraper` with an input such as `{"publications": ["lenny"], "maxPostsPerPublication": 20}` and reads the dataset
  items back.
- **API and SDKs**: `POST https://api.apify.com/v2/acts/davidts~substack-posts-scraper/run-sync-get-dataset-items`
  with the input as JSON body returns the records directly; the Apify client libraries for
  JavaScript and Python wrap the same call.
- **Schedules**: Console -> Schedules runs the Actor daily or weekly with a saved input and, combined
  with a webhook or integration, delivers only what is new.

### Troubleshooting

| Log message | What it means |
| --- | --- |
| `Publication not found (HTTP 404 ...)` | The subdomain/URL does not exist. Check the spelling; use the URL shown in the browser. |
| `did not return an RSS feed ... Either this is not a Substack publication or it is private.` | The domain is not a Substack site, the publication is private, or Substack served a block page. |
| `Substack rate-limited the requests (HTTP 429)` / `blocked the requests (HTTP 403)` | Too many requests from one IP. Try again later, scrape fewer publications per run, or switch `proxyConfiguration` to residential proxies. |
| `publishedAfter can only be used with sort "new"` | Remove `publishedAfter` or switch `sort` to `new`. |
| `The archive is empty` | The publication has no public posts yet. |

### FAQ

**How much does it cost to scrape 1,000 Substack posts?**
About **$1.50** ($0.0015 per saved post) plus $0.005 per run, with or without the full post body.
Platform usage is included. You can cap the cost of any run in the run options.

**Is it legal to scrape Substack? What about robots.txt and GDPR?**
The Actor reads only public data from paths that Substack's `robots.txt` allows, never logs in and
cannot see paywalled text. The only personal data kept is the author's public display name and
handle. Scraping public data is generally lawful, but you are responsible for how you use it: in
the EU you need a lawful basis (for example legitimate interest for research), and republishing
someone's posts needs their permission. See *Legal, privacy and robots.txt* above. This is not
legal advice.

**Can I export Substack posts to CSV, Excel or Google Sheets?**
Yes. Every run's dataset downloads as CSV, Excel, JSON, XML or HTML from the Output tab. For
Google Sheets, import the CSV, use the *Google Sheets Import & Export* Actor from the Apify Store or a Make/Zapier scenario, or
point `=IMPORTDATA()` at the API URL of the last run's dataset (keep in mind the URL contains your
API token).

**Can I get new posts automatically every day or week?**
Yes. Create a schedule in Apify Console (Schedules) with `sort: "new"` and `publishedAfter: "7 days"`
(or `1 day`) so each run only saves recent posts. Add a webhook or a Make/Zapier/n8n integration to
send them to Slack, e-mail or a sheet.

**Can I call it from my code or an AI agent?**
Yes. `POST https://api.apify.com/v2/acts/davidts~substack-posts-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN`
with the input as JSON returns the posts directly; the Apify JavaScript and Python clients and
Apify's MCP server (for Claude, ChatGPT, Cursor and other agents) work too. See *Integrations*.

**Does it scrape paywalled posts, comments or subscriber counts?**
No. Paywalled posts come with full metadata and the public preview only. Comments, Notes,
subscriber counts and recommendations are not collected.

If this Actor saves you time, a short **review on its Store page** helps other people find it -
thank you! Found a bug or need a field? Open an issue on the Issues tab and I will look at it.

### Changelog

The full history is in the Actor's `CHANGELOG.md` (also shown in the Information tab in Apify
Console).

- **0.1** (2026-10) - first release: archive paging, custom-domain handling, paywall detection,
  optional post bodies, podcast fields, `publishedAfter` cut-off, type filter, pay-per-event pricing.

# Changelog

This Actor's version history is a separate document: https://apify.com/davidts/substack-posts-scraper/changelog.md

# Actor input Schema

## `publications` (type: `array`):

Substack publications to scrape. Each entry can be a full URL (`https://www.lennysnewsletter.com`), a Substack subdomain (`lenny` or `lenny.substack.com`), or a custom domain (`newsletter.pragmaticengineer.com`). Publications that moved to a custom domain are followed automatically. Tip: copy the address shown in your browser when you open the newsletter.

## `maxPostsPerPublication` (type: `integer`):

Maximum number of posts to save for each publication. Use `0` to scrape the whole archive. Each saved post is one dataset item (and one paid result). For a quick sample use 10-20; for a full export use 0.

## `sort` (type: `string`):

`new` returns the newest posts first (chronological). `top` returns posts in Substack's own popularity order, which is not chronological, so `publishedAfter` cannot be combined with it.

## `includePostBody` (type: `boolean`):

Fetch every post individually and add `bodyHtml` and `bodyText`. Free posts get the full body; paywalled posts get only the public preview (and `paywalled: true`). This roughly doubles the number of requests and makes runs slower. Turn it on when you need the text (for analysis, AI or search); leave it off for titles, dates and engagement only. The price per post is the same.

## `publishedAfter` (type: `string`):

Only save posts published on or after this date and stop paging once older posts are reached. Accepts an absolute date (`2026-01-31` or full ISO 8601 such as `2026-01-31T10:00:00Z`, interpreted in UTC) or a relative period (`7 days`, `2 weeks`, `3 months`). Only valid with the `new` sort order. For a scheduled daily or weekly run, use `1 day` or `7 days` to save only new posts.

## `postTypes` (type: `array`):

Only keep posts of these types. Leave empty to keep all types (newsletter, podcast, video, thread).

## `proxyConfiguration` (type: `object`):

Proxies used for requests to Substack. Apify datacenter proxies (the default) work for normal volumes. If you see repeated 403/429 errors in the log, switch to residential proxies, which cost more but are rarely blocked.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com",
    "newsletter.pragmaticengineer.com"
  ],
  "maxPostsPerPublication": 20,
  "sort": "new",
  "includePostBody": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

One item per scraped Substack post (JSON, CSV or Excel via the format parameter)

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com",
        "newsletter.pragmaticengineer.com"
    ],
    "maxPostsPerPublication": 20,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("davidts/substack-posts-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": [
        "https://www.lennysnewsletter.com",
        "newsletter.pragmaticengineer.com",
    ],
    "maxPostsPerPublication": 20,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("davidts/substack-posts-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com",
    "newsletter.pragmaticengineer.com"
  ],
  "maxPostsPerPublication": 20,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call davidts/substack-posts-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,davidts/substack-posts-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/w55hLsEAYg9Ke7olR/builds/Tt4hMz6J0M2gPYXBS/openapi.json