# Substack Newsletter Scraper: Posts, Archive, Publication Data (`titian_fluorite/substack-scraper`) Actor

Export any Substack newsletter's public posts, archive and publication details as clean JSON. Body text, engagement counts, comments. Pay per post.

- **URL**: https://apify.com/titian\_fluorite/substack-scraper.md
- **Developed by:** [Gerald Dobin](https://apify.com/titian_fluorite) (community)
- **Categories:** News, Social media, Marketing
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 scraped posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Substack Newsletter Scraper: Posts, Archive and Publication Data

Give this Actor a list of Substack newsletters and it gives you back their public posts as clean,
structured data. For every post you get the title, subtitle, URL, publication date, author bylines,
post type, cover image, the full article body as both HTML and plain text, and the public
engagement counts that Substack shows on the page: likes, comments and restacks. It reads the same
public JSON that a Substack page loads in your browser, so there is no login, no account and no
browser automation involved. Custom domains such as `www.lennysnewsletter.com` work, plain
`something.substack.com` addresses work, and so does a bare handle like `lenny`. Results land in a
standard Apify dataset that you can download as JSON, CSV or Excel, or pull straight from the API.

### Who it is for

**Media monitoring and research across many newsletters.** Point the Actor at the fifty newsletters
that matter in your field, schedule it daily, and you have a searchable archive of everything they
published, with dates and authors attached. Analysts and communications teams use this to see when
a topic starts moving and who moved it first.

**Writers and analysts tracking a beat.** Pull the whole back catalogue of two or three publications
you follow, then search the plain text body offline instead of scrolling an archive page. The
`likes`, `commentCount` and `restacks` fields let you sort by what actually landed with readers,
which is a fast way to find the questions a beat keeps returning to.

**Marketers looking for sponsorship or partnership targets.** Export a shortlist of newsletters,
sort by engagement, and see publishing frequency, typical post length and which authors write what.
That is enough to size an audience and pick the three newsletters worth a real conversation,
without opening fifty tabs.

### What you get

One dataset row per post. Here is a real row from a test run, with the body fields shortened so the
example stays readable:

```json
{
  "rowType": "post",
  "publication": "newsletter.pragmaticengineer.com",
  "publicationUrl": "https://newsletter.pragmaticengineer.com",
  "publicationName": "The Pragmatic Engineer",
  "postId": 213733051,
  "slug": "the-pragmatic-engineer-five-years",
  "url": "https://newsletter.pragmaticengineer.com/p/the-pragmatic-engineer-five-years",
  "title": "The Pragmatic Engineer: Five years",
  "subtitle": "As the newsletter reaches its fifth birthday, we reflect on how the publication has changed, and what to expect.",
  "description": "As the newsletter reaches its fifth birthday, we reflect on how the publication has changed, and what to expect.",
  "type": "newsletter",
  "audience": "everyone",
  "paywalled": false,
  "authors": [
    { "name": "Gergely Orosz", "handle": "pragmaticengineer", "url": "https://substack.com/@pragmaticengineer" }
  ],
  "publishedAt": "2026-09-01T17:14:34.151Z",
  "updatedAt": "2026-09-01T17:27:08.159Z",
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/e6e5687b-9265-4270-ab3f-84ad0059e6b1_420x300.png",
  "bodyHtml": "<p><span>Wow, has it </span><em><span>already</span></em><span> been five years?...",
  "bodyText": "Wow, has it already been five years?! The newsletter hits a big milestone this week...",
  "bodyIsPreview": false,
  "bodyOmittedReason": null,
  "wordCount": 3207,
  "likes": 254,
  "commentCount": 21,
  "restacks": 5,
  "podcastUrl": null,
  "scrapedAt": "2026-09-07T22:56:05.587Z"
}
```

Field notes worth knowing before you build on this:

- `audience` is the publisher's own setting: `everyone`, `only_free`, `only_paid` or `founding`.
- `paywalled` is `true` for anything above `everyone`. `bodyIsPreview` tells you whether the body
  you received is the publicly readable preview rather than the whole article, and it is empty
  whenever no body was returned at all.
- `bodyOmittedReason` is normally empty. It is filled in on the rare occasion the body was left out
  on purpose, which is explained under Data policy below.
- `podcastUrl` is filled in for posts of type `podcast` and empty otherwise.
- `authors` is a list, because Substack posts often carry more than one byline.
- `error` on a post row means part of that post could not be fetched, so the row carries the
  listing fields and no body.
- A publication that cannot be read produces a single row with `rowType: "error"` and a plain
  explanation. Those rows are free.

Turning on **Include public comments** adds rows with `rowType: "comment"` carrying `postId`,
`commentId`, `parentCommentId`, `authorName`, `authorHandle`, `body`, `likes` and `publishedAt`.
Replies point at their parent through `parentCommentId`, so you can rebuild a thread.

### How to use it

Paste your newsletters into **Substack publications** and press Start. Everything else has a
sensible default.

| Input | What it does |
| --- | --- |
| `publications` | The newsletters to scrape. Full URL, `name.substack.com`, or just the handle. |
| `maxPostsPerPublication` | How many posts per newsletter. `0` takes the whole archive. Default 50. |
| `sort` | `new` for the latest issues, `top` for the newsletter's most popular posts of all time. |
| `postedAfter` | Optional date. Older posts are skipped and never billed. |
| `includeBody` | On by default. Turn it off for a faster, cheaper run with metadata only. |
| `includeComments` | Off by default. Adds the public comment threads as extra rows. |
| `postTypes` | Keep only `newsletter`, `podcast`, `thread` or `video` posts. |

**Scheduling for monitoring.** For a daily digest, set `sort` to `new`, set
`maxPostsPerPublication` to something small like 10, and add a schedule in the Apify console. Set
`postedAfter` to yesterday's date if you want a strict window and the smallest possible bill. Each
run appends to its own dataset, so you can diff runs or push them into your own store.

**Run it from the API.** Replace the token and the Actor id with your own:

```bash
curl -X POST "https://api.apify.com/v2/acts/YOUR_USERNAME~substack-scraper/runs?token=YOUR_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "publications": ["https://www.lennysnewsletter.com", "astralcodexten.substack.com"],
    "maxPostsPerPublication": 25,
    "sort": "new",
    "includeBody": true
  }'
```

Then fetch the results:

```bash
curl "https://api.apify.com/v2/datasets/YOUR_DATASET_ID/items?format=csv&token=YOUR_TOKEN"
```

**Use it from an AI assistant.** Apify Actors are available over MCP, so an assistant that speaks
the Model Context Protocol can call this Actor as a tool and read the dataset back. That makes
questions like "what did these five newsletters publish about pricing this month" answerable
without writing any glue code.

### Pricing

You pay per post delivered. No monthly subscription, no platform usage charge on top, no minimum.

- Each post row is one billable event.
- Each comment row is one billable event, and comments are off by default.
- Error rows are free. If a publication cannot be read, you are not charged for it.
- Posts filtered out by `postedAfter` or `postTypes` are never fetched and never billed.

The practical effect is that a small test run costs a small amount, and you can predict a big run
exactly: it is the number of publications times `maxPostsPerPublication`, capped by how many posts
those newsletters have actually published.

### Data policy and limits

This Actor reads only what a logged out visitor can see.

- **Public data only.** It calls Substack's public JSON endpoints and reads public pages. Every
  request is a fresh logged out request: no login, no cookies, no session, no authorization header
  and no attempt to work around any access control.
- **Paywalled posts return the free preview only.** When a publisher marks a post for paid
  subscribers, Substack serves the public the opening section. That is what you get, and the row is
  flagged with `paywalled: true` and `bodyIsPreview: true` so you never mistake a preview for a
  full article. If a subscriber-only post ever came back looking complete, the body is withheld
  entirely and `bodyOmittedReason` says so, rather than passing on text a subscriber paid for. The
  rest of the row still arrives, so you keep the title, date, author and engagement counts.
- **No subscriber or member data.** Subscriber lists, email addresses and anything else behind a
  publisher's account are out of scope and are never collected.
- **Substack publications only.** Before anything is scraped or billed, the address has to prove it
  is a real Substack: the page has to carry Substack's own markers and the archive has to return
  Substack post records that belong to that publication. A site that has left Substack, or never
  was one, gets a free error row instead of scraped content. Private and internal addresses are
  refused outright.
- **Comments are the public ones.** Only comments visible to a logged out reader are returned, and
  deleted comments are skipped.
- **What the endpoints will give.** Substack serves the archive in pages of 23 posts, so very large
  archives take proportionally more requests. Addresses that do not exist produce a free error row
  explaining what happened. The same newsletter given twice, once as a custom domain and once as a
  substack.com address, is recognised as one publication and scraped once.
- **Sensible ceilings.** Up to 1,000 publications per run, up to 10,000 posts per publication, up
  to 5,000 comments per post. Nothing is ever delivered or billed twice.
- **Be polite and lawful.** Requests are rate limited by default. Newsletter text is the property
  of its authors: use it for research, monitoring and analysis, and respect copyright when you
  republish anything.

### Support

Something wrong, missing or confusing? Open an issue on the Actor's Issues tab with the input you
used and the publication that gave you trouble, and it will be looked at. Feature requests are
welcome, especially if you can say what decision the extra field would help you make.

# Actor input Schema

## `publications` (type: `array`):

The newsletters to scrape. Paste a full address such as https://www.lennysnewsletter.com, a Substack subdomain such as astralcodexten.substack.com, or just the handle, for example lenny. Custom domains and substack.com addresses both work.

## `maxPostsPerPublication` (type: `integer`):

How many posts to return for each newsletter. Set 0 to take the whole archive. Every post you receive is billed, so start small when you are trying the Actor out.

## `sort` (type: `string`):

Newest first gives you the latest issues, which is what you want for monitoring. Most popular gives you the newsletter's best performing posts of all time.

## `postedAfter` (type: `string`):

Optional date filter, for example 2026-01-01. Posts published before this date are skipped and never billed. Leave empty to ignore dates.

## `includeBody` (type: `boolean`):

Fetch the full article for every post and return it as HTML and as plain text. Turn this off for a faster, lighter run that returns titles, dates, authors and engagement counts only.

## `includeComments` (type: `boolean`):

Also return the public comment threads under each post as extra rows. Comments are billed separately, so this can add up on busy newsletters.

## `postTypes` (type: `array`):

Limit the results to certain kinds of post. Newsletter is a normal written issue. Leave all four selected to get everything.

## `maxConcurrency` (type: `integer`):

How many requests to run at once. The default is polite and works for every newsletter. Raise it only if you are scraping many publications and the run feels slow.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com",
    "https://www.astralcodexten.com",
    "newsletter.pragmaticengineer.com"
  ],
  "maxPostsPerPublication": 50,
  "sort": "new",
  "includeBody": true,
  "includeComments": false,
  "postTypes": [
    "newsletter",
    "podcast",
    "thread",
    "video"
  ],
  "maxConcurrency": 5
}
```

# Actor output Schema

## `posts` (type: `string`):

Rows with rowType post or comment: title, url, authors, publishedAt, bodyText, likes, commentCount, restacks, paywalled and more.

## `postsCsv` (type: `string`):

The same rows as CSV for spreadsheets.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com",
        "https://www.astralcodexten.com",
        "newsletter.pragmaticengineer.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("titian_fluorite/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": [
        "https://www.lennysnewsletter.com",
        "https://www.astralcodexten.com",
        "newsletter.pragmaticengineer.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("titian_fluorite/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com",
    "https://www.astralcodexten.com",
    "newsletter.pragmaticengineer.com"
  ]
}' |
apify call titian_fluorite/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,titian_fluorite/substack-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/e50dhEX9z43gW7BXQ/builds/8tX7tg31PaIUaVsbf/openapi.json