# Substack Scraper — Posts, Authors, Likes, Comments (`architekture/substack-scraper`) Actor

Substack newsletter scraper (substack.com and custom domains): posts, authors, likes, comment counts, restacks, word counts, tags, dates, free-post full text, comment threads, subscriber bands. By publication name or URL, keyword, date; new-post watch mode. Pay per result, no login.

- **URL**: https://apify.com/architekture/substack-scraper.md
- **Developed by:** [architek.earth](https://apify.com/architekture) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper — Posts, Authors, Likes, Comments

**Recorded run: 3 posts with body and comments from 1 publication in 8 seconds** (2026-10-22, build 1.1.6, run `7RQyEg3nTQ1eWpE6i`). A bounded canary sample, not a full-catalogue benchmark.

architek.earth — an independent data studio. Pay-per-result scrapers with recorded runs, published canaries and stable schemas, built for people and AI agents.

Substack newsletter posts, authors, likes, comments, restacks and full free-post text from any publication you name, on `*.substack.com` or a custom domain, plus a watch mode that delivers only new posts.

| Source | Market | From | Standout fields |
|---|---|---|---|
| Substack (substack.com and Substack-hosted custom domains) | Any publication on Substack | $1.00 / 1,000 posts | `likes` / `comments` / `restacks`, `bodyText`, `commentThread` |

### ✅ Why choose this scraper

#### Substack posts with likes, comments and restacks in one row

Every post row carries `title`, `subtitle`, `authors` (name and handle), `postDate`, `likes`, `comments`, `restacks`, `wordcount`, `tags`, `coverImageUrl`, `audience` and `isPaywalled`; one archive request returns up to 50 posts.

#### Full text of free Substack posts

With `includeBody: true` each row adds `bodyHtml`, `bodyText` and `bodyScope` (`full` for free posts, `preview` for paid-only ones).

#### Substack comment threads with replies linked

With `includeComments: true` each row adds `commentThread`: `id`, `parentId`, `author`, `date`, `body` and `likes` per comment, capped by `maxCommentsPerPost`, with `commentThreadTruncated` set when the cap cut the thread.

#### Custom-domain newsletters by name or URL

`astralcodexten`, `noahpinion.substack.com` and `https://www.lennysnewsletter.com` all work; `astralcodexten` was followed to `www.astralcodexten.com` on 2026-10-22, and `host` shows the host that served the data.

#### Measured on a schedule

Two canary tasks run daily on the live site (20 archive posts from Astral Codex Ten; 3 Noahpinion posts with body and comments); the hero line above is the latest green run.

### ⚠️ Worth knowing up front

- Paywalled bodies are not returned. For paid-only posts you get the metadata and Substack's public preview, with `isPaywalled: true` and `bodyScope: "preview"`. On 2026-10-22 a paid-only Noahpinion post with `comments: 44` returned an empty `commentThread`.
- `likes`, `comments` and `restacks` are Substack's own counters at collection time. `comments` is Substack's count; `commentThread` is what its comment endpoint returned, capped at `maxCommentsPerPost`.
- `subscriberHint` on the free `publication` row is the band Substack shows on the homepage ("Hundreds of thousands of subscribers"), not an exact count, and is `null` when the homepage shows none.
- `searchTerm` uses Substack's own archive search inside each publication. Results and their order are Substack's matches, not a full-text index we control.
- Requests are sequential and at least 0.5 seconds apart, with up to 3 attempts per request. Bodies and comments each add one request per post. A failed body or comment fetch keeps the post as a `post` row at the lower price with `detailUnavailable: true`.
- A run stops cleanly 120 seconds before its timeout (at most 6 hours) and ends `PARTIAL`, keeping what it delivered; an abort or migration ends it `LIMITED`. A publication that answers 401/403 (`BLOCKED`), 429 (`RATE_LIMITED`) or 5xx (`SOURCE_UNAVAILABLE`) gets a status row and the run ends `PARTIAL` with the other publications' rows kept.
- Free status rows (`"recordType": "status"`) are written for invalid input (`INVALID_INPUT`), a `stateKey` held by another run (`STATE_BUSY`), a publication that is unknown (`NOT_FOUND`), not on Substack (`NOT_SUBSTACK`), refused, rate-limited or unavailable (`BLOCKED`, `RATE_LIMITED`, `SOURCE_UNAVAILABLE`) or has no posts for the filters (`NO_POSTS`), and at the spending limit (`BUDGET_LIMIT`). A deadline stop or an abort writes no status row; the outcome is in `RUN_SUMMARY`. A run whose delivery acknowledgement was lost ends `RECONCILIATION_REQUIRED` as a failed run and replays nothing.

### ⚡ Try the example

This actor has no free preview; the example below costs about $0.04.

Open the actor's input page, switch to JSON, paste this and start the run:

```json
{
  "publications": ["astralcodexten", "https://www.lennysnewsletter.com"],
  "maxPostsPerPublication": 20
}
```

Set `maxPostsPerPublication: 0` for the whole archive; add `"includeBody": true` or `"includeComments": true` for text and comment threads.

### ⚙️ Input

| Parameter | Type | Default | Description |
|---|---|---|---|
| `publications` | array | — (prefill `["astralcodexten", "https://www.lennysnewsletter.com"]`) | Substack names, hosts or custom-domain URLs. Up to 100; duplicates are merged. Required. |
| `maxPostsPerPublication` | integer | `20` | Posts delivered per publication, 0 to 10,000; `0` = no cap (whole archive). |
| `sort` | select | `new` | `new` (newest first) or `top` (Substack's ranking). |
| `searchTerm` | string | — | Substack archive search within each publication, up to 200 characters. |
| `postedAfter` | string | — | ISO date such as `2026-01-01`. With `new`, paging stops at the first older post; with `top`, older posts are skipped. |
| `includeBody` | boolean | `false` | Adds `bodyHtml`, `bodyText`, `bodyScope`. Rows are charged as `post-detail`. |
| `includeComments` | boolean | `false` | Adds `commentThread`. Rows are charged as `post-detail`. |
| `maxCommentsPerPost` | integer | `100` | Comment cap per post, 1 to 2,000. |
| `incrementalMode` | boolean | `false` | Watch: deliver only posts newer than the last delivered post per publication. Needs `stateKey` and `sort: "new"`. |
| `stateKey` | string | — | Name for saved watch state and checkpoints in your own account, up to 100 characters. Use a different key for different inputs. |
| `continuePreviousRun` | boolean | `false` | Resume an unfinished collection saved under `stateKey`. An unfinished watch resumes on its own. |
| `proxyConfiguration` | proxy | — | Optional Apify proxy. Not used by default; the canaries run without one. |
| `feedback` | string | — | Optional note for the maintainers. It is logged and read by a human, and it never changes output. |

Unknown input fields are rejected with an `INVALID_INPUT` status row.

### 🧱 Stable fields

These top-level fields keep the same name and the same kind of value in every release. Fields are only ever added, never renamed or removed, so an n8n, Make, Zapier or spreadsheet pipeline can bind to them.

| Field | Type | Meaning |
|---|---|---|
| `publication` | string, nullable | Publication name when its homepage metadata was readable. |
| `host` | string | Host that served the post. |
| `postId` / `slug` / `url` | number / string / string | Substack's post id, slug and canonical URL. |
| `title` / `subtitle` / `description` | string, nullable | Headline, subtitle and SEO description. |
| `type` | string | `newsletter`, `podcast`, `thread`, `video` and other Substack post types. |
| `audience` / `isPaywalled` | string, nullable / boolean | `everyone` is free; any other audience is paywalled. |
| `authors` | array | `[{name, handle}]` from the bylines. |
| `postDate` | string, nullable | UTC ISO 8601 publish time. |
| `likes` / `comments` / `restacks` | number | Substack's counters at collection time. |
| `wordcount` | number, nullable | Article length reported by Substack. |
| `tags` | array | Post tag names. |
| `coverImageUrl` / `audioUrl` | string, nullable | Cover image; podcast audio when present. |
| `scrapedAt` | string | UTC collection time. |
| `bodyHtml` / `bodyText` / `bodyScope` | string, nullable | `post-detail` rows with `includeBody`; `bodyScope` is `full` or `preview`. |
| `commentThread` | array, nullable | `post-detail` rows with `includeComments`: `[{id, parentId, author, date, body, likes}]`. |
| `commentThreadTruncated` | boolean | `true` when `maxCommentsPerPost` cut the thread. |
| `recordType` | string | `"status"` on free status rows; `"post"` or `"post-detail"` on charged rows; `"publication"` on free publication rows. |

Free `publication` rows carry `host`, `name`, `description`, `authorName`, `subscriberHint`, `customDomain`, `subdomain`, `language`, `firstPostDate`, `url` and `inputRef`. Status rows carry `inputRef`, `status`, `message` and `scrapedAt`. Charged rows also carry `deliveryId` (unique per delivered row) and, when a detail fetch failed, `detailUnavailable`. Other fields are extras, not part of the contract. To have one pinned, say so in `feedback`.

### 📦 Output example

Abridged real row from run `7RQyEg3nTQ1eWpE6i` on 2026-10-22; values are a historical snapshot.

```json
{
  "recordType": "post-detail",
  "publication": "Noahpinion",
  "host": "www.noahpinion.blog",
  "postId": 221088139,
  "slug": "can-a-million-ai-agents-create-superintelligence",
  "title": "Can a Million AI Agents Create Superintelligence?",
  "subtitle": "Ramez Naam gives a skeptic’s take on agent swarms.",
  "type": "newsletter",
  "audience": "everyone",
  "isPaywalled": false,
  "authors": [{"name": "Ramez Naam", "handle": "rameznaam"}],
  "postDate": "2026-10-19T04:33:30.712Z",
  "likes": 241,
  "comments": 18,
  "restacks": 24,
  "wordcount": 7178,
  "tags": [],
  "url": "https://www.noahpinion.blog/p/can-a-million-ai-agents-create-superintelligence",
  "bodyText": "Art by GPT-6\n\nRamez Naam is, in my opinion, one of the best futurists alive. …",
  "bodyScope": "full",
  "commentThread": [
    {"id": 364765360, "parentId": null, "author": "Jürgen Boß", "date": "2026-10-19T11:28:43.317Z", "body": "True intelligence is about extracting information from the environment. …", "likes": 7}
  ],
  "commentThreadTruncated": false,
  "scrapedAt": "2026-10-22T06:53:04.958752+00:00"
}
```

A free status row looks like this (run `qcRRhWwgvT5R3J18y`, 2026-10-22):

```json
{"recordType": "status", "inputRef": "run", "status": "BUDGET_LIMIT", "message": "Spending limit reached. Raise the limit, or set a stateKey and continuePreviousRun=true to resume", "scrapedAt": "2026-10-22T09:39:40.592611+00:00"}
```

The run's status message reads `Done: N rows, M details, K status rows (STATUS)`, and the `RUN_SUMMARY` record of its key-value store holds the outcome (run `7RQyEg3nTQ1eWpE6i`, abridged):

```json
{"status": "COMPLETE", "delivered": 3, "charged": {"post": 0, "post-detail": 3}, "publicationRows": 1, "statusRows": 0, "errors": 0, "detailUnavailable": 0, "chargeUsd": "0.009", "checkpointComplete": true}
```

### 🧠 Advanced usage

#### Watch Substack newsletters for new posts

Schedule `{"publications": [...], "incrementalMode": true, "stateKey": "my-watch", "maxPostsPerPublication": 20}`. The first run delivers up to 20 newest posts per publication; later runs deliver only newer posts and stop paging at the first one already delivered. A rerun with nothing new delivered nothing and charged $0 (runs `Uylmh7f4RxBj47l76` then `JpJcGKMubtHGZQoh6`, 2026-10-21). Changing `publications`, `sort`, `searchTerm`, `postedAfter`, `includeBody` or `includeComments` under the same key is rejected; use a new key.

#### Export a whole Substack archive within a budget

Set `maxPostsPerPublication: 0`, a `stateKey` and a spending limit. At the limit the run ends `BUDGET_LIMIT`; run again with `continuePreviousRun: true` to carry on from the saved checkpoint. A hard-aborted run of 100 posts resumed with 140 new posts and no repeated post or charge (runs `jQVcmzvojzYR7cXpY` then `B3NQ71Pk0Y25qyolb`, 2026-10-21). Check `RUN_SUMMARY.checkpointComplete` before treating an archive as finished.

#### Search several Substack newsletters for one topic

Combine `searchTerm` with several `publications` and `includeBody: true`, then join on `url`. On 2026-10-22 `searchTerm: "AI"` across Astral Codex Ten and Lenny's Newsletter returned 5 matches each.

#### Collect Substack comments on recent posts

Use `postedAfter` with `includeComments: true` and a `maxCommentsPerPost` cap; `parentId` rebuilds reply chains and `likes` ranks comments.

### 🔌 Integrations

Results land in a dataset you can export as JSON, CSV or Excel. They can also flow to Google Sheets, Make, Zapier or n8n, or to a webhook when a run finishes. The Apify Scheduler runs it on a timetable. Daily with `incrementalMode` suits watching newsletters for new posts; read `RUN_SUMMARY` as well as the platform run status. Copy-paste code in Python, Node, curl and n8n: [architek-earth/examples/substack-scraper](https://github.com/architek-earth/examples/tree/main/substack-scraper).

### 💸 Pricing

Pay per result. Platform and proxy usage are included. Status rows, publication rows and failed requests are free. Each delivered post is charged once, after it is written; with a `stateKey`, a run resumed with `continuePreviousRun` (or an unfinished watch) skips posts already delivered and does not charge them again, and a batch whose charge was unconfirmed in the previous run is recorded in `RUN_SUMMARY.recoveredUncertainCharges` instead of being charged again.

| Event | Price | What it covers |
|---|---|---|
| `post` | $1.00 / 1,000 posts | One delivered archive post, including a post kept at this price when its body or comment fetch failed. |
| `post-detail` | $3.00 / 1,000 posts | One delivered post with body and/or comment thread (`includeBody` or `includeComments`). |

1,000 archive posts cost $1.00; 1,000 posts with bodies or comments cost $3.00.

### ❓ FAQ

**Is there a Substack API for newsletter posts?** This actor is one: it reads the public JSON endpoints that Substack publications serve and returns posts as JSON through the Apify API, callable from any language or from n8n, Make and Zapier.

**Do I need a Substack account or login?** No. No account, cookies or API key; only anonymous public requests.

**Can it scrape paid Substack posts?** No. Paid-only posts return metadata and Substack's public preview with `bodyScope: "preview"`; nothing behind the paywall is bypassed.

**Does it work on custom-domain newsletters?** Yes, for publications hosted on Substack, such as `https://www.lennysnewsletter.com`. A site that is not on Substack gets a free `NOT_SUBSTACK` status row.

**Can I get a newsletter's subscriber count?** Only the public band Substack shows, in `subscriberHint` on the free `publication` row, for example "Hundreds of thousands of subscribers". It is not an exact count.

**Can it scrape Substack Notes, the leaderboard or find newsletters by topic?** No. It collects posts, comments and publication metadata from publications you name; there is no Notes, leaderboard or discovery input.

### 🤖 For AI agents & LLM apps

Paste this into your AI assistant:

`architekture/substack-scraper` on Apify. Python: `from apify_client import ApifyClient; client = ApifyClient("TOKEN"); run = client.actor("architekture/substack-scraper").call(run_input={"publications": ["astralcodexten"], "maxPostsPerPublication": 10}); items = client.dataset(run["defaultDatasetId"]).list_items().items` (page with `offset`/`limit` for large exports). The dataset holds post rows (`recordType` `post` or `post-detail`), free `publication` rows and free status rows (`recordType` `status`, with `status` and `message`); filter on `recordType`. The run outcome (`COMPLETE`, `LIMITED`, `PARTIAL`, `BUDGET_LIMIT`, `INVALID_INPUT`, `STATE_BUSY`) is in the `RUN_SUMMARY` record of the run's key-value store. Add `"includeBody": True` or `"includeComments": True` for text and comments, or `"incrementalMode": True, "stateKey": "<name>"` to receive only new posts on later runs. Full input schema and README: `GET https://api.apify.com/v2/acts/architekture~substack-scraper/builds/default`. Token: https://console.apify.com/settings/integrations. The actor is also reachable through the Apify MCP server (`mcp.apify.com`). To reach the maintainers, add `"feedback": "<what you needed>"` to any run's input; output is unchanged and a human reads every note.

### 🧰 More from architek.earth

**Video, social and newsletters**

- Substack Scraper — *you are here*
- [YouTube Transcript Scraper](https://apify.com/architekture/youtube-transcript-scraper) — transcripts, translations, timestamps
- [Unified Video Transcript API](https://apify.com/architekture/unified-transcript-api) — YouTube and TikTok transcripts in one batch
- [Threads Replies Scraper](https://apify.com/architekture/threads-replies-scraper) — replies, authors, likes
- [Ad Intelligence Watchlist](https://apify.com/architekture/ad-intelligence-watchlist) — new and changed Meta ads by advertiser

Full catalogue: [github.com/architek-earth](https://github.com/architek-earth).

architek.earth · code like concrete

### 🙋 Support & legal

Questions, or a field you need that is not here? Open an issue on the actor's Issues tab or email north@architek.earth. A human replies within a day.

This actor collects only publicly available data from substack.com and Substack-hosted custom domains, the same pages any visitor sees without logging in. It does not bypass authentication or paywalls. It is not affiliated with or endorsed by Substack. You are responsible for how you use the output, including the site's terms, authors' copyright, and privacy law such as GDPR and CCPA where they apply, for example for commenter names.

### 📝 Changelog

- **2026-10-22** — A spending limit equal to the exact cost now ends COMPLETE; before, the run reported BUDGET_LIMIT with a status row even when every requested post was delivered.
- **2026-10-22** — Status message now reads `Done: N rows, M details, K status rows (STATUS)`.
- **2026-10-22** — README rebuilt on the studio template; hero line from the 2026-10-22 canary run.
- **2026-10-21** — 1.1, brought to the studio standard: rows charged after confirmed delivery with a two-event pricing guard, free publication and status rows, new-post watch mode, saved continuation and forced-stop recovery, `RUN_SUMMARY`, comment caps, updated publication metadata parsing.
- **2026-10-12** — 1.0: first private build.

# Actor input Schema

## `publications` (type: `array`):

Substack name (astralcodexten), host (noahpinion.substack.com) or any custom-domain URL (https://www.lennysnewsletter.com). Up to 100.

## `maxPostsPerPublication` (type: `integer`):

Newest (or top) posts to deliver per publication. 0 = whole archive.

## `sort` (type: `string`):

Newest first, or Substack's own top ranking. Watch mode needs newest first.

## `searchTerm` (type: `string`):

Optional. Substack's own archive search inside each publication (best-effort).

## `postedAfter` (type: `string`):

Optional ISO date. With newest-first sorting, paging stops at the first older post.

## `includeBody` (type: `boolean`):

Adds bodyHtml and bodyText. Paywalled posts return only Substack's public preview. Charged as post-detail.

## `includeComments` (type: `boolean`):

Adds the comment thread (flattened, parentId links replies). Charged as post-detail.

## `maxCommentsPerPost` (type: `integer`):

Cap on comments kept per post when comments are included.

## `incrementalMode` (type: `boolean`):

Requires stateKey and newest-first sort. Delivers only posts newer than the last delivered post per publication.

## `stateKey` (type: `string`):

Account-isolated saved state. Use a different key for different inputs.

## `continuePreviousRun` (type: `boolean`):

Resume an unfinished collection saved under stateKey (for example after a spending limit).

## `feedback` (type: `string`):

Logged for human review; does not change output.

## `proxyConfiguration` (type: `object`):

No proxy by default; Substack's public API answers datacenter requests. A proxy adds cost.

## Actor input object example

```json
{
  "publications": [
    "astralcodexten",
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 20,
  "sort": "new",
  "searchTerm": "",
  "postedAfter": "",
  "includeBody": false,
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "incrementalMode": false,
  "stateKey": "",
  "continuePreviousRun": false,
  "feedback": ""
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "astralcodexten",
        "https://www.lennysnewsletter.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("architekture/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": [
        "astralcodexten",
        "https://www.lennysnewsletter.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("architekture/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "astralcodexten",
    "https://www.lennysnewsletter.com"
  ]
}' |
apify call architekture/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,architekture/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/CEKgWlkIKWz34S2kL/builds/vbQOoYnpX7A75bd0e/openapi.json