# Substack Scraper — Posts, Comments & Newsletters (`pixflor/substack-scraper`) Actor

Scrape Substack publications: complete post archives with no silently skipped pages, full comment threads including nested replies, publication discovery by topic, and the recommendation graph. Respects paywalls by design.

- **URL**: https://apify.com/pixflor/substack-scraper.md
- **Developed by:** [Pixflor](https://apify.com/pixflor) (community)
- **Categories:** Social media, Lead generation, MCP servers
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 row scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Substack Scraper — Posts, Comments & Newsletters

Substack as clean rows: complete post archives, full comment threads with replies, topic-based
publication discovery, and the recommendation graph between newsletters.

No API key. No login. Paste a name, press start, export CSV or JSON.

***

### Three things this does that other Substack scrapers don't

#### 1. It doesn't silently skip posts

Substack's archive endpoint accepts `limit` up to 50 — ask for 51 and you get a hard HTTP 400,
so 50 looks like the safe number. But at the first page it returns only **23 rows**, with
HTTP 200 and no hint it short-changed you. Page by `offset += 50` and you jump straight from
row 23 to row 50, never seeing the posts in between.

| Approach | Posts retrieved |
|---|---|
| Page by the limit you asked for | **346** |
| Page by the rows actually returned | **373** — complete, no duplicates, 9 requests |

That's **7.2% of the archive missing** and nothing anywhere saying so. Reproduced identically
on Big Technology, Astral Codex Ten and The Pragmatic Engineer.

The same trick appears again under load: Substack throttles by returning **HTTP 200 with an
empty array** rather than a 429, which reads exactly like "end of archive". This Actor
confirms every unexpected empty response with backoff before believing it.

#### 2. Its comment threads are actually complete

The comments endpoint returns top-level comments only — replies are nested inside them,
recursively, several levels deep. Read the array you get back and stop, and a thread of 878
comments gives you **65 rows**.

This Actor flattens the tree: **881 rows**, each with its depth, parent and reply count,
nesting up to 20 levels deep. Watch out for `child_comment_count` too — it counts *top-level*
comments, not replies, which invites precisely the wrong assumption.

#### 3. It respects the paywall — on purpose

Substack's per-post endpoint hands over the **complete text of subscriber-only posts**.
Measured: a post marked `only_paid` came back with 1,116 words against a stated word count of
1,121. The whole article, free.

**This Actor will not give you that.** Paywalled posts return their metadata, engagement
numbers and the public teaser Substack itself displays, with `paywalled: true` and
`bodyStatus: "withheld_paywalled"`. Free posts return their full text, because that is public.

That's a deliberate boundary. The paid body is the writer's livelihood, republished paywalled
text isn't something you could safely use commercially anyway, and a scraper that harvests it
is one complaint away from disappearing. You get every signal *about* paid posts — what they
cover, how they perform, when they run — without the liability.

***

### What you get

**Posts**

| Column | Example |
|---|---|
| `title` / `subtitle` | Why AI Margins Are Collapsing |
| `publication` / `publicationId` | `bigtechnology` |
| `url` | canonical post link |
| `audience` / `paywalled` | `only_paid` · `true` |
| `publishedAt` | ISO 8601 |
| `authors` | name, handle, bio, id |
| `wordCount` / `reactionCount` / `commentCount` / `restacks` | `1121` · `56` · `9` · `4` |
| `tags` / `language` / `postType` | `["ai"]` · `en` · `newsletter` |
| `podcastUrl` / `podcastDuration` / `hasAudio` | for audio posts |
| `teaser` | the public preview, always |
| `bodyHtml` / `bodyText` | free posts only |
| `bodyStatus` | `full` · `withheld_paywalled` · `not_requested` |

**Comments** — `body`, `authorName`, `authorHandle`, `depth`, `isReply`, `parentId`,
`replyCount`, `reactionCount`, `createdAt`, `editedAt`.

**Publications** — `name`, `subdomain`, `customDomain`, `url`, `tagline`, `logoUrl`,
`paymentsEnabled`, `communityEnabled`, `inviteOnly`, `createdAt`, `recommendedBy`.

***

### Modes

| Mode | What it returns |
|---|---|
| **Posts from a publication** | The archive, swept completely |
| **Search posts** | Substack's own search, inside a publication |
| **Discover publications** | Newsletters matching a topic — when you don't know the URLs |
| **Recommended publications** | Who a newsletter recommends |
| **Comments** | Whole threads, replies flattened with depth and parent |

***

### Typical uses

- **Competitive research.** Every post a rival newsletter published, how often, how it performed,
  and what sits behind the paywall.
- **Finding newsletters at all.** Discovery takes a topic and returns publications — most
  scrapers need you to already know the URL, which is useless if the question is "who writes
  about climate tech".
- **Newsletter growth and partnerships.** The recommendation graph shows who recommends whom,
  which is the main discovery mechanism on Substack.
- **Audience research.** Complete comment threads with reply structure, not just the top 7%.
- **Content and trend analysis.** Word counts, tags, cadence and engagement across a whole
  archive.
- **Feeding an LLM or agent.** One row per item, consistent field names, HTML stripped and
  entities decoded.

***

### Example input

A publication's best recent posts, with full text where it's free:

```json
{
  "mode": "posts",
  "publications": ["bigtechnology"],
  "includeBody": true,
  "publishedWithinDays": 90,
  "minReactions": 20,
  "sortBy": "reactions",
  "maxItems": 200
}
```

Find publications in a topic:

```json
{ "mode": "discover", "discoverQuery": "technology", "maxItems": 100 }
```

Walk the recommendation graph out from one newsletter:

```json
{ "mode": "recommendations", "publications": ["bigtechnology"], "maxItems": 200 }
```

***

### Settings that matter

- **`includeBody`** — off by default because it makes rows much larger. Note `bodyStatus`:
  paywalled posts come back `withheld_paywalled` with the public teaser, never the paid text.
  That's deliberate, not a gap.
- **`freeOnly` / `paidOnly`** — if you're building a text corpus, `freeOnly` is what you want,
  since paid posts return a teaser only and would just be empty rows you paid for.
- **`minReactions` / `minComments` / `minWords`** — applied before billing. `minWords` is the
  quickest way to drop one-line link posts from a writing analysis.
- **`publishedWithinDays`** — most archives are long. This is usually a bigger lever on cost
  than `maxItems`.
- **`mode: discover` vs `recommendations`** — discover searches by topic; recommendations walks
  the graph of who recommends whom, which surfaces newsletters that search doesn't rank.
- **`maxItems`** — your cost ceiling. Set it deliberately.

***

### Reliability

Everything comes from Substack's own public JSON API. There is no HTML parsing and no
bot-detection workaround anywhere in this Actor, which is why it doesn't quietly break when a
page layout changes.

An 82-assertion live test suite runs against the real API before every release — including
assertions that the paging gap still exists, that a paid body is never emitted even when the
API offers it, and that a flattened thread matches its advertised comment count.

Publications that have left Substack while keeping their domain return 404 on every API route.
This Actor says so plainly instead of returning an empty result you'd have to debug.

***

### Pricing

Pay per event. You are charged for the run start and for each row returned — nothing else, and
platform compute is included free. Full post bodies cost one extra request per post, so they're
opt-in, and they're only fetched for posts that can actually return one.

***

### Questions or a bug?

Open an issue on the **Issues** tab of this Actor. Include your input JSON and the run ID —
that's usually enough to reproduce it immediately.

# Actor input Schema

## `mode` (type: `string`):

Pick one. 'posts' = a publication's archive. 'search' = posts matching a query inside a publication. 'discover' = find publications by topic when you don't know their URLs. 'recommendations' = the publications a newsletter recommends. 'comments' = full comment threads, replies included.

## `publications` (type: `array`):

One or more Substack publications. Accepts whatever you have to hand: a bare name (bigtechnology), a host (bigtechnology.substack.com), a full URL, or a custom domain (www.astralcodexten.com). Not needed for 'discover' mode.

## `discoverQuery` (type: `string`):

Only used by 'discover' mode. The topic to find publications for, e.g. 'artificial intelligence', 'personal finance', 'climate'. This is how you find newsletters when you don't already know who writes about something.

## `searchQuery` (type: `string`):

In 'search' mode this is sent to Substack's own search inside the publication. In 'posts' mode it filters the returned rows on title, subtitle, teaser, tags and author.

## `includeBody` (type: `boolean`):

Fetch the full text of each post. Costs one extra request per post, so it is off by default. Only FREE posts return a body: paywalled posts return metadata and the public teaser, never the paid text. That is deliberate — see the Actor description.

## `sort` (type: `string`):

How Substack should order the archive before this Actor sweeps it. 'new' is newest-first, 'top' is by popularity.

## `sortBy` (type: `string`):

How to order the rows you get back. 'default' keeps the order Substack returned them in.

## `freeOnly` (type: `boolean`):

Keep only posts anyone can read. These are the ones that can return a full body.

## `paidOnly` (type: `boolean`):

Keep only subscriber-only posts. Useful for studying what a publication puts behind its paywall — you get titles, teasers and engagement, not the paid text.

## `author` (type: `string`):

Keep only posts with this name among their bylines. Partial, case-insensitive.

## `tag` (type: `string`):

Keep only posts carrying this Substack tag. Partial, case-insensitive.

## `postType` (type: `string`):

Keep only one kind of post: newsletter, podcast, thread or video.

## `withAudioOnly` (type: `boolean`):

Keep only posts with a podcast recording or a voiceover.

## `minReactions` (type: `integer`):

Keep only posts with at least this many likes.

## `minComments` (type: `integer`):

Keep only posts with at least this many comments.

## `minWords` (type: `integer`):

Keep only posts of at least this length. Useful for separating real essays from short notes.

## `publishedWithinDays` (type: `integer`):

Only posts from the last N days. 0 means the whole archive.

## `maxItems` (type: `integer`):

How many rows to return, and what you pay for. The archive sweep pages correctly, so large values really do return complete data rather than quietly skipping posts.

## Actor input object example

```json
{
  "mode": "posts",
  "publications": [
    "bigtechnology",
    "astralcodexten.substack.com"
  ],
  "discoverQuery": "artificial intelligence",
  "searchQuery": "openai",
  "includeBody": false,
  "sort": "new",
  "sortBy": "default",
  "freeOnly": false,
  "paidOnly": false,
  "postType": "",
  "withAudioOnly": false,
  "minReactions": 0,
  "minComments": 0,
  "minWords": 0,
  "publishedWithinDays": 0,
  "maxItems": 100
}
```

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("pixflor/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("pixflor/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call pixflor/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,pixflor/substack-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/0au8qhtfBEzZCdAQW/builds/m8yAbEwXq6tzdYxXu/openapi.json