# Substack Scraper - Posts, Comments, Search & Publication Intel (`apricot_blackberry/substack-all-in-one`) Actor

Scrape Substack newsletters via public JSON endpoints: post archives with full content, comment threads, global post search, and publication metadata. Fast, no login, no browser. Export JSON/CSV.

- **URL**: https://apify.com/apricot\_blackberry/substack-all-in-one.md
- **Developed by:** [Creator Fusion](https://apify.com/apricot_blackberry) (community)
- **Categories:** News, Social media, Marketing
- **Stats:** 3 total users, 2 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Substack Scraper — Posts, Comments, Search & Publication Intel for AI Agents & Data Pipelines

The all-in-one **Substack scraper** for AI agents and data pipelines: scrape Substack posts, full article content, comment threads, global search results, and publication metadata from any Substack newsletter — including custom-domain publications. Every run returns **typed JSON rows validated against a strict output schema** (documented below), read straight from Substack's public JSON endpoints: fast, stable, **no login, no cookies, no browser**. Export as JSON, CSV, or Excel, or call it as an MCP tool.

**Honest limitation:** paywalled post bodies return only the free preview — Substack does not expose subscriber-only content publicly, and this actor does not log in.

### Why AI agents use this Substack scraper

- **Deterministic typed output** — every row is validated against the dataset schema below; fields are stable across runs, safe to parse programmatically.
- **Cost-predictable per-event pricing** — tiny **$0.005 start fee** (competitors charge ~$0.09 per start) plus a small charge per row. Easy to budget for autonomous agents. Uses Apify datacenter proxy by default (included in Apify plans), so **no residential proxy costs**.
- **One actor, five jobs** — posts, full post content, comments, global search, and publication intel. No juggling four separate actor rentals.
- **No auth needed** — public JSON endpoints only; no credentials, cookies, or CAPTCHAs.
- **Clear error semantics** — bad input fails fast with a non-zero exit and a message; transient HTTP errors are retried with backoff; failures are never silent (see Error semantics).

### Input schema

| Field | Type | Required | Default | Description |
|---|---|---|---|---|
| `mode` | string (enum) | no | `"posts"` | What to scrape: `posts` (post metadata, optionally full content, from publication archives), `comments` (flattened comment threads of recent posts), `search` (global Substack post search), `publication` (publication metadata / intel). |
| `publicationUrls` | array of strings | for `posts`/`comments`/`publication` | — | Substack publication homepages, e.g. `https://astralcodexten.substack.com` or custom domains like `https://www.astralcodexten.com`. |
| `queries` | array of strings | for `search` | — | Search terms for the global Substack post search. |
| `includeContent` | boolean | no | `false` | Posts mode only: additionally fetch each post's full body (HTML + plain text). Slower; charged per `post-content` event. Paywalled posts return only the free preview. |
| `maxItems` | integer | no | `50` | Maximum total dataset rows per run, across all publications/queries. |
| `proxyConfiguration` | object | no | Apify datacenter proxy | Proxy for requests. Default datacenter proxy is included in Apify plans (no residential costs). |

### Output schema

Field presence by mode (all fields are nullable unless noted; the dataset schema enforces types on every row):

| Field | Type | Modes | Description |
|---|---|---|---|
| `id` | integer | posts, search, publication | Post ID (posts/search) or publication ID (publication). |
| `title` | string | posts, search | Post title. |
| `subtitle` | string | posts, search | Post subtitle. |
| `slug` | string | posts, search | Post URL slug. |
| `postDate` | string (ISO 8601) | posts, search | Publish date. |
| `type` | string | posts, search | Post type (`newsletter`, `podcast`, `thread`, ...). |
| `likes` | integer | posts, search, comments | Reaction count (post or comment). Defaults to 0. |
| `commentCount` | integer | posts, search | Comment count on the post. |
| `audience` | string | posts, search | `everyone` for free posts; other values mean paywalled. |
| `paywalled` | boolean | posts, search | True when the full body is subscriber-only. |
| `wordCount` | integer | posts, search | Post word count. |
| `description` | string | posts, search, publication | Post preview description, or publication hero text. |
| `canonicalUrl` | string | posts, search | Canonical post URL. |
| `coverImage` | string | posts, search | Post cover image URL. |
| `publicationId` | integer | posts, search | Publication ID the post belongs to. |
| `publicationUrl` | string | all (null in search) | Normalized publication base URL scraped from. |
| `authorName` | string | all | Author display name (byline / commenter / publication author). |
| `authorHandle` | string | all | Author Substack handle. |
| `authorId` | integer | posts, search, comments | Author user ID. |
| `bodyHtml` | string | posts (`includeContent: true`) | Full post body HTML (free preview only for paywalled posts). |
| `bodyText` | string | posts (`includeContent: true`) | Full post body as plain text. |
| `query` | string | search | The search term that produced this row. |
| `postId` | integer | comments | ID of the post the comment belongs to. |
| `postTitle` | string | comments | Title of that post. |
| `postSlug` | string | comments | Slug of that post. |
| `postUrl` | string | comments | Canonical URL of that post. |
| `commentId` | integer | comments | Unique comment ID. |
| `parentCommentId` | integer | comments | Parent comment ID; null for top-level comments. |
| `body` | string | comments | Comment text (null if deleted). |
| `bodyDeleted` | boolean | comments | True when the comment was deleted. |
| `date` | string (ISO 8601) | comments | Comment date. |
| `childCount` | integer | comments | Number of direct replies (replies are also emitted as rows). |
| `name` | string | publication | Publication display name. |
| `subdomain` | string | publication | `*.substack.com` subdomain. |
| `customDomain` | string | publication | Custom domain, if any. |
| `hostname` | string | publication | Serving hostname. |
| `authorBio` | string | publication | Author bio. |
| `logoUrl` | string | publication | Logo image URL. |
| `coverPhotoUrl` | string | publication | Cover photo URL. |
| `language` | string | publication | Language code. |
| `createdAt` | string (ISO 8601) | publication | Publication creation date. |
| `firstPostDate` | string (ISO 8601) | publication | Date of first post. |
| `freeSubscriberCount` | string/integer | publication | Free subscriber count hint (often rounded). |
| `subscriberCountOrderOfMagnitude` | number | publication | Order-of-magnitude subscriber hint. |
| `rankingDetail` | string/number | publication | Substack leaderboard/ranking detail. |
| `paymentsEnabled` | string/boolean | publication | Payments state (paid subscriptions enabled). |
| `podcastEnabled` | boolean | publication | True when the publication has a podcast. |
| `twitter` | string | publication | Twitter/X screen name. |
| `copyright` | string | publication | Copyright line. |

#### Example item — posts mode

```json
{
    "id": 210672789,
    "title": "Your Book Review: The Escape Artist",
    "subtitle": "Finalist #5 in the Book Review Contest",
    "slug": "your-book-review-the-escape-artist",
    "postDate": "2026-08-14T22:21:13.104Z",
    "type": "newsletter",
    "likes": 61,
    "commentCount": 14,
    "audience": "everyone",
    "paywalled": false,
    "wordCount": 6807,
    "description": "Finalist #5 in the Book Review Contest",
    "canonicalUrl": "https://www.astralcodexten.com/p/your-book-review-the-escape-artist",
    "coverImage": "https://substackcdn.com/image/fetch/w_1200/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fexample.png",
    "publicationId": 89120,
    "publicationUrl": "https://astralcodexten.substack.com",
    "authorName": "Scott Alexander",
    "authorHandle": "astralcodexten",
    "authorId": 12345678
}
```

With `includeContent: true` each row additionally contains `bodyHtml` and `bodyText`.

#### Example item — comments mode

```json
{
    "postId": 210672789,
    "postTitle": "Your Book Review: The Escape Artist",
    "postSlug": "your-book-review-the-escape-artist",
    "postUrl": "https://www.astralcodexten.com/p/your-book-review-the-escape-artist",
    "publicationUrl": "https://astralcodexten.substack.com",
    "commentId": 98765432,
    "parentCommentId": null,
    "body": "Great review — the framing of the escape as a moral act is what stuck with me.",
    "bodyDeleted": false,
    "authorName": "Jane Reader",
    "authorHandle": "janereader",
    "authorId": 2345678,
    "date": "2026-08-15T03:12:44.000Z",
    "likes": 12,
    "childCount": 3
}
```

#### Example item — search mode

```json
{
    "query": "AI agents",
    "id": 198765432,
    "title": "The Agentic Web Is Coming",
    "subtitle": "What happens when most web traffic is bots with budgets",
    "slug": "the-agentic-web-is-coming",
    "postDate": "2026-07-02T15:00:00.000Z",
    "type": "newsletter",
    "likes": 214,
    "commentCount": 45,
    "audience": "everyone",
    "paywalled": false,
    "wordCount": 2140,
    "description": "What happens when most web traffic is bots with budgets",
    "canonicalUrl": "https://www.example-newsletter.com/p/the-agentic-web-is-coming",
    "coverImage": null,
    "publicationId": 55501,
    "publicationUrl": null,
    "authorName": "Alex Writer",
    "authorHandle": "alexwriter",
    "authorId": 7654321
}
```

#### Example item — publication mode

```json
{
    "publicationUrl": "https://astralcodexten.substack.com",
    "id": 89120,
    "name": "Astral Codex Ten",
    "subdomain": "astralcodexten",
    "customDomain": "www.astralcodexten.com",
    "hostname": "www.astralcodexten.com",
    "authorName": "Scott Alexander",
    "authorHandle": "astralcodexten",
    "authorBio": "P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary.",
    "description": "P(A|B) = [P(A)*P(B|A)]/P(B), all the rest is commentary.",
    "logoUrl": "https://substackcdn.com/image/fetch/w_256/https%3A%2F%2Fexample-logo.png",
    "coverPhotoUrl": null,
    "language": "en",
    "createdAt": "2021-01-21T00:00:00.000Z",
    "firstPostDate": "2021-01-21T08:00:00.000Z",
    "freeSubscriberCount": "100000",
    "subscriberCountOrderOfMagnitude": 100000,
    "rankingDetail": "#2 in Science",
    "paymentsEnabled": "enabled",
    "podcastEnabled": false,
    "twitter": "slatestarcodex",
    "copyright": "Scott Alexander"
}
```

### Error semantics

- **Bad input fails fast**: unknown `mode`, `search` without `queries`, or other modes without `publicationUrls` throw immediately — the run exits non-zero with a clear status message. Nothing is charged beyond the start fee.
- **Transient HTTP errors** (429, 5xx, network) are retried 3 times with increasing backoff (30 s request timeout). If all retries fail, the run fails with a non-zero exit — never a silently truncated dataset.
- **Non-retryable 4xx** on a publication endpoint (e.g. a wrong URL returning 404) yields zero rows for that source; other sources in the same run still proceed.
- **Zero results** (empty archive, no search hits) is a successful run with an empty dataset — check the item count, not the exit code, to distinguish "nothing found" from "failed".
- **Publication mode is best-effort**: if homepage parsing fails, the actor falls back to archive data and still pushes a row (with nulls for unavailable fields); it never throws mid-run.

### Use from AI agents (MCP)

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com/?tools=apricot_blackberry/substack-all-in-one",
      "headers": { "Authorization": "Bearer <YOUR_APIFY_TOKEN>" }
    }
  }
}
```

Works in Claude, Cursor, ChatGPT deep research connectors, and any MCP client; the input schema above is the tool's parameter schema.

### Use from code

curl (synchronous run, returns dataset items directly):

```bash
curl -X POST "https://api.apify.com/v2/acts/apricot_blackberry~substack-all-in-one/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"mode":"posts","publicationUrls":["https://astralcodexten.substack.com"],"maxItems":50}'
```

JavaScript (`apify-client`):

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('apricot_blackberry/substack-all-in-one').call({
    mode: 'posts',
    publicationUrls: ['https://astralcodexten.substack.com'],
    maxItems: 50,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

Python (`apify-client`):

```python
from apify_client import ApifyClient

client = ApifyClient(token=os.environ["APIFY_TOKEN"])
run = client.actor("apricot_blackberry/substack-all-in-one").call(run_input={
    "mode": "posts",
    "publicationUrls": ["https://astralcodexten.substack.com"],
    "maxItems": 50,
})
items = client.dataset(run["defaultDatasetId"]).list_items().items
print(items)
```

### Use from automation platforms

- **n8n / Make / Zapier** — use the native Apify integration and pick this actor by name (`apricot_blackberry/substack-all-in-one`).
- **LangChain / LlamaIndex** — use the Apify actor tool wrappers (`ApifyActorsTool` / `ApifyActor` reader) with the same input JSON.
- **Webhooks** — Apify webhooks can fire on run completion (e.g. `ACTOR.RUN.SUCCEEDED`) to push new posts/comments into your own pipeline automatically, no polling required.

### Pricing

Pay per event — no monthly rental, just a tiny $0.005 start fee (vs ~$0.09 typical for competitors):

| Event | Charged for |
|---|---|
| `actor-start` | Once per run |
| `post` | Each post metadata row |
| `post-content` | Each post row including the full body |
| `comment` | Each comment row |
| `search-result` | Each search result row |
| `publication` | Each publication intel row |

Runs use the Apify datacenter proxy by default (included in Apify subscription plans), so there are **no residential proxy surcharges**.

### FAQ

**Can I scrape Substack posts without the API or login?**
Yes — this actor reads Substack's public JSON endpoints (the same ones the website uses), so no credentials, cookies, or browser automation are needed.

**Does it work with newsletters on custom domains?**
Yes. Pass the custom domain (e.g. `https://www.lennysnewsletter.com`) or the original `*.substack.com` URL — redirects are followed automatically.

**Can I get the full text of paywalled posts?**
No. Paywalled posts return only the free preview. Free posts return the complete article body.

**How do I export Substack data to CSV or Excel?**
Every run stores results in an Apify dataset; download as JSON, CSV, Excel, or XML from the console or via API — ideal for newsletter research and content analysis.

**Can my AI agent use this?**
Yes — see the MCP section above. The actor is a first-class MCP tool with a typed input and output schema.

### Changelog

- **1.1** — Added strict dataset output schema (typed, validated rows) and agent-first documentation: MCP config, code snippets, error semantics.
- **1.0** — Initial release: posts, comments, search, publication modes; full-content option; pay-per-event pricing.

# Actor input Schema

## `mode` (type: `string`):

What to scrape. <b>posts</b>: post metadata (optionally full content) from publication archives. <b>comments</b>: flattened comment threads of recent posts. <b>search</b>: global Substack post search. <b>publication</b>: publication metadata / intel.

## `publicationUrls` (type: `array`):

Substack publication homepages, e.g. <code>https://stratechery.substack.com</code> or custom domains like <code>https://www.astralcodexten.com</code>. Used by posts, comments and publication modes.

## `queries` (type: `array`):

Search terms for the global Substack post search. Used by search mode only.

## `includeContent` (type: `boolean`):

Posts mode only: additionally fetch each post's full body (HTML + plain text). Slower and charged per post-content event. Paywalled posts return only the free preview.

## `maxItems` (type: `integer`):

Maximum total number of dataset rows pushed per run, counted across all publication URLs or queries combined. The run stops as soon as this limit is reached.

## `proxyConfiguration` (type: `object`):

Proxy for outgoing requests. Defaults to Apify datacenter proxy (included in Apify plans — no residential proxy costs). Substack's public API is lenient; the default is sufficient for most runs.

## Actor input object example

```json
{
  "mode": "posts",
  "publicationUrls": [
    "https://astralcodexten.substack.com"
  ],
  "queries": [
    "AI agents"
  ],
  "includeContent": false,
  "maxItems": 50,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Typed JSON rows in the default dataset - posts, comments, search results, and publication intel.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publicationUrls": [
        "https://astralcodexten.substack.com"
    ],
    "queries": [
        "AI agents"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("apricot_blackberry/substack-all-in-one").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publicationUrls": ["https://astralcodexten.substack.com"],
    "queries": ["AI agents"],
}

# Run the Actor and wait for it to finish
run = client.actor("apricot_blackberry/substack-all-in-one").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publicationUrls": [
    "https://astralcodexten.substack.com"
  ],
  "queries": [
    "AI agents"
  ]
}' |
apify call apricot_blackberry/substack-all-in-one --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,apricot_blackberry/substack-all-in-one"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/aDproBrGcqNe0BfO7/builds/jMtEg5lA42SIqSZd2/openapi.json