# Substack Newsletter Scraper (`devilscrapes/substack-newsletter-scraper`) Actor

Scrape posts from any Substack publication — title, subtitle, date, paywall status, reaction count, comment count, word count, and full body HTML for free posts. Handles custom domains. Paginates to the full archive.

- **URL**: https://apify.com/devilscrapes/substack-newsletter-scraper.md
- **Developed by:** [DevilScrapes](https://apify.com/devilscrapes) (community)
- **Categories:** News, Social media
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, NaN bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-usage

## What's an Apify Actor?

Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md):

```bash
npm install apify-client
```

In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md):

```bash
pip install apify-client
```

In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md):

````bash
# MacOS / Linux
curl -fsSL https://apify.com/install-cli.sh | bash
# Windows
irm https://apify.com/install-cli.ps1 | iex
```bash

In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md).

If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).


# README

<div align="center">
  <img src=".actor/icon.svg" width="160" alt="Devil Scrapes mark" />

## Substack Newsletter Scraper

**💰 $2.50 / 1 000 results** &nbsp;·&nbsp; pay only for results &nbsp;·&nbsp; no credit card to try

_We do the dirty work so your dataset stays clean._ 😈

Scrape posts from any Substack publication — title, subtitle, date, paywall status, reaction count, comment count, word count, and full body HTML for free posts. Handles custom domains. Paginates to the full archive.

</div>

---

### 🎯 What this scrapes

Substack hosts thousands of newsletters across every niche — from tech and finance to culture and investigative journalism. This Actor taps each publication's public JSON API to extract the full post archive: metadata for every post, plus body HTML for free-tier content. Works with native Substack domains (`author.substack.com`) and custom domains alike.

### 🔥 What we handle for you

- 🛡️ **Browser fingerprint rotation** — `curl-cffi` impersonates real Chrome / Firefox / Safari TLS handshakes so the target sees a browser, not Python.
- 🌐 **Residential proxy rotation** via Apify Proxy — fresh session and exit IP on every block.
- 🔁 **Retries with exponential backoff** on `408 / 429 / 5xx` — up to 5 attempts per page, `Retry-After` honoured.
- 🧱 **Rate-limit-aware pacing** — when the target pushes back, we slow down instead of getting banned.
- 🧊 **Clean, typed dataset rows** — Pydantic-validated, ISO-8601 timestamps, stable IDs, JSON / CSV / Excel export straight from the Apify Console.
- 💰 **Pay-Per-Event pricing** — you only pay for results that hit your dataset. No data, no charge.

### 💡 Use cases

- **Newsletter intelligence** — track a competitor publication's cadence, topics, and engagement over time.
- **Content research** — aggregate posts across multiple newsletters on a niche to find trends and gaps.
- **Lead gen** — identify high-engagement free posts to understand what resonates with a target audience.
- **Archive backfill** — pull the full post history into a warehouse for analysis or ML training.
- **Paywall mapping** — understand the free-vs-paid content split for any publication.

### ⚙️ How to use it

1. Click **Try for free** at the top of the page.
2. Fill in the input form — most fields have sensible defaults.
3. Click **Start**. Output streams into the run's dataset.
4. Export from **Storage → Dataset** as JSON, CSV, or Excel — or fetch via the API.

### 📥 Input

| Field | Type | Required | Default | Notes |
|---|---|:--:|---|---|
| `publications` | `array` | **yes** | ['https://www.lennysnewsletter.com'] | One or more Substack publication URLs or custom domains. Accepts both native substack.com URLs and custom domains (e.g.  |
| `maxPostsPerPublication` | `integer` | no | 50 | Maximum number of posts to collect per publication. Set to 0 to collect the full archive. |
| `includeBody` | `boolean` | no | True | When enabled, body_html is included for free posts. Paywalled posts return null body. |
| `proxyConfiguration` | `object` | no | {'useApifyProxy': True} | Apify Proxy configuration. Substack's public API is well-behaved; the default shared pool is sufficient for most volumes |

#### Example input

```json
{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 5,
  "includeBody": false,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
````

### 📤 Output

Every row is one dataset item.

| Field | Type | Notes |
|---|---|---|
| `publication` | `string` | Base URL of the publication that produced this post. |
| `post_id` | `integer` | Substack internal numeric post ID. |
| `title` | `string` | Post title. |
| `subtitle` | `['string', 'null']` | Post subtitle / deck (may be null). |
| `url` | `string` | Canonical URL of the post. |
| `post_date` | `string` | Publication timestamp (ISO-8601). |
| `audience` | `string` | Access tier: `free` or `only_paid`. |
| `is_paywalled` | `boolean` | True when the post is behind the paywall. |
| `reaction_count` | `integer` | Total like/heart reactions. |
| `comment_count` | `integer` | Number of public comments. |
| `word_count` | `['integer', 'null']` | Word count when provided by the API. |
| `body_html` | `['string', 'null']` | Full post HTML for free posts; null for paywalled posts or when includeBody is false. |
| `scraped_at` | `string` | ISO-8601 timestamp of when this row was recorded. |

#### Example output

```json
{
  "publication": "https://www.lennysnewsletter.com",
  "post_id": 123456,
  "title": "The most important metric you're ignoring",
  "subtitle": "How to think about retention in your product",
  "url": "https://www.lennysnewsletter.com/p/the-most-important-metric",
  "post_date": "2024-03-15T09:00:00.000Z",
  "audience": "free",
  "is_paywalled": false,
  "reaction_count": 847,
  "comment_count": 43,
  "word_count": 2100,
  "body_html": null,
  "scraped_at": "2026-06-07T10:00:00Z"
}
```

### 💰 Pricing

Pay-Per-Event — you pay only when these events fire:

| Event | USD | What it is |
|---|---:|---|
| `actor-start` | $0.005 | One-off warm-up charge per run |
| `result` | $0.0025 | Per dataset item |

Example: 1 000 results at the rates above ≈ **$2.50**. No subscription, no minimum, no card to start — Apify gives every new account $5 of free credit.

### 🚧 Limitations

Body HTML is only available for posts the publication makes free. Paywalled posts return metadata only. Private Substack communities (invite-only) are not accessible. The Actor honours Substack's public API rate limits — very large archives may take several minutes.

### ❓ FAQ

**Does this work with custom domains?**

Yes. Pass the full base URL of the publication — whether it's `author.substack.com` or a custom domain like `https://www.lennysnewsletter.com` — and the Actor resolves the correct API endpoint.

**Can I get paywalled post bodies?**

No. Paywalled posts require a paid subscription; `body_html` returns null for those. You get all public metadata (title, date, reaction count, etc.) regardless of paywall status.

**How far back does the archive go?**

As far as the publication's API returns. We paginate until we hit `maxPostsPerPublication` or exhaust the archive, whichever comes first.

**What if a publication has thousands of posts?**

Set `maxPostsPerPublication` to `0` and the Actor will paginate to the full archive. Runs are priced per result, so longer archives cost proportionally more.

### 💬 Your feedback

Spotted a bug, hit a weird edge case, or need a new field? Open an
issue on the Actor's **Issues** tab on Apify Console — we ship
fixes weekly and we read every report.

***

<div align="center">

Built by **[Devil Scrapes](https://apify.com/DevilScrapes)** 😈 — a small fleet of
opinionated public-data Actors. Honest pricing, real engineering, zero fine print.

</div>

# Actor input Schema

## `publications` (type: `array`):

One or more Substack publication URLs or custom domains. Accepts both native substack.com URLs and custom domains (e.g. https://www.lennysnewsletter.com).

## `maxPostsPerPublication` (type: `integer`):

Maximum number of posts to collect per publication. Set to 0 to collect the full archive.

## `includeBody` (type: `boolean`):

When enabled, body\_html is included for free posts. Paywalled posts return null body.

## `proxyConfiguration` (type: `object`):

Apify Proxy configuration. Substack's public API is well-behaved; the default shared pool is sufficient for most volumes.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 50,
  "includeBody": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `datasetItems` (type: `string`):

All dataset items as JSON.

## `datasetItemsCsv` (type: `string`):

Same data exported to CSV.

## `datasetView` (type: `string`):

Open the run dataset in the Console.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com"
    ],
    "maxPostsPerPublication": 50,
    "includeBody": true,
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("devilscrapes/substack-newsletter-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["https://www.lennysnewsletter.com"],
    "maxPostsPerPublication": 50,
    "includeBody": True,
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("devilscrapes/substack-newsletter-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 50,
  "includeBody": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call devilscrapes/substack-newsletter-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=devilscrapes/substack-newsletter-scraper",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

```json
{
    "openapi": "3.0.1",
    "info": {
        "title": "Substack Newsletter Scraper",
        "description": "Scrape posts from any Substack publication — title, subtitle, date, paywall status, reaction count, comment count, word count, and full body HTML for free posts. Handles custom domains. Paginates to the full archive.",
        "version": "0.1",
        "x-build-id": "zXN6oPbZS2iwVOpAb"
    },
    "servers": [
        {
            "url": "https://api.apify.com/v2"
        }
    ],
    "paths": {
        "/acts/devilscrapes~substack-newsletter-scraper/run-sync-get-dataset-items": {
            "post": {
                "operationId": "run-sync-get-dataset-items-devilscrapes-substack-newsletter-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        },
        "/acts/devilscrapes~substack-newsletter-scraper/runs": {
            "post": {
                "operationId": "runs-sync-devilscrapes-substack-newsletter-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor and returns information about the initiated run in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK",
                        "content": {
                            "application/json": {
                                "schema": {
                                    "$ref": "#/components/schemas/runsResponseSchema"
                                }
                            }
                        }
                    }
                }
            }
        },
        "/acts/devilscrapes~substack-newsletter-scraper/run-sync": {
            "post": {
                "operationId": "run-sync-devilscrapes-substack-newsletter-scraper",
                "x-openai-isConsequential": false,
                "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.",
                "tags": [
                    "Run Actor"
                ],
                "requestBody": {
                    "required": true,
                    "content": {
                        "application/json": {
                            "schema": {
                                "$ref": "#/components/schemas/inputSchema"
                            }
                        }
                    }
                },
                "parameters": [
                    {
                        "name": "token",
                        "in": "query",
                        "required": true,
                        "schema": {
                            "type": "string"
                        },
                        "description": "Enter your Apify token here"
                    }
                ],
                "responses": {
                    "200": {
                        "description": "OK"
                    }
                }
            }
        }
    },
    "components": {
        "schemas": {
            "inputSchema": {
                "type": "object",
                "required": [
                    "publications"
                ],
                "properties": {
                    "publications": {
                        "title": "Publications",
                        "type": "array",
                        "description": "One or more Substack publication URLs or custom domains. Accepts both native substack.com URLs and custom domains (e.g. https://www.lennysnewsletter.com).",
                        "default": [
                            "https://www.lennysnewsletter.com"
                        ],
                        "items": {
                            "type": "string"
                        }
                    },
                    "maxPostsPerPublication": {
                        "title": "Max posts per publication",
                        "minimum": 0,
                        "maximum": 10000,
                        "type": "integer",
                        "description": "Maximum number of posts to collect per publication. Set to 0 to collect the full archive.",
                        "default": 50
                    },
                    "includeBody": {
                        "title": "Include post body HTML",
                        "type": "boolean",
                        "description": "When enabled, body_html is included for free posts. Paywalled posts return null body.",
                        "default": true
                    },
                    "proxyConfiguration": {
                        "title": "Proxy configuration",
                        "type": "object",
                        "description": "Apify Proxy configuration. Substack's public API is well-behaved; the default shared pool is sufficient for most volumes.",
                        "default": {
                            "useApifyProxy": true
                        }
                    }
                }
            },
            "runsResponseSchema": {
                "type": "object",
                "properties": {
                    "data": {
                        "type": "object",
                        "properties": {
                            "id": {
                                "type": "string"
                            },
                            "actId": {
                                "type": "string"
                            },
                            "userId": {
                                "type": "string"
                            },
                            "startedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "finishedAt": {
                                "type": "string",
                                "format": "date-time",
                                "example": "2025-01-08T00:00:00.000Z"
                            },
                            "status": {
                                "type": "string",
                                "example": "READY"
                            },
                            "meta": {
                                "type": "object",
                                "properties": {
                                    "origin": {
                                        "type": "string",
                                        "example": "API"
                                    },
                                    "userAgent": {
                                        "type": "string"
                                    }
                                }
                            },
                            "stats": {
                                "type": "object",
                                "properties": {
                                    "inputBodyLen": {
                                        "type": "integer",
                                        "example": 2000
                                    },
                                    "rebootCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "restartCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "resurrectCount": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "computeUnits": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "options": {
                                "type": "object",
                                "properties": {
                                    "build": {
                                        "type": "string",
                                        "example": "latest"
                                    },
                                    "timeoutSecs": {
                                        "type": "integer",
                                        "example": 300
                                    },
                                    "memoryMbytes": {
                                        "type": "integer",
                                        "example": 1024
                                    },
                                    "diskMbytes": {
                                        "type": "integer",
                                        "example": 2048
                                    }
                                }
                            },
                            "buildId": {
                                "type": "string"
                            },
                            "defaultKeyValueStoreId": {
                                "type": "string"
                            },
                            "defaultDatasetId": {
                                "type": "string"
                            },
                            "defaultRequestQueueId": {
                                "type": "string"
                            },
                            "buildNumber": {
                                "type": "string",
                                "example": "1.0.0"
                            },
                            "containerUrl": {
                                "type": "string"
                            },
                            "usage": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "integer",
                                        "example": 1
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            },
                            "usageTotalUsd": {
                                "type": "number",
                                "example": 0.00005
                            },
                            "usageUsd": {
                                "type": "object",
                                "properties": {
                                    "ACTOR_COMPUTE_UNITS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATASET_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "KEY_VALUE_STORE_WRITES": {
                                        "type": "number",
                                        "example": 0.00005
                                    },
                                    "KEY_VALUE_STORE_LISTS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_READS": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "REQUEST_QUEUE_WRITES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_INTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "DATA_TRANSFER_EXTERNAL_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_RESIDENTIAL_TRANSFER_GBYTES": {
                                        "type": "integer",
                                        "example": 0
                                    },
                                    "PROXY_SERPS": {
                                        "type": "integer",
                                        "example": 0
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }
}
```