# Substack Scraper - Newsletter Posts, Authors & Engagement (`deanko/substack-scraper`) Actor

Scrape Substack newsletters: posts, authors, likes, comments, word count and full text of free posts from any publication or search. Public data only. $2 per 1,000 posts.

- **URL**: https://apify.com/deanko/substack-scraper.md
- **Developed by:** [dean ko](https://apify.com/deanko) (community)
- **Categories:** Social media, News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper: newsletter posts, authors and engagement

Export posts from any **Substack** newsletter, on a `*.substack.com` address or a custom domain, as clean, flat rows for Excel, Google Sheets, BI tools or your database. Every row includes the **title, subtitle, URL, publish date, free/paid audience, author names, likes, comments, restacks, word count and cover image**. You can also add the **full plain text** of free posts. You can also **search all of Substack** by keyword.

It uses Substack's own public JSON endpoints, so it needs no login, no cookies, no proxies and no browser. It costs **$2 per 1,000 posts**.

### What does the Substack Scraper do?

It turns a list of Substack publications (or a search query) into a dataset with **one row per post**:

- publication name, URL, subdomain, main author and public subscriber badge (e.g. "Over 1,200,000 subscribers")
- post title, subtitle, URL, publish date and post type (newsletter, podcast, thread...)
- audience (`free` / `paid`) and an `isPaid` flag
- author names (all bylines, including guest authors)
- like count (reactions), comment count and restack count
- word count and cover image
- the public preview text, and the full body as plain text (HTML stripped) when you turn on `includeBody`

**Public data only.** Paid posts are never unlocked. For them you get the metadata and the public preview, exactly what a logged-out visitor sees.

#### Who is it for?

- **Newsletter creators and growth teams.** Benchmark competitors: posting cadence, which topics get the most likes and comments, how long their posts are, and how much they put behind the paywall.
- **Marketers and PR teams.** Find the newsletters that write about your category, see who the authors and guest writers are, and build outreach and sponsorship lists.
- **Investors and analysts.** Track what influential writers in finance, tech or politics publish, and feed it to dashboards or alerts.
- **Researchers, journalists and AI/NLP teams.** Build text datasets from free posts for topic modeling, sentiment analysis or RAG, with engagement numbers attached.
- **Monitoring.** Schedule a daily run to catch every new post from a set of newsletters.

### Why use this Substack scraper?

- **Low price.** $2 per 1,000 posts, less than half the price of the leading alternative. You pay only for rows saved, and you can set a maximum charge per run.
- **Reliable.** It reads Substack's JSON API instead of parsing HTML, retries automatically on rate limits (HTTP 429) and server errors, and one broken publication never stops the run.
- **Works with custom domains.** `lennysnewsletter.com`, `astralcodexten.com` and others work just like `*.substack.com` addresses. Subdomain redirects are followed for you.
- **Flat, analysis-ready rows.** Authors come as a simple list, engagement as plain numbers, and body text as clean plain text. The Overview table opens straight in the Console.
- **Polite.** Requests are sequential with a short pause between them, and it only uses paths that Substack's `robots.txt` allows.
- **Fast.** The default 2-publication run finishes in seconds.

### How to scrape Substack

1. Click **Try for free** (or **Start**) on this page.
2. Add publications: a subdomain (`astralcodexten`), a URL (`https://lenny.substack.com`) or a custom domain (`https://www.lennysnewsletter.com`). Any post URL also works.
3. Set **Max posts per publication** (newest posts first).
4. Optionally turn on **Include full text**, or enter a **Search query** to search all of Substack.
5. Click **Start**, then download the results as **CSV, Excel, JSON, XML or HTML**, or read them via the API.

#### Input examples

The 50 latest posts from two newsletters, with full text:

```json
{
  "publications": ["https://www.lennysnewsletter.com", "astralcodexten"],
  "maxPostsPerPublication": 50,
  "includeBody": true
}
```

Search all of Substack for a topic:

```json
{
  "searchQuery": "AI agents",
  "maxPostsPerPublication": 100
}
```

| Field | What it does | Example |
|---|---|---|
| `publications` | Substack subdomains, URLs, custom domains or post URLs | `astralcodexten`, `https://www.lennysnewsletter.com` |
| `maxPostsPerPublication` | Max posts per publication, newest first (default 20). Also caps search results | `100` |
| `includeBody` | Add plain-text body: full text for free posts, preview only for paid posts (default `false`) | `true` |
| `searchQuery` | Optional keyword search across all of Substack (posts) | `product management` |

### Output sample

```json
{
  "publication": "Astral Codex Ten",
  "publicationUrl": "https://www.astralcodexten.com",
  "publicationSubdomain": "astralcodexten",
  "publicationAuthor": "Scott Alexander",
  "publicationSubscribers": null,
  "postId": 221215977,
  "title": "Why Global Trepanation?",
  "subtitle": "...",
  "url": "https://www.astralcodexten.com/p/why-global-trepanation",
  "postDate": "2026-10-21T21:53:11.440Z",
  "audience": "free",
  "isPaid": false,
  "postType": "newsletter",
  "authors": ["Scott Alexander"],
  "likeCount": 337,
  "commentCount": 192,
  "restackCount": 10,
  "wordCount": 1307,
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/73a7ca5d-..._2340x1316.jpeg",
  "previewText": "...",
  "bodyText": "A friend asked me to investigate the following mystery: why do 5-10% of skulls from 10,000 BC to 3,000 BC...",
  "scrapedAt": "2026-10-23T04:40:12.118Z"
}
```

| Field | Type | Description |
|---|---|---|
| `publication`, `publicationUrl`, `publicationSubdomain` | string | Newsletter name, home URL and substack.com subdomain |
| `publicationAuthor` | string | Main author of the publication |
| `publicationSubscribers` | string | Substack's public subscriber badge, when the publication shows one |
| `postId` | number | Substack post ID |
| `title`, `subtitle` | string | Post title and subtitle |
| `url` | string | Canonical post URL |
| `postDate` | string | Publish date (ISO 8601) |
| `audience`, `isPaid` | string, boolean | `free` or `paid` (paid subscribers / founding members only) |
| `postType` | string | `newsletter`, `podcast`, `thread`... |
| `authors` | array | Byline names |
| `likeCount`, `commentCount`, `restackCount` | number | Engagement counts |
| `wordCount` | number | Length of the full post (also for paid posts) |
| `coverImage` | string | Cover image URL |
| `previewText` | string | Public preview / description |
| `bodyText` | string | Plain-text body when `includeBody` is on: full for free posts, preview for paid posts |
| `scrapedAt` | string | When the row was collected (ISO 8601) |

### How much does it cost to scrape Substack?

This Actor uses **pay-per-event** pricing:

- **$0.002 per post saved** ($2 per 1,000 posts)
- **$0.00005 per run start** (per GB of memory, a fraction of a cent; the default 256 MB run is charged about $0.0000125)

**Worked example:** the latest 200 posts from each of 10 newsletters is 2,000 posts × $0.002 = **$4.00**, plus well under a cent for the run start. The default example (2 newsletters × 10 posts) costs about **$0.04**, so the $5 of monthly free credit on the Apify Free plan covers about 2,500 posts.

Set **Max posts per publication** or a maximum charge per run, and the Actor stops as soon as it reaches the limit.

### Integrations and API

Call the Actor from your own code with the official `apify-client`, or connect it to Make, Zapier, n8n, Google Sheets, Slack, webhooks and more from the **Integrations** tab.

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("deanko/substack-scraper").call(run_input={
    "publications": ["https://www.lennysnewsletter.com", "platformer"],
    "maxPostsPerPublication": 50,
})
for post in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(post["publication"], post["title"], post["likeCount"], post["commentCount"])
```

**JavaScript / Node.js**

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_APIFY_TOKEN>' });
const run = await client.actor('deanko/substack-scraper').call({
    searchQuery: 'climate tech',
    maxPostsPerPublication: 100,
    includeBody: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.length, 'posts');
```

You can also run it with one HTTP request through the `run-sync-get-dataset-items` endpoint and get the rows back in the response.

### FAQ

**Do I need a Substack account or login?**
No. The Actor reads only public data, the same JSON that Substack's own website loads for a logged-out visitor.

**Can it get the full text of paid posts?**
No, by design. For paid posts you get the title, metadata, engagement numbers, word count and the public preview, and `isPaid` is `true`. Paywalled content is never accessed.

**Does it work with custom domains?**
Yes. Enter the custom domain (e.g. `https://www.lennysnewsletter.com`) or the `*.substack.com` address, and redirects are followed automatically.

**What does "likes" mean?**
`likeCount` is Substack's reaction count (the heart button) for the post.

**How does search work?**
`searchQuery` uses Substack's public site search and returns matching posts from any publication, in Substack's relevance order. Notes, comments and profiles are skipped.

**How do I get only new posts every day?**
Schedule the Actor (Console → Schedules) with a small `maxPostsPerPublication`, such as 5. Posts come newest first, so you can de-duplicate on `postId` in your sheet or database.

**Is scraping Substack allowed?**
The Actor only uses paths that Substack's `robots.txt` allows for all crawlers, sends requests one at a time with pauses, and collects only publicly available data. You are responsible for how you use the data. Respect authors' copyright, and check the laws that apply to you, especially for personal data.

### Limitations

- No paywalled content. Paid posts return only the public preview.
- Comments themselves (text, commenters) and Substack Notes are not included, only the comment count.
- Search returns posts only and follows Substack's own ranking. Very broad queries return the top matches, not every post.
- Subscriber numbers are Substack's public badge text, not exact counts, and many publications don't show one.
- Engagement numbers are a snapshot at the time of the run.

### Legal and privacy

This Actor is not affiliated with or endorsed by Substack Inc. It collects only data that is publicly visible without logging in. You are responsible for using the data in line with Substack's terms, authors' copyright and the laws that apply to you.

### Support and feedback

Missing a field you need, or found a bug? Open an issue in the **Issues** tab on this page. If the Actor saves you time, a short review helps other people find it.

# Actor input Schema

## `publications` (type: `array`):

Substack publications to scrape. Use a subdomain (e.g. astralcodexten), a substack.com URL (https://lenny.substack.com) or a custom domain (https://www.lennysnewsletter.com). Any post URL works too. Newest posts come first.

## `maxPostsPerPublication` (type: `integer`):

Maximum number of posts to save from each publication (newest first). Also caps the number of search results. You are charged per post saved, so this caps your cost (20 posts is about $0.04).

## `includeBody` (type: `boolean`):

Add the post text as plain text (HTML stripped) in bodyText. Free posts get the full text. Paid posts get only the public preview, never paywalled content. Needs one extra request per free post, so runs are slower.

## `searchQuery` (type: `string`):

Search posts across all of Substack, like the search box on substack.com. Examples: product management, AI agents, bitcoin. Returns up to 'Max posts per publication' posts. Can be combined with Publications or used alone.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com",
    "astralcodexten"
  ],
  "maxPostsPerPublication": 10,
  "includeBody": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com",
        "astralcodexten"
    ],
    "maxPostsPerPublication": 10,
    "searchQuery": ""
};

// Run the Actor and wait for it to finish
const run = await client.actor("deanko/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": [
        "https://www.lennysnewsletter.com",
        "astralcodexten",
    ],
    "maxPostsPerPublication": 10,
    "searchQuery": "",
}

# Run the Actor and wait for it to finish
run = client.actor("deanko/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com",
    "astralcodexten"
  ],
  "maxPostsPerPublication": 10,
  "searchQuery": ""
}' |
apify call deanko/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,deanko/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Cf18qhXkDOwrAXfHU/builds/pix3KWvIPzCcORS8p/openapi.json