# Substack Scraper | Posts, Full Text & Comments | $0.99/1K (`codetr/substack-scraper`) Actor

Scrape Substack newsletters: posts, full text (HTML + plain), comments, authors, tags and engagement. Search by keyword, filter by date, get only new posts. $0.99 per 1,000 results (less on paid plans), content included. No login, no cookies.

- **URL**: https://apify.com/codetr/substack-scraper.md
- **Developed by:** [Yusuf Barış](https://apify.com/codetr) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper | Posts, Full Text & Comments | $0.99/1K

Scrape any public **Substack newsletter** into clean, structured data: post metadata, **full article text (HTML and plain text)**, authors, tags, engagement counts and **comments with replies**. No login, no cookies and no browser needed, so runs are fast, cheap and reliable.

**$0.99 per 1,000 results at most, full content included, and cheaper on paid Apify plans.** No surcharge for content or comments, and only a negligible $0.00005 per-run start charge applied by Apify.

### What you can do with it

- **Sponsor and partnership research** — find newsletters in your niche, see how often they publish, how many likes, restacks and comments each post earns, and who writes them.
- **Content and competitor analysis** — export a publication's whole archive, track topics and tags over time, compare engagement between newsletters.
- **LLM and RAG datasets** — get clean plain text plus original HTML for thousands of articles, ready for fine-tuning, embeddings or summarisation.
- **Audience research** — read what readers say in comments, who replies to whom, and which posts spark discussion.
- **Monitoring** — run on a schedule with **incremental mode** and receive only posts you have not seen yet.
- **Keyword discovery** — search all of Substack for a topic and scrape every matching post.

### Features

- Full archive of one or many publications (`name.substack.com` or custom domains such as `www.lennysnewsletter.com`)
- A single post URL works too (`https://example.substack.com/p/some-post`)
- Full content as HTML **and** plain text
- Comments with nested replies, author flag and likes (optional)
- Date range filter (`startDate` / `endDate`)
- Free-only filter (`onlyFree`)
- **Incremental mode** — return only new posts since the last run
- Keyword search across Substack (`searchQueries`)
- Built-in retries and rate-limit handling, so long runs finish cleanly

### Pricing

Pay per result: **$0.99 / 1,000 results** (`$0.00099` per post) on the standard price. Paid Apify plans get lower per-result prices automatically. Each post written to the dataset is one result.

| What you get per result | Included |
| --- | --- |
| Metadata (title, date, audience, counts, tags, authors, cover image) | Yes |
| Full content (HTML + plain text) | Yes |
| Comments tree | Yes |
| Per-run start charge (set by Apify) | $0.00005 |

| Results | Cost at the standard price |
| --- | --- |
| 100 | $0.099 |
| 1,000 | $0.99 |
| 10,000 | $9.90 |
| 100,000 | $99.00 |

Use `maxItems` to cap spend. Apify's free plan credits cover your first runs.

### Input

| Field | Type | Description |
| --- | --- | --- |
| `publications` | string\[] | Subdomain (`lenny`), URL (`https://lenny.substack.com`) or custom domain. A `/p/slug` URL scrapes one post. |
| `searchQueries` | string\[] | Optional keywords to search all of Substack. |
| `maxPostsPerPublication` | integer | Max posts per publication and per search query. Default `100`. |
| `maxItems` | integer | Total result cap for the run. Default `100`. |
| `includeContent` | boolean | Fetch full HTML and plain text. Default `true`. |
| `includeComments` | boolean | Fetch comments and replies. Default `false`. |
| `maxCommentsPerPost` | integer | Comments kept per post (`0` = unlimited). Default `100`. |
| `startDate` / `endDate` | string | Only posts published within this range (`YYYY-MM-DD`, inclusive). |
| `onlyFree` | boolean | Skip paid posts. Default `false`. |
| `incremental` | boolean | Only return posts not returned by earlier incremental runs. Default `false`. |
| `proxyConfiguration` | object | Optional. Usually not needed. |

#### Example input

```json
{
    "publications": ["https://www.lennysnewsletter.com", "astralcodexten"],
    "searchQueries": ["product management"],
    "maxPostsPerPublication": 50,
    "maxItems": 100,
    "includeContent": true,
    "includeComments": true,
    "maxCommentsPerPost": 50,
    "startDate": "2026-01-01",
    "onlyFree": true,
    "incremental": false
}
```

### Output

Each dataset item is one post:

```json
{
    "postId": 208730073,
    "slug": "announcing-lennys-jobs-the-best-place",
    "publication": "Lenny's Newsletter",
    "publicationUrl": "https://www.lennysnewsletter.com",
    "title": "Announcing Lenny’s Jobs: The best place in the world to find, vet, and land your dream job",
    "subtitle": "Where product managers, engineers, designers, and growth/marketing professionals discover high-quality open roles at tech companies",
    "url": "https://www.lennysnewsletter.com/p/announcing-lennys-jobs-the-best-place",
    "publishedAt": "2026-08-18T15:40:06.921Z",
    "postType": "newsletter",
    "audience": "everyone",
    "isPaywalled": false,
    "wordcount": 933,
    "likes": 382,
    "restacks": 9,
    "commentCount": 18,
    "tags": ["Career"],
    "authors": [
        {
            "name": "Lenny Rachitsky",
            "handle": "lenny",
            "bio": "Writing • Angel investing • Advising",
            "photoUrl": "https://bucketeer-e05bbc84-baa3-437e-9518-adb32be77984.s3.amazonaws.com/public/images/afba5161-65bb-4d99-8d6b-cce660917fa1_1540x1540.png",
            "url": "https://substack.com/@lenny"
        }
    ],
    "coverImage": "https://substackcdn.com/image/fetch/…",
    "contentHtml": "<p><em><span>👋 Hey there, I’m Lenny. …</span></em></p>",
    "contentText": "👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, …",
    "comments": [
        {
            "id": 317389445,
            "body": "The Lenny 100 list is GENIUS 👏",
            "author": "Tom Orbach",
            "handle": "marketingideas",
            "date": "2026-08-18T15:48:15.832Z",
            "likes": 7,
            "isAuthor": false,
            "children": []
        }
    ]
}
```

`comments` is present only when `includeComments` is on. Items from keyword search also carry `searchQuery`. Download results as JSON, CSV, Excel, XML or HTML, or use the API.

### Paid (paywalled) posts

Substack only exposes the **public preview** of subscriber-only posts to anonymous readers, and so does this Actor. Paid posts are returned with `audience: "only_paid"`, `isPaywalled: true` and the preview text that appears before the paywall; `wordcount` still reflects the full article length. Comments on paid posts are hidden by Substack from non-subscribers, so they usually come back empty. Use `onlyFree` to skip paid posts entirely. The Actor does not log in or bypass paywalls.

### Incremental mode

Turn on `incremental` and schedule the Actor. The Actor remembers which posts it has already returned for each publication or search keyword and returns only new ones. Repeat runs are fast and cheap.

### Tips

- Start with a small `maxItems` to preview the output, then raise it.
- Metadata-only runs (`includeContent: false`) are the fastest, and the price per result is the same.
- To scrape a whole archive, set both `maxPostsPerPublication` and `maxItems` high.

### FAQ

**Do I need a Substack account or cookies?**
No. Only publicly available data is collected.

**Does it work with custom domains?**
Yes. Pass the domain (`https://www.lennysnewsletter.com`) or the `name.substack.com` address; both work.

**Why is the text of some posts short?**
Those are paid posts. Anonymous access returns only the preview that Substack shows before the paywall.

**Why did I get fewer results than `maxItems`?**
The publication has fewer posts than the limits, or your date range, `onlyFree` or `incremental` filters excluded the rest.

**What happens if Substack rate-limits me?**
The Actor slows down and retries automatically, so your run keeps going without any setup.

**How does the search work?**
`searchQueries` searches Substack for matching posts. Each match is returned with the same fields as any other post. For keyword search inside a single publication, scrape its archive and filter on `title` or `contentText`.

**Is scraping Substack legal?**
The Actor only reads publicly available data. You are responsible for using it in line with Substack's terms, applicable law and privacy regulations such as GDPR, especially when handling commenters' names and handles.

**Something is broken or missing. Who do I ask?**
Open an issue in the Actor's **Issues** tab and include your run URL.

# Actor input Schema

## `publications` (type: `array`):

Substack publications to scrape. Accepts a subdomain (`lenny`), a `x.substack.com` URL, or a custom domain (`https://www.lennysnewsletter.com`). A single post URL (`.../p/some-slug`) scrapes just that post.

## `searchQueries` (type: `array`):

Optional. Search all of Substack for posts matching these keywords. Results are scraped the same way as publication posts. Use together with, or instead of, publications.

## `maxPostsPerPublication` (type: `integer`):

Maximum number of posts collected from each publication and from each search query (after filters are applied).

## `maxItems` (type: `integer`):

Hard limit on the total number of results pushed to the dataset in one run. You are billed per result.

## `includeContent` (type: `boolean`):

Fetch each post's full HTML and plain text. Turn off for a faster metadata-only run. Price is the same.

## `includeComments` (type: `boolean`):

Fetch the comment tree (with replies) of every post.

## `maxCommentsPerPost` (type: `integer`):

Maximum comments (top-level plus replies) kept per post. 0 means no limit.

## `startDate` (type: `string`):

Only posts published on or after this date (YYYY-MM-DD).

## `endDate` (type: `string`):

Only posts published on or before this date (YYYY-MM-DD).

## `onlyFree` (type: `boolean`):

Skip paid (subscriber-only) posts. Without this, paid posts are returned with their public preview only.

## `incremental` (type: `boolean`):

Only return posts not returned by a previous incremental run with the same publication / query. State is kept in a named key-value store, so it survives between runs.

## `proxyConfiguration` (type: `object`):

Optional. Usually not needed - leave it as is.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 10,
  "maxItems": 10,
  "includeContent": true,
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "onlyFree": false,
  "incremental": false,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `posts` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com"
    ],
    "maxPostsPerPublication": 10,
    "maxItems": 10,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("codetr/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["https://www.lennysnewsletter.com"],
    "maxPostsPerPublication": 10,
    "maxItems": 10,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("codetr/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 10,
  "maxItems": 10,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call codetr/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,codetr/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/G4gWdqaUPaGKnBI6r/builds/BhF5Ah0B7QKhsNZul/openapi.json