# Substack Scraper (`mina_safwat/substack-scraper`) Actor

Scrapes posts from any Substack newsletter or custom domain, single posts, profiles and Substack search: title, authors, date, likes, comments, restacks, free or paid, word count and optional full text.

- **URL**: https://apify.com/mina_safwat/substack-scraper.md
- **Developed by:** [Mina](https://apify.com/mina_safwat) (community)
- **Categories:** News, For creators, Marketing
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.90 / 1,000 posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What does Substack Scraper do?

**Substack Scraper** collects posts from **any Substack newsletter**, whether
it lives on `name.substack.com` or on its **own custom domain**. For every post
you get the **title, subtitle, authors, publish date, likes, comments,
restacks, free or paid status, word count and cover image**, and, if you want
it, the **full text of free posts**. Paste newsletter links, single post links
or author profiles, or **search all of Substack** by keyword.

Try it with the prefilled newsletter: open the Actor, click **Start**, and the
20 newest posts of [Lenny's Newsletter](https://www.lennysnewsletter.com)
arrive in a few seconds. Because it runs on the Apify platform, you also get
API access, scheduled runs, integrations (Google Sheets, Make, Zapier,
webhooks), run monitoring and automatic retries through Apify Proxy when
[Substack](https://substack.com) is slow to answer.

### Why scrape Substack?

- **Creator and market research**: see which posts of a newsletter earn the
  most likes, comments and restacks, and how often it publishes.
- **Competitor tracking**: follow the newsletters in your niche and get every
  new post as structured data on a schedule.
- **Content and trend analysis**: search Substack for a topic and compare what
  dozens of writers publish about it.
- **Sponsorship and partnership scouting**: find active newsletters on a
  subject and gauge their engagement before reaching out.
- **Reading lists, archives and AI datasets**: export the full text of free
  posts for analysis, summaries or model training.

### How to scrape Substack newsletters

1. Copy the address of a newsletter, such as `https://example.substack.com`
   or a custom domain like `https://www.lennysnewsletter.com`. You can leave
   out `https://`.
2. Paste it into **Substack links**, one link per line. A link to a single
   post (`…/p/post-name`), one section of a newsletter (`…/s/section-name`),
   a search of its archive (`…/archive?search=pricing`) or an author's profile
   (`https://substack.com/@handle`) works too.
3. Optionally, type words into **Search Substack** to find posts across all
   newsletters.
4. Set **Max posts per newsletter or search**, choose **Newest first** or
   **Most popular first**, and turn on **Include full text of free posts** if
   you need the text.
5. Click **Start**, then download the results when the run finishes.

### Input

Everything is set on the Input tab. In JSON, the input looks like this:

```json
{
    "start_urls": [
        "https://www.lennysnewsletter.com",
        "astralcodexten.substack.com/archive?sort=top",
        "https://www.noahpinion.blog/p/this-thing-will-fail",
        "https://www.lennysnewsletter.com/s/community-wisdom",
        "https://substack.com/@lenny"
    ],
    "search_queries": ["product management"],
    "max_posts_per_newsletter": 50,
    "sort": "new",
    "include_full_text": false
}
```

| Field | What it does |
|---|---|
| `start_urls` | Newsletter homepages or `/archive` pages (on `substack.com` or a custom domain), one section of a newsletter (`/s/section-name`), a search of one newsletter's archive (`/archive?search=your+words`), post links (`/p/post-name`, quote share links `/i/123/…`, `substack.com/home/post/p-123`, `substack.com/inbox/post/123`, `open.substack.com/pub/name/p/post-name`), author profiles (`substack.com/@handle`) and Substack-wide search pages (`substack.com/search/your%20words`). Notes, tag (`/t/…`) and podcast (`/podcast`) pages are not supported yet and are listed in `FAILED_INPUTS`, as are a search inside a section (`/s/…?search=…`, which Substack answers with nothing; use `/archive?search=…`) and any other page of a newsletter, so a link is never widened to the whole newsletter. With or without `https://`; API callers can also send `{"url": "…"}` objects. |
| `search_queries` | Optional. Words or phrases; each runs Substack's post search across all newsletters, most relevant first. Up to 500 characters each; longer searches are listed in `FAILED_INPUTS`. |
| `max_posts_per_newsletter` | Stops after this many posts for each newsletter, section, profile or search. Default: 50. Single post links always give their one post. |
| `sort` | `new` (newest first, the default) or `top` (most popular first), also for section links. A link that already ends in `?sort=top` or `?sort=new` keeps its own order. Searches, including archive searches, are in Substack's best-match order. |
| `include_full_text` | Adds a `text` field with the plain text of each **free** post. Default: off. Paid posts never get text. |
| `proxy_configuration` | Optional; leave it empty. Substack answers the Actor directly, and the Actor switches to Apify's datacenter and then residential proxies by itself if it is ever refused. |

### Output

Each post becomes one row. Here is a real free post, saved with full text on
(the text is shortened here):

```json
{
    "id": 215694124,
    "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
    "title": "All of the Lenny & Friends Summit talks are now online!",
    "subtitle": "Plus, some reflections and takeaways from the day",
    "newsletter_name": "Lenny's Newsletter",
    "newsletter_url": "https://www.lennysnewsletter.com",
    "authors": ["Lenny Rachitsky"],
    "author_handles": ["lenny"],
    "published_at": "2026-09-29T13:15:57.512Z",
    "audience": "free",
    "is_paywalled": false,
    "post_type": "newsletter",
    "reactions": 283,
    "comments": 6,
    "restacks": 5,
    "word_count": 1276,
    "cover_image": "https://substack-post-media.s3.amazonaws.com/public/images/9fed347c-0e02-4190-a590-1236719b99de_9213x6142.jpeg",
    "text": "👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice. …\n\nI’m excited to share that every main-stage talk from the recent Lenny & Friends Summit is now online. …",
    "source": "https://www.lennysnewsletter.com",
    "scraped_at": "2026-10-10T14:03:07.146048+00:00"
}
```

A paid post from the same newsletter has `"audience": "paid"`,
`"is_paywalled": true` and no `text`. The Output tab has two views, **Posts**
and **Full text**. You can download the dataset in various formats such as
JSON, HTML, CSV, or Excel.

### Data fields

| Field | Description |
|---|---|
| `id` | Substack's id for the post. |
| `url` | Link to the post. |
| `title` | The post's title. |
| `subtitle` | The subtitle, when the post has one. |
| `newsletter_name` | The newsletter's name. |
| `newsletter_url` | The newsletter's address (its custom domain when it has one). |
| `authors` | Names of the post's authors, guests included. |
| `author_handles` | Their Substack handles; each profile is at `https://substack.com/@handle`. |
| `published_at` | When the post was published (UTC, ISO 8601). |
| `audience` | `free` (anyone can read it), `paid` (paid subscribers), `founding` (founding members only) or `subscribers` (free sign-up required). |
| `is_paywalled` | `true` for every post that is not free to read without subscribing. |
| `post_type` | Substack's kind of post: `newsletter` for a written post, `podcast` for an episode, and so on. |
| `reactions` | Likes (hearts) on the post. |
| `comments` | Number of comments, replies included. |
| `restacks` | Number of restacks. |
| `word_count` | Substack's word count for the whole post, also shown for paid posts. |
| `cover_image` | The cover image URL, when the post has one. |
| `text` | Only with `include_full_text`, and only for free posts: the post as plain text, one paragraph per line, footnotes as `[1]`. |
| `source` | The link or search the post was found through, so posts from a section or archive-search link can be told apart. |
| `scraped_at` | When the row was saved. |

### How much does it cost to scrape Substack?

Substack Scraper uses **pay-per-event** pricing, so you pay only for results:
a fee for each post saved, plus a separate fee for each post saved **with its
full text**, which takes an extra request per post. See the **Pricing** tab
for the exact prices.

Inputs that return nothing are never saved as rows, so you are not charged for
them, and each post is saved and charged only once per run, even when several
links lead to it. If you set a maximum cost for a run, the Actor stops as soon
as it is reached. Apify's free plan includes monthly credit you can use to try
it.

### Tips for scraping Substack

- **Leave full text off** when you only need titles, dates and engagement:
  runs are faster and cheaper, because a whole page of 25 posts arrives in one
  request.
- **Most popular first** (`sort: top`) with a small limit gives a newsletter's
  best-performing posts in one go.
- **Whole archives:** set `max_posts_per_newsletter` to a large number such as
  `10000`. The Actor pages through the archive until it ends.
- **Monitoring:** schedule a daily run with a small limit (10–20) and newest
  first to catch new posts.
- **One section or topic of a newsletter:** paste a section link
  (`https://www.lennysnewsletter.com/s/community-wisdom`) or an archive search
  (`https://www.lennysnewsletter.com/archive?search=pricing`) instead of the
  whole newsletter. Some sections, such as members' digests, are not listed
  by popularity on Substack; use newest first for them.
- **Search finds posts, not newsletters.** To scrape a whole newsletter you
  found in a search, paste its address into `start_urls`. Substack's search
  shows some posts on more than one page; each is saved once and counted once
  toward the limit.
- **Profiles:** `substack.com/@handle` gives the posts of that person's own
  newsletter. For posts they wrote as a guest elsewhere, paste those
  newsletters' links.

### FAQ, disclaimers and support

**Does it unlock paid posts?** No. The Actor reads Substack exactly as a
logged-out visitor does and never bypasses a paywall. Paid, founding-member
and subscriber-only posts are saved with their public details (title,
subtitle, authors, date, likes, comments, restacks, word count, cover image)
and marked `is_paywalled: true`, but never with their text, even when
**Include full text** is on.

**Does it work with custom domains?** Yes. Any newsletter hosted on Substack
works, whether its address is `name.substack.com` or its own domain.

**What happens with a wrong or deleted link?** Links that are not Substack
newsletters, newsletters that don't exist, addresses with a mistyped domain,
sections that don't exist, deleted posts and searches without results are
skipped and listed with the reason in the `FAILED_INPUTS` record
of the run's key-value store. The run still succeeds with the posts it could
read; it fails only when Substack could not be reached for any input.

**Can it scrape comments or Notes?** Not yet. It saves the comment count of
each post, but not the comments themselves. Notes links
(`substack.com/@handle/note/…`) are skipped and listed in `FAILED_INPUTS`, so
they are never charged.

**Is it legal to scrape Substack?** The Actor collects only data Substack
shows publicly to any visitor. Posts can contain personal data such as author
names; make sure you have a legitimate reason to collect it and follow the
GDPR and other regulations that apply to you. Respect authors' copyright when
you reuse their writing. Substack's robots.txt and Terms of Service apply to
your use; if you are unsure, consult your lawyer.

**Something missing or not working?** Please open an issue on the **Issues**
tab. We read every report and usually fix problems quickly. Need a custom
solution, such as comments, Notes or subscriber leaderboards? Get in touch
through the Issues tab.

# Actor input Schema

## `start_urls` (type: `array`):

One link per line, with or without https://. A newsletter (https://example.substack.com, a custom domain such as https://www.lennysnewsletter.com, or its /archive page) gives its posts; a section link (…/s/section-name) or an archive search (…/archive?search=pricing) gives only those posts; a post link (…/p/post-name) gives that one post; an author profile (https://substack.com/@handle) gives the posts of the author's own newsletter; a search page (https://substack.com/search/ai%20agents) gives the posts Substack's search finds. Links that cannot be read, such as Substack Notes, tag or podcast pages, are listed in the FAILED_INPUTS record and never widened to the whole newsletter.

## `search_queries` (type: `array`):

Words or phrases, one per line, such as product management. Each one runs Substack's own post search across all newsletters and saves the posts it finds, most relevant first.

## `max_posts_per_newsletter` (type: `integer`):

Stops after this many posts for each newsletter, profile or search. Single post links always give their one post. Set a large number, such as 10000, to read a whole archive.

## `sort` (type: `string`):

The order a newsletter's posts are read in, so the limit above keeps the newest or the most popular. A newsletter link that already says ?sort=top or ?sort=new keeps its own order. Searches are always ordered by relevance.

## `include_full_text` (type: `boolean`):

Adds each free post's text, as plain text, in a "text" field. This needs one extra request per post, so runs are slower, and each row with text is billed as a full-text post on top of the row. Paid and subscriber-only posts never get text: the Actor reads only what a logged-out visitor sees and never unlocks a paywall.

## `proxy_configuration` (type: `object`):

Leave this empty: Substack answers the Actor directly, and if it ever refuses, the Actor retries through Apify's datacenter proxy and then residential proxies on its own.

## Actor input object example

```json
{
  "start_urls": [
    "https://www.lennysnewsletter.com"
  ],
  "search_queries": [],
  "max_posts_per_newsletter": 20,
  "sort": "new",
  "include_full_text": false,
  "proxy_configuration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "start_urls": [
        "https://www.lennysnewsletter.com"
    ],
    "max_posts_per_newsletter": 20
};

// Run the Actor and wait for it to finish
const run = await client.actor("mina_safwat/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "start_urls": ["https://www.lennysnewsletter.com"],
    "max_posts_per_newsletter": 20,
}

# Run the Actor and wait for it to finish
run = client.actor("mina_safwat/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "start_urls": [
    "https://www.lennysnewsletter.com"
  ],
  "max_posts_per_newsletter": 20
}' |
apify call mina_safwat/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,mina_safwat/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/9A90i1KRWkiCTnNon/builds/Fj1BKQD2e2npD1A0g/openapi.json