# Substack Scraper — Newsletter Posts, Search & Profiles (`fabrikit/substack-scraper`) Actor

Scrape Substack newsletters: every post of any newsletter or custom domain (title, date, likes, comments, restacks, paid or free, word count, authors, tags), post search across Substack, top newsletters by category and newsletter profiles. Optional full text.

- **URL**: https://apify.com/fabrikit/substack-scraper.md
- **Developed by:** [Fabrikit](https://apify.com/fabrikit) (community)
- **Categories:** News, Social media, Lead generation
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 66.7% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.75 / 1,000 post scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper — Newsletter Posts, Search & Profiles

Substack Scraper extracts posts and newsletter profiles from substack.com and from any newsletter that runs on Substack under its own domain. Give it newsletter addresses, keywords or categories and get one flat JSON row per post: title, subtitle, link, publication date in ISO 8601, free or paid, word count, likes, comment count, restacks, tags, authors, cover image, podcast audio, and the newsletter it belongs to. Turn on the body add-on to also get each post's text or HTML. No browser and no login: the Actor reads the same public JSON that Substack's own web pages load.

**Pay per event.** Post bodies are an add-on, charged per post returned with text. Newsletter profiles are charged per newsletter row.

### What you can scrape

- **Whole newsletter archives.** Paste `simonw`, `https://simonw.substack.com` or a custom domain such as `https://www.lennysnewsletter.com`. The archive is read newest first, or most popular first with `sort: "top"`, 50 posts per request, as deep as you want (`maxPostsPerSource: 0` reads the whole archive).
- **Single posts.** Post links (`https://name.substack.com/p/slug`, custom-domain `/p/` links, `open.substack.com/pub/name/p/slug`).
- **Search inside a newsletter.** `archiveSearch` uses the newsletter's own archive search, for example every Lenny's Newsletter post about `pricing`.
- **Post search across Substack.** `postSearchQueries` returns the posts Substack's search finds for a keyword, from any newsletter.
- **Newsletter discovery.** `searchQueries` finds newsletters by keyword; `categories` takes the top of Substack's category leaderboards (32 categories, from Technology, Business and Finance to Culture, Politics and Science). Their posts are then scraped like any other newsletter.
- **Newsletter profiles.** Set `outputType: "newsletters"` to get one row per newsletter instead of posts.

### Output

One row per post:

| Field | Meaning |
|---|---|
| `postId`, `title`, `subtitle`, `url`, `slug` | Identity and canonical link (custom domain when there is one) |
| `publishedAt`, `updatedAt` | ISO 8601 UTC dates (`updatedAt` for single post links and the body add-on) |
| `type`, `audience`, `isPaywalled` | `newsletter` or `podcast`; `everyone` or `only_paid` |
| `description`, `previewText` | SEO description and the opening lines shown in the archive |
| `wordCount` | Length of the full post (also known for paid posts) |
| `likes`, `commentCount`, `restacks` | Engagement as shown on Substack |
| `tags`, `section`, `language`, `coverImageUrl` | Post metadata |
| `podcastUrl`, `podcastDurationSecs`, `hasVoiceover` | Audio for podcast episodes |
| `authors`, `authorNames` | Bylines: name, handle, profile link, guest flag |
| `newsletterId`, `newsletterName`, `newsletterUrl`, `newsletterSubdomain` | The newsletter |
| `contentText`, `contentHtml`, `contentWordCount`, `contentIsPreview` | Body add-on (`includeContent`): the full post, or the free preview for paid posts |
| `source`, `sourceQuery`, `scrapedAt` | Which input produced the row |

With `outputType: "newsletters"` each row describes a newsletter: `name`, `url`, `substackUrl`, `customDomain`, `description`, `authorName`, `authorHandle`, `contributors`, `language`, `launchedAt`, `subscriberCount` (as displayed by Substack, when the author shows it), `subscriberCountLabel` (for example `73K+`), `audienceLabel`, `paidSubscriptionsEnabled`, `monthlyPrice`, `yearlyPrice`, `foundingMemberPrice`, `priceCurrency`, `hasPodcast`, `podcastTitle`, `logoUrl`, `twitterHandle`, `rank` (position in the search or leaderboard).

The key-value store also gets `SUMMARY` (rows saved and charged, posts seen and filtered, requests, stop reason) and `SOURCES` (status of every newsletter, search and category, with the reason when something returned nothing).

### Input examples

Default run: the 10 newest posts of two newsletters.

```json
{
  "newsletters": ["https://simonw.substack.com", "https://www.lennysnewsletter.com"],
  "maxPostsPerSource": 10,
  "maxItems": 20
}
```

Every free post of the last 90 days from the top 10 Finance newsletters, with the text:

```json
{
  "categories": ["finance"],
  "maxNewslettersPerSearch": 10,
  "maxPostsPerSource": 0,
  "publishedAfter": "90 days",
  "audience": "free",
  "includeContent": true,
  "maxItems": 2000
}
```

Sponsorship research: profiles of newsletters about product management, with subscriber labels and plan prices:

```json
{
  "outputType": "newsletters",
  "searchQueries": ["product management", "growth marketing"],
  "maxNewslettersPerSearch": 25
}
```

Posts about a topic from all of Substack, most liked only:

```json
{ "postSearchQueries": ["AI agents"], "maxPostsPerSource": 50, "minLikes": 50 }
```

### Example output

```json
{
  "postId": 221405112,
  "title": "Battle Drones - a vibe coded RTS with audio-based multiplayer",
  "subtitle": "Plus Muse Wonder and Gemini 4 Argon",
  "url": "https://simonw.substack.com/p/battle-drones-a-vibe-coded-rts-with",
  "publishedAt": "2026-10-21T00:01:39.484Z",
  "type": "newsletter",
  "audience": "everyone",
  "isPaywalled": false,
  "wordCount": 6518,
  "likes": 38,
  "commentCount": 6,
  "restacks": 1,
  "tags": ["anthropic", "llms", "ai"],
  "authors": [{ "name": "Simon Willison", "handle": "simonw", "profileUrl": "https://substack.com/@simonw", "isGuest": false }],
  "newsletterName": "Simon Willison’s Newsletter",
  "newsletterUrl": "https://simonw.substack.com",
  "source": "newsletter",
  "scrapedAt": "2026-10-21T13:19:11.526Z"
}
```

### Use cases

- **Content and trend research:** what the biggest newsletters in a niche publish, how often, and which posts earn the most likes and restacks.
- **Sponsorship and partnership prospecting:** newsletter profiles with audience labels, paid plan prices and authors, ranked by category.
- **Competitive monitoring:** schedule a daily run with `publishedAfter: "1 day"` on the newsletters you follow and push new posts to Slack, a sheet or a CRM.
- **AI and RAG pipelines:** `contentText` of free posts, ready to summarise or embed; an AI agent can call the Actor with a keyword and get the latest posts on the topic.
- **Academic and media analysis:** full archives with dates and engagement for a set of writers.

### Pricing

Pay per event, no subscription:

- `post-result`: charged per post row saved.
- `post-content` (add-on, only with `includeContent`): charged per post whose body was returned.
- `newsletter-result`: charged per newsletter row.

Apify adds its small compute cost; the default run takes a few seconds. `maxItems` is exact and the Actor also stops at the maximum charge you set for the run. See the Pricing tab for current prices; paid Apify plans get lower rates.

### Compared with alternatives

| | This Actor | A newsletter's RSS feed | Browser-based scrapers |
|---|---|---|---|
| Archive depth | entire archive, paged | recent posts only | slow, page by page |
| Likes, comments, restacks, paid flag, word count | yes | no | sometimes |
| Search and category discovery | yes | no | rarely |
| Speed | 50 posts per request | 1 request | one page load per post |

If you only need the latest items of many different feeds (Substack or not), our [RSS & Atom Feed Reader](https://apify.com/fabrikit/rss-feed-scraper) is the general tool; this Actor is built on the same HTTP core and adds what only Substack's API has: engagement, paid/free status, full archives, search and newsletter profiles.

### Integrations

Use the Apify API, the JavaScript or Python client, the CLI, schedules and webhooks, and export to JSON, CSV, Excel, Google Sheets, Make, Zapier, n8n or Airbyte. The Actor also works through the Apify MCP server, so AI agents can ask for "the 20 most liked posts of a newsletter" or "newsletters about climate tech".

### FAQ

**Is it legal?** The Actor reads public pages and the public JSON Substack serves to every visitor, without logging in, and skips paywalled text beyond the preview the author makes public. Reader comments are never collected, and e-mail addresses and international phone numbers found in text are removed. Respect each author's copyright when you reuse the content.

**Do I get paid posts in full?** No. Paid posts give their metadata (title, date, likes, word count) and, with the add-on, the free preview the author chose to show (`contentIsPreview: true`).

**Do I need a proxy?** Normally not: requests go out directly. If Substack blocks the run, the Actor switches to Apify datacenter proxies and then residential proxies by itself (turn this off with `autoProxyFallback: false`).

**Why is `subscriberCount` empty for some newsletters?** Substack shows a count only when the author allows it; `subscriberCountLabel` and `audienceLabel` still give the range.

**Limits?** Up to 100,000 rows per run. Post search pages through whatever Substack's own search returns for a keyword, so very generic keywords stop when Substack stops returning results.

### En español

Substack Scraper extrae los posts de cualquier newsletter de Substack (también con dominio propio): título, fecha ISO, gratis o pago, palabras, likes, comentarios, restacks, etiquetas, autores y el texto opcional. También busca posts en todo Substack, encuentra newsletters por palabra clave o categoría y entrega perfiles de newsletters con suscriptores y precios. Consulta la pestaña Pricing para ver los precios actuales; los planes de pago de Apify tienen tarifas más bajas.

### Changelog

- **0.1 (October 2026):** first release. Archives (new/top, search inside a newsletter), single post links, post search, newsletter search, category leaderboards, newsletter profiles, date / type / audience / likes filters, body add-on, automatic proxy fallback.

# Actor input Schema

## `newsletters` (type: `array`):

One per line: a subdomain (simonw), a Substack address (https://simonw.substack.com), a custom domain (https://www.lennysnewsletter.com) or a post link (https://name.substack.com/p/slug, open.substack.com/pub/name/p/slug). The newsletter's archive is read newest first.

## `searchQueries` (type: `array`):

Keywords for Substack's newsletter search (for example: product management, climate tech). The best matches (up to 'Newsletters per search or category') are added to the list above.

## `categories` (type: `array`):

Substack leaderboard categories. The top newsletters of each (up to 'Newsletters per search or category', in leaderboard order) are added to the list.

## `maxNewslettersPerSearch` (type: `integer`):

How many newsletters each keyword search or category contributes.

## `postSearchQueries` (type: `array`):

Keywords searched in Substack's post search; the matching posts from any newsletter become rows (up to 'Posts per source' per keyword). With the newsletters output, the newsletters behind those posts are listed instead.

## `outputType` (type: `string`):

posts = one row per post (default). newsletters = one row per newsletter: name, author, description, subscriber count or label, paid plan prices, launch date, podcast, language.

## `maxPostsPerSource` (type: `integer`):

Maximum posts taken from each newsletter (or each post search). 0 = the whole archive.

## `maxItems` (type: `integer`):

Stop after this many rows in total. Exact: the run never saves or charges more.

## `sort` (type: `string`):

new = newest posts first. top = the newsletter's most popular posts first.

## `archiveSearch` (type: `string`):

Keyword passed to each newsletter's own archive search, for example pricing. Leave empty for every post.

## `publishedAfter` (type: `string`):

Keep posts published on or after this date: 2026-01-31, an ISO timestamp, or a relative period such as 7 days, 2 weeks, 3 months.

## `publishedBefore` (type: `string`):

Keep posts published before this date (same formats).

## `postTypes` (type: `array`):

Keep only these types. Empty = all types.

## `audience` (type: `string`):

all = every post. free = only posts open to everyone. paid = only posts for paying subscribers (their content is the free preview).

## `minLikes` (type: `integer`):

Skip posts with fewer likes (reactions) than this. 0 = no minimum.

## `includeContent` (type: `boolean`):

Fetch each post's body: contentText (plain text) and/or contentHtml, with its word count. Paid posts return only the free preview the author allows (contentIsPreview = true). Charged as 'post-content' per post with a body, on top of the post price; one extra request per post.

## `contentFormat` (type: `string`):

Used with the body add-on: text, html or both.

## `maxConcurrency` (type: `integer`):

How many newsletters are read at the same time. Requests to one host are always paced.

## `proxyConfiguration` (type: `object`):

Substack's API is public, so the default is no proxy. If Substack blocks the run, it switches to Apify datacenter and then residential proxies by itself; pick a proxy here to start with one.

## `autoProxyFallback` (type: `boolean`):

After repeated blocks, move to Apify datacenter proxies, then residential ones (residential traffic is billed by Apify as usual).

## Actor input object example

```json
{
  "newsletters": [
    "https://simonw.substack.com",
    "https://www.lennysnewsletter.com"
  ],
  "maxNewslettersPerSearch": 5,
  "outputType": "posts",
  "maxPostsPerSource": 10,
  "maxItems": 20,
  "sort": "new",
  "audience": "all",
  "minLikes": 0,
  "includeContent": false,
  "contentFormat": "text",
  "maxConcurrency": 3,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "autoProxyFallback": true
}
```

# Actor output Schema

## `results` (type: `string`):

All rows saved during the run

## `summary` (type: `string`):

Rows saved and charged, posts seen and filtered, requests, network used and stop reason

## `sources` (type: `string`):

For each newsletter, search or category: status, pages read, posts seen, filtered and saved, and any error

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "newsletters": [
        "https://simonw.substack.com",
        "https://www.lennysnewsletter.com"
    ],
    "maxPostsPerSource": 10,
    "maxItems": 20,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("fabrikit/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "newsletters": [
        "https://simonw.substack.com",
        "https://www.lennysnewsletter.com",
    ],
    "maxPostsPerSource": 10,
    "maxItems": 20,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("fabrikit/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "newsletters": [
    "https://simonw.substack.com",
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerSource": 10,
  "maxItems": 20,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call fabrikit/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,fabrikit/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Kz28emsjMA0U7hAsv/builds/oZ1C0Li1o6q7WzlxK/openapi.json