# Substack Scraper - Newsletter Posts, Authors and Engagement (`sweet_stickball/substack-scraper`) Actor

Substack scraper and API alternative: newsletter posts from any Substack publication or custom domain, or find publications by keyword, with title, authors, date, free or paid audience, word count, likes, comments, restacks, tags and optional full text of free posts. No login.

- **URL**: https://apify.com/sweet_stickball/substack-scraper.md
- **Developed by:** [CrawlKit](https://apify.com/sweet_stickball) (community)
- **Categories:** News
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 75.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.48 / 1,000 newsletter posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper - Newsletter Posts, Authors and Engagement

Scrape newsletter posts from any Substack publication, on a substack.com subdomain or on a custom domain, or find publications by keyword first. Each post becomes one flat dataset item with title, subtitle, authors, publication date, audience (free or paid), word count, likes, comments, restacks, tags, cover image and, optionally, the post text.

The Actor reads the public JSON API that every Substack publication serves (`/api/v1/archive`, `/api/v1/posts/<slug>`) and Substack's public publication search. No login, no browser.

### What it does

- Takes one or more publications (homepage URL, subdomain, custom domain or any post URL) and returns their posts, newest first or most popular first.
- Optional keyword search (`searchQuery`) finds publications on Substack and scrapes their posts too.
- Optional `postedAfter` filter: an absolute date (`2026-01-31`) or a relative period (`30 days`, `3 months`).
- Optional post text (`includeContent`): full text of free posts; for paid posts, only the public preview shown before the paywall.
- Posts are deduplicated by `postId`.

### Input

| Parameter | Type | Required | Default | Description |
|-----------|------|----------|---------|-------------|
| `publications` | array of strings | Yes, unless `searchQuery` is set | — | Publications to scrape: `https://www.lennysnewsletter.com`, `https://astralcodexten.substack.com`, `astralcodexten`, `newsletter.pragmaticengineer.com` or any post URL of the publication. |
| `searchQuery` | string | No | — | Keyword to find publications with Substack's publication search (e.g. `product management`). |
| `maxPublications` | integer | No | 5 | Number of publications taken from the search (1-100). |
| `sort` | `new` or `top` | No | `new` | Newest posts first, or the publication's most popular posts first. |
| `includeContent` | boolean | No | `false` | Adds `contentText` (one extra request per post). |
| `postedAfter` | string | No | — | Only posts published after this date (`2026-01-31`, `7 days`, `2 weeks`, `3 months`, `1 year`). |
| `maxItems` | integer | No | 20 | Maximum posts per publication (1-5000). |

Example:

```json
{
    "publications": ["https://www.lennysnewsletter.com", "astralcodexten"],
    "sort": "new",
    "postedAfter": "30 days",
    "includeContent": true,
    "maxItems": 50
}
```

### Output

One dataset item per post (real example, `contentText` shortened):

```json
{
    "postId": 170393023,
    "publication": "Climate Town",
    "publicationUrl": "https://www.climatetown.news",
    "title": "We Made a Graph",
    "subtitle": null,
    "description": "This one might be easier for Joe Rogan to understand.",
    "slug": "we-made-a-graph",
    "url": "https://www.climatetown.news/p/we-made-a-graph",
    "postType": "newsletter",
    "authors": "Matt Nelsen, Rollie Williams",
    "postDate": "2025-08-20T13:02:59.406000+00:00",
    "audience": "everyone",
    "isPaywalled": false,
    "wordCount": 1986,
    "reactionCount": 108,
    "commentCount": 31,
    "restacks": 20,
    "coverImage": "https://substackcdn.com/image/fetch/$s_!rHfm!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fe8609ef9-99c8-4046-80ef-a38c40690946_1600x966.jpeg",
    "tags": "climate change, climate, mel gibson, bernie sanders, science, joe rogan",
    "language": "en",
    "contentText": "Hello friend!\nHere at Climate Town we almost never make our own graphs. ...",
    "scrapedAt": "2026-10-28T06:52:06.846700+00:00"
}
```

| Field | Description |
|-------|-------------|
| `postId` | Substack post id |
| `publication`, `publicationUrl` | Publication name and homepage (custom domain when it has one) |
| `title`, `subtitle`, `description`, `slug`, `url` | Post identity |
| `postType` | `newsletter`, `podcast`, `thread` or `video` |
| `authors` | Public byline names, comma-separated |
| `postDate` | Publication date, ISO 8601 UTC |
| `audience`, `isPaywalled` | `everyone` (free), `only_paid` or `founding`; `isPaywalled` is true when the full post is for paid subscribers |
| `wordCount` | Word count of the full post, paid posts included |
| `reactionCount`, `commentCount`, `restacks` | Likes, comments and restacks |
| `coverImage`, `tags`, `language` | Cover image URL, tags (comma-separated), language code |
| `contentText` | Plain text of the post, only with `includeContent` (paid posts: public preview only) |
| `scrapedAt` | Time of extraction |

### Pricing

Pay-per-event, no start fee: one `POST` event per post returned. Proposed grid, in USD per 1,000 posts:

| Event | Free | Starter | Scale | Business | Enterprise |
|-------|------|---------|-------|----------|------------|
| `POST` | $0.80 | $0.68 | $0.60 | $0.48 | $0.48 |

The run stops cleanly when the maximum cost per run you set is reached.

### Limitations

- Paid posts: only the public preview (the part before the paywall) is available in `contentText`; `wordCount` still reports the full length. The Actor does not log in.
- Comments are counted (`commentCount`) but not extracted, and subscriber counts are not returned.
- Profile URLs (`substack.com/@handle`) and Substack Notes are not supported; give the publication URL instead.
- The keyword search returns the publications Substack's own search ranks first (about 20 per page); `maxPublications` caps how many are used.
- `sort: top` with `postedAfter` filters the popular posts by date; it may need to read a large part of the archive for old publications.
- Publication names come from the publication homepage; when it cannot be read, the host name is used.

# Actor input Schema

## `publications` (type: `array`):

Substack publications to scrape. Accepts a homepage URL (https://www.lennysnewsletter.com), a substack.com subdomain URL (https://astralcodexten.substack.com), a bare subdomain (astralcodexten), a custom domain (newsletter.pragmaticengineer.com) or any post URL of the publication. Profile URLs such as substack.com/@handle are not supported. Required unless 'searchQuery' is set.

## `searchQuery` (type: `string`):

Optional. Finds publications with Substack's public publication search (e.g. 'product management', 'climate') and scrapes their posts, in addition to 'publications'.

## `maxPublications` (type: `integer`):

How many publications to take from the keyword search (used only with 'searchQuery').

## `sort` (type: `string`):

'new' = most recent posts first (archive order). 'top' = the publication's most popular posts first.

## `includeContent` (type: `boolean`):

Adds 'contentText' (plain text of the post body). Full text for free posts; for paid posts only the public preview shown before the paywall. Makes one extra request per post, so runs are slower.

## `postedAfter` (type: `string`):

Optional. Only posts published after this date: an absolute date (2026-01-31) or a relative period ('7 days', '2 weeks', '3 months', '1 year'). Empty = no date filter.

## `maxItems` (type: `integer`):

Maximum number of posts returned for each publication.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com",
    "astralcodexten",
    "https://newsletter.pragmaticengineer.com"
  ],
  "searchQuery": "product management",
  "maxPublications": 5,
  "sort": "new",
  "includeContent": false,
  "postedAfter": "30 days",
  "maxItems": 10
}
```

# Actor output Schema

## `posts` (type: `string`):

Newsletter posts shown with the 'posts' view.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com"
    ],
    "maxItems": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("sweet_stickball/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["https://www.lennysnewsletter.com"],
    "maxItems": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("sweet_stickball/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "maxItems": 10
}' |
apify call sweet_stickball/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,sweet_stickball/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/IjMOm2RdiFP3o8ZTE/builds/kppQxCSDq6YUERhcR/openapi.json