# Substack Scraper — Posts, Free vs Paid & Analytics (`kaankaan2635/substack-scraper`) Actor

Substack analytics: export publications and posts with engagement metrics, word counts, publishing cadence and the free vs paid split.

- **URL**: https://apify.com/kaankaan2635/substack-scraper.md
- **Developed by:** [Kaan Salgır](https://apify.com/kaankaan2635) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $5.00 / 1,000 publication scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper — Posts, Free vs Paid & Analytics

This **Substack scraper** exports publications and their full post archives, then computes the **Substack analytics** you cannot read off the site: how often a newsletter ships, what share of it sits behind the paywall, and what engagement that actually earns.

No account, no API key. It reads Substack's own public endpoints, so it does not break when the site changes its layout.

### Substack free vs paid: the ratio that matters

Any scraper can return a list of post titles. The question a newsletter operator actually has is **what does a working paid newsletter look like** — and that answer lives in the ratio between free and paywalled posts, the cadence behind it, and the engagement it earns.

This scraper computes that per publication:

| Publication | Paywalled | Avg words | Avg reactions | Posts/week |
| --- | --- | --- | --- | --- |
| Pirate Wires | 91% | 3,082 | 86 | 3.8 |
| The Pragmatic Engineer | 61% | 4,154 | 182 | 2.2 |
| ByteByteGo | 17% | 2,609 | 233 | 5.3 |
| Career Brew | 0% | 708 | 77 | 3.6 |

Four technology newsletters, four completely different paywall strategies. That comparison is the product.

**Cadence is reported with its measurement window.** Twelve posts from a daily publication span about a day, and extrapolating that to "57 posts per week" is arithmetically correct and practically useless. Every summary carries `cadenceWindowDays` next to `postsPerWeek` — raise **Maximum posts per publication** to widen it.

### Substack post scraper: what it exports

**Per publication:** `name` · `handle` · `subdomain` · `customDomain` · `publicationId` · `heroText` · `language` · `createdAt` · `hasPaidPlan` · `plans` (interval, amount, currency) · `freeSubscriptionBenefits` · `paidSubscriptionBenefits` · `foundingSubscriptionBenefits` · `communityEnabled` · `logoUrl`

**Per publication summary:** `postsAnalysed` · `firstPostDate` · `lastPostDate` · `postsPerWeek` · `cadenceWindowDays` · `freePosts` · `paywalledPosts` · `paywalledShare` · `avgWordcount` · `avgReactions` · `avgComments` · `avgRestacks` · `totalReactions` · `totalComments` · `podcastPosts` · `voiceoverPosts`

**Per post:** `title` · `subtitle` · `url` · `postDate` · `type` · `audience` · `isFree` / `isPaywalled` · `wordcount` · `reactionCount` · `commentCount` · `restacks` · `sectionName` · `hasPodcast` · `hasVoiceover` · `podcastDurationSec` · `language`

### How to use this Substack scraper

Two starting points, either or both:

- **Publications** — handles (`astralcodexten`), `*.substack.com` URLs, or custom domains (`www.thefp.com`). All three work; Substack serves the same data from a publication's own domain.
- **Category slugs** — `technology`, `business`, `finance`, `culture`, `us-politics`, `food` and more, plus subcategories like `artificial-intelligence-robotics`. The scraper walks the category's leading publications, paginating until it hits your limit.

```json
{
  "categorySlugs": ["technology"],
  "maxPublications": 50,
  "maxPostsPerPublication": 100,
  "postsOutput": "separate"
}
```

Set **Include posts** to false for publication profiles only — no archive requests, no post charges.

### FAQ

**Do I need a Substack account or API key?**
No. This reads Substack's public endpoints, the same data the site ships to any browser.

**Can it scrape paywalled post content?**
No. It reports *whether* a post is paywalled (`isPaywalled`, `audience`) and the metadata around it — title, date, word count, engagement — but never the body of a paid post. The paywall split is the product; the paid text is not.

**Can it export subscriber counts or email lists?**
No. Substack does not publish subscriber numbers and this scraper collects no email addresses at all.

**Does it work with custom domains?**
Yes. `www.thefp.com` and `bariweiss.substack.com` both resolve to the same publication.

**Can it scrape Substack comments?**
No. Substack's robots.txt disallows `/p/*/comment/*`, and comments are the one surface on a publication that is unambiguously personal. They are never fetched.

**How many posts can I get per publication?**
Up to 2,000. Posts load 30 per request going backwards through the archive.

**Is scraping Substack legal?**
This reads publicly published content without authentication, honours Substack's robots.txt, and collects no personal data. How you use the output is your responsibility.

### No personal data

Substack publications are businesses; the people who write them are not. This scraper collects publication identity and post metadata, and deliberately drops author names, author ids, bylines and email addresses. The test suite asserts it: a run that leaked a byline or an email address fails the build.

### Pricing

Pay per event: one charge per publication record and one per post record. No charge for compute time or failed requests.

### Development

```bash
npm install
npm test          # offline regression tests against saved fixtures
npm start         # local run; put an INPUT.json in storage/key_value_stores/default/
```

# Actor input Schema

## `publications` (type: `array`):

Handles, <code>\*.substack.com</code> URLs or custom domains — for example <code>astralcodexten</code>, <code>https://bariweiss.substack.com</code> or <code>www.thefp.com</code>. All three forms work.

## `categorySlugs` (type: `array`):

Scrape the leading publications in a category, for example <code>technology</code>, <code>business</code>, <code>finance</code>, <code>culture</code>, <code>us-politics</code>, <code>food</code>. Subcategory slugs such as <code>artificial-intelligence-robotics</code> also work. Combine with <b>Maximum publications</b>.

## `maxPublications` (type: `integer`):

Hard cap on how many publications this run scrapes.

## `includePosts` (type: `boolean`):

Fetch the post archive for each publication. Turning this off returns publication profiles only and skips all post charges.

## `maxPostsPerPublication` (type: `integer`):

Posts are fetched 50 per request. More posts widen the window the cadence and paywall figures are measured over, so 100+ gives a far more reliable picture than 10 for a publication that posts daily.

## `postsOutput` (type: `string`):

<b>Embedded</b> gives one row per publication with a <code>posts</code> array. <b>Separate</b> gives one row per post plus one profile row per publication, which loads straight into a spreadsheet.

## `maxConcurrency` (type: `integer`):

Lower this if requests start failing.

## `maxRequestsPerMinute` (type: `integer`):

Caps the request rate against Substack.

## `proxyConfiguration` (type: `object`):

Apify Proxy is recommended for larger runs.

## Actor input object example

```json
{
  "publications": [
    "astralcodexten",
    "bariweiss"
  ],
  "categorySlugs": [],
  "maxPublications": 25,
  "includePosts": true,
  "maxPostsPerPublication": 50,
  "postsOutput": "embedded",
  "maxConcurrency": 5,
  "maxRequestsPerMinute": 120,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `publications` (type: `string`):

One row per publication with its computed summary, or one row per post when separate output is selected.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "astralcodexten",
        "bariweiss"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("kaankaan2635/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publications": [
        "astralcodexten",
        "bariweiss",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("kaankaan2635/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "astralcodexten",
    "bariweiss"
  ]
}' |
apify call kaankaan2635/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,kaankaan2635/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/pfTNSA2E5K8vaye4F/builds/GV859OShvcM7qOKun/openapi.json