# Substack Scraper — Posts, Paywall & Bestseller Signals (`viralanalyzer/substack-newsletter-intelligence`) Actor

Extract posts, paywall status, engagement and bestseller signals from any public Substack publication. Pure HTTP via the public archive API — no login, no API key, no browser, no residential proxy. See which competitors monetize and how much sits behind the paywall.

- **URL**: https://apify.com/viralanalyzer/substack-newsletter-intelligence.md
- **Developed by:** [viralanalyzer](https://apify.com/viralanalyzer) (community)
- **Categories:** Social media, News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.31 / 1,000 post scrapeds

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Substack Scraper — Posts, Paywall & Bestseller Signals

Extract posts, **paywall status**, engagement and **bestseller signals** from any public Substack
publication. Pure HTTP against the public archive API — **no login, no API key, no headless browser,
no residential proxy**.

Most Substack scrapers tell you *what* was published. This one tells you **what is behind the paywall**
— which is the question that decides whether a newsletter is a business or a hobby.

> 🇧🇷 **Versão em português no final deste README.**

***

### Features

- **Paywall intelligence** — every post carries `audience` (`everyone` / `only_paid`) and a derived
  `isPaywalled`, plus a per-publication `paywallRatio`.
- **Monetization signals** — `paymentsEnabled`, `foundingPlanName` and the author's `bestsellerTier`
  straight from the source.
- **Engagement** — reactions, restacks, comments and word count per post.
- **Works on custom domains** — pass `bigtechnology` or `https://www.bigtechnology.com`; both resolve.
- **Audience filter** — keep only free posts, only paywalled posts, or everything.
- **Publication summary** — one extra row per publication with paywall ratio, average reactions,
  average word count and top bestseller tier.
- **Honest failures** — when the source blocks us or answers with an empty archive, you get a
  **labeled diagnostic and you are not charged**. Schema drift still fails loudly instead of
  returning silence.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `publications` | array | — | Handles (`bigtechnology`) or URLs (`https://bigtechnology.substack.com`, custom domains work). 1–50 items. |
| `maxPostsPerPublication` | integer | `50` | Max posts per publication (1–1000). Paginated in pages of 50. |
| `audienceFilter` | select | `all` | `all` · `free` · `paid` |
| `includePublicationSummary` | boolean | `true` | Adds one summary row per publication. |
| `proxyConfiguration` | object | datacenter | Optional. The archive API is public — **residential is not needed**. |

### Output

Two item types share one dataset, distinguished by `itemType`.

**`itemType: "post"`**

`publication` · `publicationName` · `customDomain` · `postId` · `title` · `subtitle` · `slug` · `url` ·
`postDate` · `postType` · **`audience`** · **`isPaywalled`** · `wordCount` · `reactionCount` ·
`restacks` · `commentCount` · `language` · `coverImage` · `description` · `hasAudio` ·
`podcastDuration` · `authors[]` (name, handle, bio, **bestsellerTier**, twitter) ·
`paymentsEnabled` · `foundingPlanName` · `scrapedAt`

**`itemType: "publication_summary"`**

`totalPosts` · `paidPosts` · **`paywallRatio`** · `avgReactions` · `avgWordCount` ·
`maxBestsellerTier` · `paymentsEnabled` · `foundingPlanName`

### Use cases

- **Competitive newsletter analysis** — how much of a rival's output is paywalled, and what the free
  posts look like.
- **Creator economy research** — find publications with payments enabled and a high bestseller tier.
- **Content strategy** — correlate word count and format with reactions and restacks.
- **Partnership and sponsorship sourcing** — rank publications by engagement, not by follower vanity.
- **Media monitoring** — track a set of newsletters on a schedule and diff the archive.

### FAQ

**Do I need a Substack account or API key?** No. The archive endpoint is public.

**Does it read paywalled article text?** No. It reports **that** a post is paywalled and its metadata.
Paid body text is not extracted — that content is not public.

**Does it work with custom domains?** Yes. `https://www.bigtechnology.com` resolves the same as
`bigtechnology.substack.com`.

**Do I need a residential proxy?** No. That is exactly what makes this Actor cheap to run.

**What happens when a publication does not exist?** You get a `SUCCEEDED` run with one diagnostic
item explaining why, and **no charge**.

**How fast is it?** One HTTP request returns up to 50 posts. No browser to boot.

### Pricing

Pay-per-event on posts actually delivered. Diagnostic and summary rows are **not** charged.
Platform usage (compute) is billed to your account at Apify cost — this Actor is pure HTTP, so it is
a fraction of a cent per run. See the pricing panel on this page for the current rate and tiers.

### Related Actors

- `viralanalyzer/telegram-channel-scraper` — the same pure-HTTP approach for Telegram channels
- `viralanalyzer/rss-news-intelligence` — feed monitoring at scale
- `viralanalyzer/linkedin-ads-library` — B2B ad intelligence

### Changelog

**1.0** — First release. Archive API pagination, audience filter, publication summary,
block-gated diagnostics, owner-skip PPE.

***

***

## 🇧🇷 Substack Scraper — Posts, Paywall e Sinais de Bestseller

Extraia posts, **status de paywall**, engajamento e **sinais de bestseller** de qualquer publicação
pública do Substack. HTTP puro contra a API pública de arquivo — **sem login, sem API key, sem
navegador headless, sem proxy residencial**.

A maioria dos scrapers de Substack diz *o que* foi publicado. Este diz **o que está atrás do
paywall** — que é a pergunta que separa uma newsletter que é negócio de uma que é hobby.

### Funcionalidades

- **Inteligência de paywall** — cada post traz `audience` (`everyone` / `only_paid`) e o campo
  derivado `isPaywalled`, mais o `paywallRatio` por publicação.
- **Sinais de monetização** — `paymentsEnabled`, `foundingPlanName` e o `bestsellerTier` do autor,
  direto da fonte.
- **Engajamento** — reações, restacks, comentários e contagem de palavras por post.
- **Funciona com domínio próprio** — aceita `bigtechnology` ou `https://www.bigtechnology.com`.
- **Filtro de audiência** — só posts gratuitos, só pagos, ou tudo.
- **Resumo por publicação** — uma linha extra com razão de paywall, média de reações e de palavras.
- **Falha honesta** — se a fonte bloquear ou devolver arquivo vazio, você recebe um **diagnóstico
  rotulado e não é cobrado**. Mudança de schema continua falhando alto, em vez de devolver silêncio.

### Entrada

| Campo | Tipo | Padrão | Descrição |
|---|---|---|---|
| `publications` | array | — | Handles (`bigtechnology`) ou URLs. 1–50 itens. |
| `maxPostsPerPublication` | inteiro | `50` | Máximo de posts por publicação (1–1000). |
| `audienceFilter` | seleção | `all` | `all` · `free` · `paid` |
| `includePublicationSummary` | booleano | `true` | Adiciona uma linha de resumo por publicação. |
| `proxyConfiguration` | objeto | datacenter | Opcional. A API é pública — **residencial não é necessário**. |

### Saída

Dois tipos de item no mesmo dataset, distinguidos por `itemType`: **`post`** e
**`publication_summary`** (campos listados na seção em inglês acima).

### Casos de uso

- **Análise competitiva de newsletters** — quanto da produção do concorrente está no paywall.
- **Pesquisa de creator economy** — achar publicações com pagamentos ativos e bestseller alto.
- **Estratégia de conteúdo** — correlacionar tamanho e formato com reações e restacks.
- **Prospecção de patrocínio** — ranquear por engajamento real, não por vaidade de seguidores.
- **Monitoramento** — acompanhar um conjunto de newsletters em agendamento e comparar o arquivo.

### FAQ

**Preciso de conta ou API key do Substack?** Não. O endpoint de arquivo é público.

**Ele lê o texto de posts pagos?** Não. Ele informa **que** o post é pago e os metadados dele. O
conteúdo pago não é público e não é extraído.

**Funciona com domínio próprio?** Sim.

**Preciso de proxy residencial?** Não — e é justamente isso que torna este Actor barato de rodar.

**E se a publicação não existir?** O run termina `SUCCEEDED` com um item de diagnóstico explicando o
motivo, e **sem cobrança**.

### Preço

Cobrança por evento, sobre posts efetivamente entregues. Itens de diagnóstico e de resumo **não** são
cobrados. O custo de plataforma é debitado na sua conta ao custo da Apify — como é HTTP puro, é uma
fração de centavo por run. Veja o painel de preço nesta página para a tarifa e os tiers atuais.

### Changelog

**1.0** — Primeira versão. Paginação da API de arquivo, filtro de audiência, resumo por publicação,
diagnóstico com gate de bloqueio, owner-skip no PPE.

# Actor input Schema

## `publications` (type: `array`):

Substack publication handles (e.g. 'bigtechnology') or full URLs (e.g. 'https://bigtechnology.substack.com' or a custom domain like 'https://www.bigtechnology.com'). Only public publications are supported.

## `maxPostsPerPublication` (type: `integer`):

Maximum number of posts to extract per publication. The archive API is paginated in pages of up to 50.

## `audienceFilter` (type: `string`):

Keep only free posts, only paywalled posts, or everything. Useful to measure how much of a competitor's output sits behind the paywall.

## `includePublicationSummary` (type: `boolean`):

Push one extra summary item per publication with paywall ratio, average engagement, average word count and bestseller tier.

## `proxyConfiguration` (type: `object`):

Optional. The Substack archive API is public and works on a direct connection — datacenter proxy is only useful to spread requests across IPs on large runs. Residential is NOT needed and only adds cost.

## Actor input object example

```json
{
  "publications": [
    "bigtechnology"
  ],
  "maxPostsPerPublication": 5,
  "audienceFilter": "all",
  "includePublicationSummary": true,
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

Dataset containing all scraped posts and publication summaries. Each item follows the dataset schema.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "bigtechnology"
    ],
    "maxPostsPerPublication": 5,
    "audienceFilter": "all",
    "proxyConfiguration": {
        "useApifyProxy": true
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("viralanalyzer/substack-newsletter-intelligence").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["bigtechnology"],
    "maxPostsPerPublication": 5,
    "audienceFilter": "all",
    "proxyConfiguration": { "useApifyProxy": True },
}

# Run the Actor and wait for it to finish
run = client.actor("viralanalyzer/substack-newsletter-intelligence").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"])
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "bigtechnology"
  ],
  "maxPostsPerPublication": 5,
  "audienceFilter": "all",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}' |
apify call viralanalyzer/substack-newsletter-intelligence --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "command": "npx",
            "args": [
                "mcp-remote",
                "https://mcp.apify.com/?tools=viralanalyzer/substack-newsletter-intelligence",
                "--header",
                "Authorization: Bearer <YOUR_API_TOKEN>"
            ]
        }
    }
}

```

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/u2LOQtZiwGC2mvDEf/builds/mQ4o30UOURemjq2y4/openapi.json