# Substack Scraper – Export Posts, Newsletter Archives & Search (`scrapeworkshq/substack-scraper`) Actor

Substack scraper & unofficial Substack API: export all posts from any newsletter archive, single posts or keyword search. Full text (HTML/Markdown), likes, comments, restacks, author & publication data. Export to JSON, CSV, Excel. No login.

- **URL**: https://apify.com/scrapeworkshq/substack-scraper.md
- **Developed by:** [Scrapeworks HQ](https://apify.com/scrapeworkshq) (community)
- **Categories:** Social media, News, AI
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per event

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper & API – Export All Posts, Newsletter Archives & Search (Markdown/CSV)

Scrape **public Substack data** fast and cheaply: full newsletter archives, individual posts, and **Substack-wide keyword search** — with post bodies as **HTML, Markdown or plain text**, engagement metrics (reactions, comments, restacks) and **publication metadata** (author, description, public subscriber-count text, sections). Works with `*.substack.com` and custom domains. No login, no browser, no proxy needed.

### 🚀 Quick start (60 seconds)

1. Click **Try for free** – the input is prefilled with a working example.
2. Press **Start**. A small run finishes in seconds and costs fractions of a cent.
3. Open the **Output** tab (table view) and export **CSV, Excel, JSON** or connect Google Sheets.

```json
{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 10,
  "bodyFormat": "markdown"
}
```

#### Sample output

| title | publicationName | publishedAt | reactionCount | commentCount | isPaywalled |
|---|---|---|---|---|---|
| How to tell if a startup has a moat | Lenny's Newsletter | 2026-10-13 | 350 | 2 | true |
| (free post) … | Astral Codex Ten | 2026-10-10 | 412 | 380 | false |

### Why this Substack scraper / Substack API alternative?

- **3 input types in one run** – publication URLs / bare subdomains (`astralcodexten`), post URLs, and search keywords. Results are de-duplicated across inputs.
- **Markdown & plain-text bodies** – ready for LLMs, RAG pipelines and NLP, not just raw HTML.
- **Paywall-safe** – paywalled posts are returned with metadata and `isPaywalled: true`; their body is never collected. Option to skip them entirely.
- **Privacy by design** – never collects subscriber emails, support emails or contact data. Subscribe forms are stripped from bodies.
- **Cost controls** – per-publication cap, per-keyword cap, global `maxPostsTotal`, date range, post-type filter. Metadata-only mode costs half.
- **Polite & reliable** – Substack's public JSON API with HTML fallback, rate limiting (`maxRequestsPerMinute`), retries with backoff on 429/5xx, low default concurrency.
- **Clean, flat, consistent schema** – same fields for every source, plus ready-made dataset views (Overview, Content, Publications).

### Use cases – export Substack posts, research & AI datasets

- **Content & competitor research** – track what top newsletters in your niche publish, how often, and what gets engagement.
- **Newsletter sponsorship research** – shortlist publications by topic and public subscriber-count tier.
- **AI / LLM datasets** – export free long-form posts as Markdown for RAG, summarization or fine-tuning (respect authors' rights).
- **Media monitoring** – schedule daily keyword searches ("climate policy", "OpenAI") and pipe new posts to Slack, Sheets or webhooks.
- **Academic & data journalism** – build archives of newsletter discourse with dates, tags and engagement.

### Input

| Field | Type | Default | Description |
|---|---|---|---|
| `publications` | string\[] | – | Newsletter URL, custom domain, or bare subdomain |
| `postUrls` | string\[] | – | Individual post URLs (`…/p/slug`) |
| `searchKeywords` | string\[] | – | Substack-wide post search |
| `maxPostsPerPublication` | int | 50 | `0` = full archive |
| `maxSearchResultsPerKeyword` | int | 20 | Up to 1000 |
| `maxPostsTotal` | int | 0 | Global cap (0 = none) |
| `dateFrom` / `dateTo` | date | – | `YYYY-MM-DD` |
| `postTypes` | enum\[] | all | `newsletter`, `podcast`, `thread` |
| `onlyFree` | bool | false | Skip paywalled posts |
| `bodyFormat` | enum | `none` | `none`, `html`, `markdown`, `text`, `all` |
| `includeReactions` | bool | true | Per-emoji reaction breakdown |
| `includePublicationMetadata` | bool | true | Nested `publication` object |
| `maxConcurrency` / `maxRequestsPerMinute` | int | 4 / 90 | Politeness |
| `proxyConfiguration` | object | off | Only if you hit 429s |

#### Example input

```json
{
  "publications": ["https://www.lennysnewsletter.com", "astralcodexten", "https://newsletter.pragmaticengineer.com"],
  "searchKeywords": ["artificial intelligence"],
  "maxPostsPerPublication": 20,
  "maxSearchResultsPerKeyword": 20,
  "dateFrom": "2025-01-01",
  "bodyFormat": "markdown"
}
```

### Output

One dataset item per post. Example (trimmed):

```json
{
  "postId": 219482066,
  "title": "How to tell if a startup has a moat",
  "subtitle": "A framework for assessing whether the company you're considering joining is built to last",
  "url": "https://www.lennysnewsletter.com/p/how-to-tell-if-a-startup-has-a-moat",
  "publishedAt": "2026-10-13T12:45:41.160Z",
  "type": "newsletter",
  "audience": "only_paid",
  "isPaywalled": true,
  "wordCount": 4153,
  "excerpt": "Hey there, I'm Lenny. Each week, I share deeply researched product, growth, and career advice...",
  "tags": ["Career"],
  "authors": [{ "name": "Dan Hockenmaier", "handle": "danhock", "bio": "Strategy, growth, and marketplaces.", "photoUrl": "https://substackcdn.com/..." }],
  "authorName": "Dan Hockenmaier",
  "reactionCount": 350,
  "commentCount": 2,
  "restacks": 18,
  "reactions": { "❤": 350 },
  "bodyMarkdown": null,
  "bodyNote": "Paywalled post: body not collected",
  "publicationName": "Lenny's Newsletter",
  "publication": {
    "name": "Lenny's Newsletter",
    "subdomain": "lenny",
    "customDomain": "www.lennysnewsletter.com",
    "description": "Deeply researched product, growth, and career advice for product leaders, founders, and ambitious builders.",
    "authorName": "Lenny Rachitsky",
    "subscriberCountText": "Millions of subscribers",
    "subscriberCountApprox": "1.2M+",
    "paidSubscriberText": "Tens of thousands of paid subscribers",
    "categories": ["..."],
    "paymentsEnabled": true
  },
  "source": "publication",
  "sourceInput": "https://www.lennysnewsletter.com",
  "scrapedAt": "2026-10-21T19:26:38.210Z"
}
```

Free posts additionally carry `bodyHtml` / `bodyMarkdown` / `bodyText` depending on `bodyFormat`. Download as JSON, CSV, Excel, XML or via API.

### ⭐ Unique: Monitor mode, CSV-ready output & webhook summary

Track a list of newsletters daily: `monitorMode: "onlyNew"` returns **only posts published since the last run** (old posts aren't output or charged). `"changes"` also re-emits posts whose reactions/comments/restacks changed – great for engagement tracking.

- **Monitor mode** (`monitorMode`: `all` | `changes` | `onlyNew`, plus `monitorKey`) – cross-run dedupe and field-level diffs (`changeType`, `changedFields`, `changes`, `firstSeenAt`). Skipped items cost **$0**.
- **Flat / CSV-ready output** (`outputFormat: "flat"`) – one column per field, perfect for Excel and Google Sheets.
- **Run summary** – a free `SUMMARY` record in the run's key-value store (also linked in the Output tab) and optionally POSTed to `summaryWebhookUrl`.
- **Changes** dataset view – see what's new/changed at a glance.

### 💰 Pricing (pay per result)

| Event | Price |
|---|---|
| Run start | $0.001 |
| Post (metadata) | **$0.75 / 1,000** |
| Post with full body (HTML/Markdown/text) | **$1.50 / 1,000** |

That's ≥25% below the typical Substack Actors in the Store ($1.00–$5.00 / 1,000). Platform usage is included.

🎁 **Sweetener:** in Monitor mode you only pay for new/changed items, and duplicates within a run are never charged. Run summary + webhook are free.

### API example

```js
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('YOUR_USERNAME/substack-scraper').call({
  publications: ['astralcodexten'], maxPostsPerPublication: 100, bodyFormat: 'markdown',
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

### FAQ

**Can it scrape paywalled posts?** No. It returns their public metadata (title, excerpt, word count, engagement) and marks `isPaywalled: true`. It never bypasses paywalls or uses logins.

**Does it collect emails or personal data?** No subscriber lists, emails or contact details are collected. Only public author bylines (name, handle, bio, photo) shown on each post.

**Custom domains?** Yes – any Substack-hosted domain. Sites that left Substack (e.g. moved to Ghost) are reported in the log and skipped.

**Subscriber counts?** Substack publicly shows only rounded text (e.g. "1.2M+", "Tens of thousands of paid subscribers"); we return exactly that when shown.

**Comments?** Comment and reply counts are included. Full comment text is intentionally not collected (commenter personal data).

**Is scraping Substack legal?** This actor reads only publicly available pages/APIs that Substack serves to any visitor. You are responsible for complying with Substack's Terms, copyright and data-protection laws (GDPR/CCPA) in how you use the data.

### 🔌 Integrations & API

Works with Zapier, Make, n8n, Google Sheets, Slack, Airbyte, LangChain/LlamaIndex and the Apify MCP server (AI agents).

**JavaScript**

```js
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('scrapeworkshq/substack-scraper').call({"publications": ["https://www.lennysnewsletter.com"], "maxPostsPerPublication": 10, "bodyFormat": "markdown"});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

**Python**

```python
from apify_client import ApifyClient
client = ApifyClient('<APIFY_TOKEN>')
run = client.actor('scrapeworkshq/substack-scraper').call(run_input={"publications": ["https://www.lennysnewsletter.com"], "maxPostsPerPublication": 10, "bodyFormat": "markdown"})
items = client.dataset(run['defaultDatasetId']).list_items().items
```

**cURL (sync, returns items)**

```bash
curl -X POST 'https://api.apify.com/v2/acts/scrapeworkshq~substack-scraper/run-sync-get-dataset-items?token=<APIFY_TOKEN>&format=csv' \
  -H 'content-type: application/json' -d '{"publications": ["https://www.lennysnewsletter.com"], "maxPostsPerPublication": 10, "bodyFormat": "markdown"}'
```

### ❓ More FAQ

**Does it get paywalled content?**\
No. Paywalled posts are returned with metadata and `isPaywalled: true`, never their body.

**Is Markdown good for LLMs/RAG?**\
Yes – choose *Markdown* as Post body; it is clean, heading-preserving text ready for chunking.

**How does Monitor mode work?**\
The Actor remembers the IDs and key fields of items it returned before (in a named key-value store in *your* account, keyed by `monitorKey`). Next run, each item gets `changeType` = `new` / `changed` / `unchanged` plus a `changes` diff. In *Only new* / *Only new or changed* modes, the rest is skipped **and not charged**.

**How do I get a clean CSV / Excel / Google Sheet?**\
Set **Output format → Flat / CSV-ready**. Nested objects become `dot.columns` and lists are joined with `|`, so every export is one row per item with no JSON blobs.

**Can I get notified when a run finishes?**\
Set **Summary webhook URL** (Zapier/Make/n8n catch hook, Slack workflow, your API). You receive one compact JSON: counts, new/changed, price drops/rises, dataset ID. Or use Apify's native webhooks/integrations.

**How do I limit cost?**\
Use the max-results inputs and Apify's *Maximum cost per run*. The Actor stops cleanly when the limit is reached.

Found a bug or need a field? Open an issue on the **Issues** tab – we usually reply within 24 h.

### Changelog

- **2026-10-21** – Monitor mode (new/changed only, free skips), flat CSV-ready output, SUMMARY record + webhook, Changes view, lower prices.
- 1.0 – Initial release: archives, post URLs, keyword search, HTML/Markdown/text bodies, publication metadata, pay-per-event.

# Actor input Schema

## `publications` (type: `array`):

Newsletter URLs, custom domains or bare subdomains, e.g. https://www.lennysnewsletter.com, astralcodexten, platformer.news. Post URLs here also work.

## `postUrls` (type: `array`):

Individual post URLs (…/p/slug).

## `searchKeywords` (type: `array`):

Search all public Substack posts by keyword (Substack's own search).

## `maxPostsPerPublication` (type: `integer`):

0 = full archive.

## `maxSearchResultsPerKeyword` (type: `integer`):

Posts per keyword.

## `maxPostsTotal` (type: `integer`):

Hard cap across the whole run (cost control). 0 = no cap.

## `dateFrom` (type: `string`):

Only posts on/after this date (YYYY-MM-DD).

## `dateTo` (type: `string`):

Only posts on/before this date (YYYY-MM-DD).

## `postTypes` (type: `array`):

Empty = all types.

## `onlyFree` (type: `boolean`):

Skip paywalled posts entirely (otherwise they are returned with metadata and isPaywalled=true).

## `bodyFormat` (type: `string`):

Full body of free posts. Paywalled bodies are never collected.

## `includeReactions` (type: `boolean`):

Per-emoji reaction counts (reactionCount, commentCount and restacks are always included).

## `includePublicationMetadata` (type: `boolean`):

Name, author, description, public subscriber-count text, sections/categories, logo. One extra request per publication.

## `maxConcurrency` (type: `integer`):

Parallel requests. Keep low to be polite.

## `maxRequestsPerMinute` (type: `integer`):

Global rate limit.

## `proxyConfiguration` (type: `object`):

Usually not needed. Enable if you hit rate limits (HTTP 429).

## `monitorMode` (type: `string`):

Remembers items from previous runs (per Monitor key) and adds changeType = new/changed/unchanged plus a field-level diff. In the two 'only' modes, unchanged/old items are not output and not charged.

## `monitorKey` (type: `string`):

Name of the memory used by Monitor mode. Use a different key per schedule/task (e.g. 'daily-pokemon'), so separate watchlists don't mix.

## `outputFormat` (type: `string`):

Flat makes CSV/Excel/Google Sheets exports one clean column per field.

## `summaryWebhookUrl` (type: `string`):

At the end of the run, POST a compact JSON summary (counts, new/changed, price drops, dataset ID) to this URL – e.g. Zapier/Make/n8n catch hook or Slack workflow. Free.

## Actor input object example

```json
{
  "publications": [
    "https://www.lennysnewsletter.com",
    "astralcodexten"
  ],
  "postUrls": [],
  "searchKeywords": [
    "artificial intelligence"
  ],
  "maxPostsPerPublication": 10,
  "maxSearchResultsPerKeyword": 20,
  "maxPostsTotal": 0,
  "onlyFree": false,
  "bodyFormat": "none",
  "includeReactions": true,
  "includePublicationMetadata": true,
  "maxConcurrency": 4,
  "maxRequestsPerMinute": 90,
  "proxyConfiguration": {
    "useApifyProxy": false
  },
  "monitorMode": "off",
  "monitorKey": "default",
  "outputFormat": "full"
}
```

# Actor output Schema

## `results` (type: `string`):

All scraped items in the default dataset

## `overview` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com",
        "astralcodexten"
    ],
    "postUrls": [],
    "searchKeywords": [
        "artificial intelligence"
    ],
    "maxPostsPerPublication": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("scrapeworkshq/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": [
        "https://www.lennysnewsletter.com",
        "astralcodexten",
    ],
    "postUrls": [],
    "searchKeywords": ["artificial intelligence"],
    "maxPostsPerPublication": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("scrapeworkshq/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com",
    "astralcodexten"
  ],
  "postUrls": [],
  "searchKeywords": [
    "artificial intelligence"
  ],
  "maxPostsPerPublication": 10
}' |
apify call scrapeworkshq/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapeworkshq/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lowTqqK8BhsrKzMzf/builds/bZ6naiU3arWB1dTRA/openapi.json