# 📰 Newsletter Scraper - Substack Archives, LLM-Ready (`benthepythondev/newsletter-scraper`) Actor

Extract newsletter archives from Substack, Beehiiv, and Ghost platforms. Get full content in markdown format, complete metadata, embedded images, word counts, and AI-ready token counts. Perfect for content research, competitive analysis, and training AI models.

- **URL**: https://apify.com/benthepythondev/newsletter-scraper.md
- **Developed by:** [Ben](https://apify.com/benthepythondev) (community)
- **Categories:** AI, Developer tools, News
- **Stats:** 60 total users, 2 monthly users, 100.0% runs succeeded, 1 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.80 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## 📰 Newsletter Scraper — Substack Archives as Clean, LLM-Ready Data

Extract **full newsletter content from Substack** — entire archives, individual
posts and publication metadata — as clean, structured data with Markdown, HTML and
plain-text versions plus word and token counts. It's the content layer you need for
AI/LLM training sets, competitive research and content archiving, captured straight
from the public web with no copy-pasting. Export to JSON/CSV/Excel, run on a
schedule, call via API, or connect to Make, Zapier or n8n.

### 📚 What is the Newsletter Scraper?

It turns any Substack newsletter into a structured dataset. Give it a newsletter
homepage URL (or a single post URL) and choose a mode — full archive, single post,
or metadata only — and it returns each post with its full content in your chosen
format, engagement metrics, images and a built-in token count for LLM cost
planning. Beehiiv and Ghost detection is built in, with those platform scrapers in
active development; Substack (including custom domains) is fully supported today.

#### What data does it extract?

- **Title, subtitle and author** plus the post **URL**
- **Published date** and **scraped_at** timestamp
- **Content in three formats** — `content_markdown`, `content_html`, `content_text`
- **Word count** and **LLM token count** (`word_count`, `token_count`)
- **Images** array with `url`, `alt_text`, `width`, `height`, `caption`
- **Engagement metadata** — likes, comments, shares, views (when available)
- **Premium / paywall flag** (`is_premium`), `tags`, `category` and `language`

### ⬇️ Input

Pick a mode, point it at a newsletter, and choose your content format:

| Field | Description |
|-------|-------------|
| `newsletterUrl` | Newsletter homepage, e.g. `https://platformer.substack.com` |
| `scrapeMode` | `archive` (all posts), `single-post`, or `newsletter` (metadata only) |
| `postUrl` | URL of a specific post (used in `single-post` mode) |
| `maxPosts` | Maximum posts to scrape (`0` = unlimited) |
| `outputFormat` | `markdown`, `html`, `text`, or `all` |
| `includeImages` | Extract image URLs and details (default on) |
| `includeMetadata` | Extract engagement metrics — likes, comments, shares |
| `sinceDate` | Only scrape posts published after `YYYY-MM-DD` |
| `delaySeconds` | Polite delay between requests (default 1.0) |

#### Example input

```json
{
  "newsletterUrl": "https://platformer.substack.com",
  "scrapeMode": "archive",
  "outputFormat": "markdown",
  "maxPosts": 50,
  "includeImages": true,
  "includeMetadata": true,
  "delaySeconds": 1.0
}
```

### ⬆️ Output

Every post is one clean row (view as a **table**, or export **JSON / CSV / Excel**):

```json
{
  "title": "How to get the most out of your 1-on-1s",
  "subtitle": "A tactical guide for managers and ICs",
  "url": "https://platformer.substack.com/p/how-to-get-the-most-out-of-your",
  "author": "Casey Newton",
  "published_date": "2026-01-15T10:00:00",
  "content_markdown": "# How to get the most out of your 1-on-1s\n\nA tactical guide...",
  "content_html": "<article><h1>How to get the most out...</h1></article>",
  "content_text": "How to get the most out of your 1-on-1s...",
  "word_count": 2450,
  "token_count": 3267,
  "images": [
    {
      "url": "https://cdn.substack.com/image/example.jpg",
      "alt_text": "Meeting room illustration",
      "width": 800,
      "height": 600,
      "caption": null
    }
  ],
  "metadata": { "likes": 245, "comments": 18, "shares": 8, "views": null },
  "is_premium": false,
  "tags": ["management", "career"],
  "category": null,
  "language": "en",
  "scraped_at": "2026-06-27T18:00:00"
}
```

In `single-post` mode you get one post; in `newsletter` mode you get publication metadata (name, description, platform, author, subscriber and post counts when available).

### 💡 Use cases

- **🤖 AI & LLM training:** build domain-specific training sets and RAG corpora in clean Markdown, with token counts ready for cost planning.
- **📊 Content research:** track topics, posting cadence and writing style across newsletters over time.
- **🔍 Competitive intelligence:** monitor competitor newsletters and benchmark engagement (likes, comments, shares).
- **🗄️ Content archiving:** back up your own newsletter or preserve historical posts in a searchable, portable format.

### ❓ FAQ

**How do I scrape a newsletter?** Paste the newsletter homepage URL, pick a
`scrapeMode` (`archive`, `single-post` or `newsletter`), choose an `outputFormat`,
then Run. You get structured posts with full content, metrics and images.

**Do I need an API key or login?** No. It reads publicly visible newsletter posts —
just paste the URL. Paid/paywalled posts are detected and flagged, but full paid
content (which requires authentication) is not retrieved.

**Which platforms are supported?** Substack is fully supported today, including
custom domains. Beehiiv and Ghost are auto-detected with their scrapers in active
development.

**What content formats can I get?** Markdown (LLM-optimized), HTML (original), plain
text, or `all` three at once via `outputFormat`.

**How accurate is the token count?** It's an estimate using a standard GPT-style
ratio (roughly 1 token ≈ 0.75 words) — great for planning LLM costs. For exact
counts, run a tokenizer on the output text.

**Can I get only recent posts?** Yes — set `sinceDate` (`YYYY-MM-DD`) to keep only
posts published after that date, and use `maxPosts` to cap the run.

**Does it capture engagement metrics?** Yes — with `includeMetadata` on, likes,
comments, shares and views are captured when the platform exposes them.

**Why did I only get a limited number of posts?** Substack archive pages surface a
limited set of posts by default — that's a platform behavior, not a scraper limit.

**Can I run it on a schedule or via API?** Yes — schedule recurring runs in Apify,
call it via the API/SDK, or connect it to Make, Zapier or n8n.

**Is scraping newsletters legal?** It extracts publicly available content — the same
posts anyone can read in a browser. Respect copyright, don't republish without
permission, follow the newsletter's terms, and use the data responsibly.

### 🔗 You might also like

- **[Spotify Podcast Scraper](https://apify.com/benthepythondev/spotify-podcast-scraper)** — podcast episodes & metadata
- **[Podcast Intelligence Aggregator](https://apify.com/benthepythondev/podcast-intelligence-aggregator)** — multi-source podcast data
- **[Dev.to Articles Intelligence](https://apify.com/benthepythondev/devto-articles-intelligence)** — developer articles
- **[YouTube Transcript Scraper](https://apify.com/benthepythondev/youtube-transcript-scraper)** — video transcripts for LLMs

***

**Keywords:** newsletter scraper, Substack scraper, newsletter archive, LLM training data, Markdown content extractor, content research, competitive intelligence, RAG dataset, token count, email newsletter data, content scraping API, AI training dataset, newsletter content extractor, Substack archive scraper.

# Actor input Schema

## `newsletterUrl` (type: `string`):

The URL of the newsletter homepage (e.g., https://example.substack.com)

## `scrapeMode` (type: `string`):

What to scrape from the newsletter

## `postUrl` (type: `string`):

URL of a specific post to scrape (only used in single-post mode)

## `maxPosts` (type: `integer`):

Maximum number of posts to scrape (0 = unlimited). Useful for testing or limiting archive scrapes.

## `outputFormat` (type: `string`):

Format for post content output

## `includeImages` (type: `boolean`):

Extract image URLs from posts

## `includeMetadata` (type: `boolean`):

Extract likes, comments, shares (if available)

## `sinceDate` (type: `string`):

Only scrape posts published after this date (YYYY-MM-DD). Leave empty for all posts.

## `delaySeconds` (type: `number`):

Delay in seconds between scraping each post (to be respectful to servers)

## `debugMode` (type: `boolean`):

Enable verbose logging for troubleshooting

## Actor input object example

```json
{
  "newsletterUrl": "https://platformer.substack.com",
  "scrapeMode": "archive",
  "postUrl": "https://example.substack.com/p/some-post",
  "maxPosts": 3,
  "outputFormat": "markdown",
  "includeImages": true,
  "includeMetadata": true,
  "sinceDate": "2024-01-01",
  "delaySeconds": 1,
  "debugMode": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "newsletterUrl": "https://platformer.substack.com",
    "maxPosts": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("benthepythondev/newsletter-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "newsletterUrl": "https://platformer.substack.com",
    "maxPosts": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("benthepythondev/newsletter-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "newsletterUrl": "https://platformer.substack.com",
  "maxPosts": 3
}' |
apify call benthepythondev/newsletter-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,benthepythondev/newsletter-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/dMzfivEaaPSyvCoGD/builds/vi8rNGW4uIfCYuX7S/openapi.json