# Substack Scraper: Posts, Newsletters, Notes & Leaderboards (`tenlakessoftware/substack-scraper`) Actor

Scrape Substack newsletters, posts, comments, Notes and leaderboards. Find newsletters by keyword with subscriber counts, paid prices, posting cadence, sponsors and public contact emails. Track leaderboard growth. No login needed; pay only for results.

- **URL**: https://apify.com/tenlakessoftware/substack-scraper.md
- **Developed by:** [Ten Lakes Software](https://apify.com/tenlakessoftware) (community)
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What is Substack Scraper?

**Substack Scraper** extracts public data from Substack newsletters: **posts, newsletters, subscriber counts, paid plan prices, sponsors, comments, Notes, author profiles, recommendations and category leaderboards**. Search Substack by keyword, enrich every newsletter with business intelligence, or pull a publication's full post archive, all without a Substack account or API key.

It's built for people who use Substack data to make money decisions: **newsletter ad buyers, sponsorship and influencer agencies, marketers building outreach lists, investors tracking the creator economy, and AI teams that need clean newsletter text**.

Try it now: click **Try for free**, keep the default input (search for "AI" newsletters) and press **Start**. You'll have about 100 newsletters with subscriber counts in under a minute.

- 🔎 **Find newsletters by keyword** with public subscriber counts, follower counts and bestseller tier
- 💰 **Newsletter intelligence**: monthly and yearly paid prices, posting cadence, active or dormant status, paid-post share
- 🤝 **Sponsor detection**: who sponsors which newsletter, with brand, domain, link and placement
- 📈 **Leaderboard growth tracking**: rank and subscriber changes between scheduled runs
- 📬 **Public business contacts**: emails a newsletter publishes itself, with the source of each one
- 📝 **Posts, comments, Notes, profiles and recommendations**, with only-new mode for scheduled runs
- ✅ **Compliance-safe by default**: no paywall bypass, no commenter identities unless you turn them on

### What Substack data can you extract?

| Mode | What you get |
|---|---|
| **Search** | Newsletters (or posts) matching your keywords, with subscriber count, follower count, bestseller tier, author and description |
| **Publication intelligence** | Monthly and yearly price, currency, posts in the last 90 days, average days between posts, last post date, active flag, paid-post share, recent sponsors, "takes sponsors" signal, advertise page, X/Twitter, optional public contact emails |
| **Posts** | Title, subtitle, URL, date, authors, audience (free or paid), paywall flag, word count, likes, comments, restacks, cover image, podcast info, optional full text, sponsors and comments |
| **Leaderboards** | Rank, category, paid or overall board, subscriber band, price, logo, plus rank change, subscriber change and new entries since your last run |
| **Comments** | Comment text, date, likes, replies, reply depth, author-reply flag, and the commenter's own newsletter (optional) |
| **Notes** | Note text, date, likes, restacks, replies, and attachments such as recommended newsletters and shared links |
| **Profiles** | Name, handle, bio, subscriber and follower counts, bestseller tier and the newsletters they write |
| **Recommendations** | Which newsletters recommend which, so you can map a niche or find cross-promotion partners |

Results open in tidy tables in Apify Console, with separate views for **Newsletters, Contacts, Sponsors, Posts, Leaderboard, Comments, Notes, Profiles and Recommendations**. Export to **JSON, CSV, Excel, XML or HTML**, or fetch them through the API.

### Who uses this Substack scraper?

#### Newsletter ad buyers and brands

See which companies sponsor which newsletters, how often, and where the ad sits in the post. Find newsletters in your niche that already take sponsors, then check their size, price and posting rhythm before you reach out.

#### Sponsorship and influencer agencies

Build prospect lists of newsletters by topic with subscriber counts, prices, X/Twitter handles and the contact emails they publish. Track which brands are spending in your clients' categories.

#### Marketers and growth teams

Find partners for cross-promotion through the recommendations graph, monitor competitors' new posts, and see what topics get the most likes and comments.

#### Investors and analysts

Schedule the leaderboard mode weekly and get a feed of **which newsletters are climbing fastest** in each category, with rank and subscriber changes calculated for you.

#### AI and research teams

Pull clean full text of public posts for analysis, summarization or retrieval. Turn on **only new since last run** so scheduled runs never send you the same post twice.

### How to scrape Substack

1. Click **Try for free** and sign in to Apify (a free account comes with monthly credit).
2. Under **What to scrape**, pick a mode: Search, Publication intelligence, Posts, Leaderboard, Recommendations, Profiles or Notes.
3. Fill in what you're looking for: keywords, newsletter URLs (for example `lenny` or `https://www.lennysnewsletter.com`), leaderboard categories or author handles.
4. Turn on extras if you want them: full post text, sponsor detection, comments, publication intelligence or contact emails.
5. Optionally set **Max cost per run** so the run stops at your budget.
6. Press **Start**. When the run finishes, open the **Output** tab and pick the view you want, or export the data.
7. To keep the data fresh, **schedule** the run (daily or weekly) and turn on **Only new since last run** or **Track rank & subscriber growth**.

Custom domains work too: `www.lennysnewsletter.com` and `lenny.substack.com` are treated as the same newsletter.

### How much does it cost to scrape Substack?

Substack Scraper uses **pay-per-event** pricing. You pay only for the data you get, and **platform usage is included**: there's no separate compute bill.

| Event | Price |
|---|---|
| Run start | $0.005 (at the default 2 GB memory) |
| Result: post, newsletter, leaderboard row, profile, Note or recommendation | $0.001 ($1.00 per 1,000) |
| Comment | $0.0005 ($0.50 per 1,000) |
| Publication intelligence (one fully enriched newsletter) | $0.005 |
| Sponsor mention found | $0.01 |
| Public contact email found | $0.02 |
| Leaderboard growth change detected | $0.01 |

#### Real-world cost examples

| Job | Roughly costs |
|---|---|
| Last 100 posts from one newsletter | about $0.11 |
| Top 100 of one leaderboard category | about $0.11 |
| 500 comments from a popular post | about $0.26 |
| 100 "AI" newsletters with full intelligence | about $0.51, plus $0.01 per sponsor and $0.02 per email found |
| Weekly growth report for 3 categories | about $0.30, plus $0.01 per newsletter that moved |

Sponsor and email charges apply **only when something is actually found**. A lookup that finds nothing is free. Set **Max cost per run** and the scraper stops cleanly at your limit, so you never pay more than you planned.

### Input example

Find AI newsletters that take sponsors, with prices and public contacts:

```json
{
  "mode": "search",
  "keywords": ["AI", "machine learning"],
  "searchType": "publications",
  "enrichPublications": true,
  "includeContactEmails": true,
  "maxItemsPerSource": 50
}
```

Weekly growth tracker for two leaderboard categories (schedule it):

```json
{
  "mode": "leaderboard",
  "categories": ["Technology", "Business"],
  "leaderboardType": "paid",
  "trackGrowth": true
}
```

New posts with sponsors and comments from newsletters you follow:

```json
{
  "mode": "posts",
  "publications": ["https://newsletter.pragmaticengineer.com", "lenny"],
  "detectSponsors": true,
  "includeComments": true,
  "onlyNew": true
}
```

### Output example

Publication intelligence for one newsletter (real output, shortened):

```json
{
  "type": "publication",
  "name": "The Pragmatic Engineer",
  "url": "https://newsletter.pragmaticengineer.com",
  "subscriberCountLabel": "1.1M+",
  "monthlyPrice": 15,
  "yearlyPrice": 150,
  "priceCurrency": "USD",
  "postsLast90Days": 12,
  "avgDaysBetweenPosts": 2.5,
  "isActive": true,
  "paidPostShare": 0.58,
  "acceptsSponsors": true,
  "recentSponsors": ["turbopuffer.com", "workos.com", "linear.app"],
  "twitter": "https://x.com/GergelyOrosz"
}
```

A detected sponsor:

```json
{
  "type": "sponsor",
  "sponsorName": "WorkOS",
  "sponsorDomain": "workos.com",
  "placement": "top",
  "confidence": "high",
  "cue": "Brought to You by",
  "postUrl": "https://newsletter.pragmaticengineer.com/p/..."
}
```

A leaderboard row with growth tracking (illustrative values):

```json
{
  "type": "leaderboard",
  "rank": 4,
  "previousRank": 9,
  "rankChange": 5,
  "name": "Example Newsletter",
  "categoryName": "Technology",
  "subscriberCount": 120000,
  "subscriberChange": 3000,
  "isNewEntry": false
}
```

Every row has a `type` field, so you can filter one export into posts, newsletters, sponsors and comments in a spreadsheet.

### How does sponsor detection work?

The scraper reads each post and looks for sponsorship cues such as "Brought to you by", "Sponsored by", "Today's sponsor" and "In partnership with". It then takes the brand and link that follow the cue. Each sponsor becomes its own row with the brand name, website domain, link, **placement** (top, middle or bottom of the post), a **confidence** level and the surrounding text, so you can check any match in seconds.

In publication intelligence mode, it scans a newsletter's most recent posts (6 by default) and summarizes them as `recentSponsors` and an `acceptsSponsors` flag. That's the fastest way to shortlist newsletters that already sell ads.

### Track Substack leaderboard growth

Substack's category leaderboards show who's on top today, but not who's rising. Turn on **Track rank & subscriber growth** and schedule the run. On every run after the first, each row includes `previousRank`, `rankChange`, `subscriberChange` and `isNewEntry`, and each real change is reported as a growth event. You get a ready-made "fastest growing newsletters" feed for any category, paid or overall.

### Use Substack Scraper with the API

Run it from your own code with the Apify API client.

**JavaScript / Node.js**

```javascript
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('tenlakessoftware/substack-scraper').call({
    mode: 'search',
    keywords: ['personal finance'],
    enrichPublications: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);
```

**Python**

```python
from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("tenlakessoftware/substack-scraper").call(run_input={
    "mode": "posts",
    "publications": ["lenny"],
    "maxItemsPerSource": 50,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["url"])
```

You can also call it over plain HTTP with the `run-sync-get-dataset-items` endpoint and get the results back in a single request. See the **API** tab on this page for every option.

### Use with AI agents (MCP)

AI assistants such as **Claude, ChatGPT and Cursor** can run Substack Scraper through the **Apify MCP server**. Connect your assistant to Apify, then ask in plain language, for example:

- "Find 30 newsletters about climate tech that take sponsors and list their prices."
- "Which Technology newsletters gained the most rank this week?"
- "Pull the last 20 posts from Lenny's Newsletter and summarize the main topics."

The assistant calls `tenlakessoftware/substack-scraper` with the right input and reads the results for you.

### Integrations

Send Substack data wherever you work. Connect runs to **Google Sheets, Slack, Zapier, Make, n8n, Airtable** or any **webhook**, and use Apify **schedules** to run it daily or weekly. A common setup: a weekly leaderboard run with growth tracking that posts the biggest movers to Slack.

### Tips for best results

- Substack search returns about 100 results per keyword. Use **several related keywords** ("AI", "machine learning", "LLM") for wider coverage. A newsletter found under more than one keyword is returned, and billed, only once.
- For lead lists, combine **Search** with **Add publication intelligence** and **Include publicly listed contact emails**.
- For monitoring, schedule **Posts** mode with **Only new since last run** turned on.
- Use the **Overview** view for a quick look and the specific views (Sponsors, Contacts, Leaderboard) for exports.
- If a newsletter can't be reached, the run skips it, keeps going, and lists the reason in the **Run summary**.

### Is it legal to scrape Substack?

Substack Scraper collects only **public, logged-out data**, the same information anyone can see in a browser. It **never bypasses paywalls**, never logs in, and doesn't collect commenter names or handles unless you switch that on. Contact emails are only ones a newsletter **publishes itself** (for example on its About page), and each comes with its source.

Scraping public data is generally lawful, but your use of it must follow the laws that apply to you, such as **GDPR** for personal data and **CAN-SPAM** for email outreach. If you're unsure, ask a lawyer. Read more in Apify's guide to the [legality of web scraping](https://blog.apify.com/is-web-scraping-legal/).

### FAQ

#### Do I need a Substack account or API key?

No. Substack Scraper works with public pages and needs no login, cookies or API key.

#### How accurate are the subscriber counts?

They're the public figures Substack shows on each newsletter, such as "1.1M+ subscribers" or "Thousands of paid subscribers". Large newsletters are shown rounded by Substack itself. The raw label is always included next to the number.

#### Can it scrape paid or paywalled posts?

It returns metadata for every post, including paid ones, and flags them as paywalled. It never accesses paid content, so full text is only available for free posts.

#### Does it work with custom domains?

Yes. Both `name.substack.com` and custom domains such as `www.lennysnewsletter.com` work.

#### How do I get only new posts?

Turn on **Only new since last run** and schedule the run. The scraper remembers what it already delivered to you and only returns, and only charges for, new posts.

#### Why did my run stop early?

Either it reached **Max total results** or your **Max cost per run** budget. The run's status message tells you which. Raise the limit to get more.

#### What happens if Substack changes its site?

We run automated health checks against live Substack every few hours. If Substack changes something, the run fails loudly with a clear message instead of returning empty data, and we're alerted to fix it.

#### Can I get the data in Excel or Google Sheets?

Yes. Export any run as Excel or CSV from the **Output** tab, or connect the Google Sheets integration to update a sheet automatically.

#### I need a field or mode that isn't here.

Open an issue on the **Issues** tab with an example newsletter URL. We actively maintain this Actor and add requested fields regularly.

### Support

Found a bug or have an idea? Use the **Issues** tab and we'll reply quickly. Need a custom Substack or newsletter data solution for your team? Reach us through our [Apify profile](https://apify.com/tenlakessoftware).

If Substack Scraper saves you time, a **rating** on this page helps other people find it. Thank you!

*Substack Scraper is built and maintained by Ten Lakes Software, Minnesota. It isn't affiliated with or endorsed by Substack Inc.*

# Actor input Schema

## `mode` (type: `string`):

Pick what you want. Search finds newsletters by topic; Publication intelligence gives sponsorship-ready data per newsletter.

## `keywords` (type: `array`):

For mode = Search. Topics like "AI", "personal finance", "history".

## `searchType` (type: `string`):

Find newsletters (with subscriber counts) or individual posts.

## `publications` (type: `array`):

For modes Posts, Publication intelligence, Recommendations. Subdomain ("lenny"), substack URL, or custom domain.

## `categories` (type: `array`):

For mode = Leaderboards. e.g. "Technology", "Business", "Culture". Empty = all top-level categories.

## `leaderboardType` (type: `string`):

Bestsellers = paid leaderboard. Rising/all = general category leaderboard.

## `maxRankPerCategory` (type: `integer`):

How many leaderboard positions to collect per category.

## `trackGrowth` (type: `boolean`):

Leaderboards: compare with your previous run and add subscriberChange, rankChange and isNewEntry. Schedule daily or weekly.

## `handles` (type: `array`):

For modes Profiles / Notes. "@handle" or profile URL.

## `enrichPublications` (type: `boolean`):

Search & Leaderboards: add pricing, posting cadence, sponsors detected and sponsorship signals to every newsletter found (charged as publication intelligence).

## `sponsorScanPosts` (type: `integer`):

Publication intelligence: how many recent free posts to scan for sponsor placements. 0 = skip.

## `includeContactEmails` (type: `boolean`):

Publication intelligence: business contact emails the publication publishes on its own pages (About page, support email). Each email includes its source. Use lawfully (GDPR / CAN-SPAM).

## `includeBody` (type: `boolean`):

Posts: full text of free posts (paywalled posts return metadata only).

## `detectSponsors` (type: `boolean`):

Posts: find sponsor placements ("Brought to you by…") and output one sponsor row per brand with link and context.

## `includeComments` (type: `boolean`):

Posts: comment text, threading, likes and replies. Commenter identity is excluded unless enabled below.

## `maxCommentsPerPost` (type: `integer`):

Cap on comments collected per post.

## `includeCommenterPublications` (type: `boolean`):

For commenters who run their own Substack, add that publication's name and URL — find engaged creators in a niche.

## `includeCommenterIdentity` (type: `boolean`):

Adds commenter display name and handle. Personal data: you are responsible for lawful use (GDPR).

## `postedAfter` (type: `string`):

Posts: stop at older posts.

## `maxItemsPerSource` (type: `integer`):

Per publication, author or keyword.

## `onlyNew` (type: `boolean`):

Return only items you haven't received before. Ideal for scheduled monitoring.

## `maxItems` (type: `integer`):

Hard cap on total results for this run.

## `proxy` (type: `object`):

Proxy settings. Apify Proxy (datacenter) is recommended.

## `selfTest` (type: `boolean`):

For the Actor maintainer: runs a health check of all Substack endpoints instead of scraping. Returns no data and charges nothing. Leave off.

## Actor input object example

```json
{
  "mode": "search",
  "keywords": [
    "AI"
  ],
  "searchType": "publications",
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "leaderboardType": "paid",
  "maxRankPerCategory": 100,
  "trackGrowth": true,
  "enrichPublications": false,
  "sponsorScanPosts": 6,
  "includeContactEmails": false,
  "includeBody": false,
  "detectSponsors": false,
  "includeComments": false,
  "maxCommentsPerPost": 100,
  "includeCommenterPublications": false,
  "includeCommenterIdentity": false,
  "maxItemsPerSource": 100,
  "onlyNew": false,
  "maxItems": 1000,
  "proxy": {
    "useApifyProxy": true
  },
  "selfTest": false
}
```

# Actor output Schema

## `results` (type: `string`):

All rows from this run (switch views to see posts, newsletters, sponsors, contacts and more).

## `runSummary` (type: `string`):

Counts, skipped sources and any format-change warnings.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "keywords": [
        "AI"
    ],
    "publications": [
        "https://www.lennysnewsletter.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("tenlakessoftware/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "keywords": ["AI"],
    "publications": ["https://www.lennysnewsletter.com"],
}

# Run the Actor and wait for it to finish
run = client.actor("tenlakessoftware/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "keywords": [
    "AI"
  ],
  "publications": [
    "https://www.lennysnewsletter.com"
  ]
}' |
apify call tenlakessoftware/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,tenlakessoftware/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/1jMRTEc25jH4UB8nV/builds/AdTo9avqh207AkKfi/openapi.json
