# Substack Scraper: Posts, Full Text, Newsletters & Leaderboards (`winterlock/substack-scraper`) Actor
Export every post of any Substack newsletter (custom domains too) with likes, comments, restacks and dates; full text and HTML of free posts; publication info with subscriber counts and prices; and category bestseller lists. No login.
- **URL**: https://apify.com/winterlock/substack-scraper.md
- **Developed by:** [Alex Burton](https://apify.com/winterlock) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet
## Pricing
from $2.00 / 1,000 post (archive row)s
This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event
## What's an Apify Actor?
An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.
## How to integrate an Actor?
If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.
For examples already wired to this Actor's own input schema, see the [API](#api) section below.
Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).
# README
## Substack Scraper: Posts, Full Text, Newsletters & Leaderboards
Export Substack newsletter data to JSON, CSV or Excel: every post of a publication with its date, likes, comments and restacks; the full text and HTML of free posts, ready for analysis or an AI/RAG pipeline; publication profiles with subscriber counts and subscription prices; and lists of publications from Substack's category bestseller leaderboards. It reads the same public JSON that Substack's own website loads, so there is no login, no browser and no API key.
One Actor, four jobs:
| Mode | What you get |
|---|---|
| **Posts of publications** | Every post in a publication's archive, newest first: title, subtitle, link, date, type (newsletter, podcast, thread), free or paid, word count, likes, comments, restacks, authors, tags, cover image. Optionally the full content too. |
| **Full content of specific posts** | For each post link: everything above plus the post as plain text and as HTML, its outbound links and a word count. Free posts in full; paid posts as the public preview Substack shows to everyone. |
| **Publication info** | Name, author, description, subscriber count or label, paid-subscriber label, monthly, annual and founding prices, bestseller badge, contributors, sections, launch date, and whether the writer opted out of AI crawlers. |
| **Find publications** | Substack's category leaderboards (Bestsellers, Rising, Top) for Technology, Business, Finance, Culture, U.S. Politics and the other top-level categories. |
### Why use this one
- **Custom domains just work.** Give `lennysnewsletter.com`, `lenny.substack.com`, just `lenny`, a profile link such as `substack.com/@lenny`, or any post link. The same publication given twice in different spellings is scraped and billed once.
- **The whole archive, not the first page.** Substack's archive often returns a short first page; the Actor pages by post count until the archive really ends, and stops cleanly at your limit.
- **Text a model can use.** Post HTML is turned into clean paragraphs and bullet lists with footnotes kept as `[1]`. "Subscribe now" buttons, image toolbars, embedded tweets and other posts' teasers are left out. The original HTML is kept in `contentHtml`.
- **Honest about paywalls.** Every content row says `contentAccess`: `full` (free post), `preview` (the public preview of a paid post) or `none`. Paywalls are never bypassed.
- **Respects AI opt-outs.** Substack lets writers block AI crawlers. `aiCrawlersBlocked` on content and publication rows tells you which ones did, so your pipeline can leave them out.
- **Monitoring built in.** Turn on *Only new posts since the last run*, schedule the Actor, and each run returns only posts published since the previous one.
- **Market data in one table.** Publication rows carry subscriber labels as Substack displays them ("Over 1,200,000 subscribers", "Tens of thousands of paid subscribers") next to the actual prices, so you can size and price a niche in one run.
### How to use it
1. Choose **What to scrape**.
2. Add **Publications**, **Post URLs** or **Categories**, depending on the mode.
3. Click **Start**, then download the results or pull them through the Apify API.
#### Example input: the last six months of two newsletters, with full text
```json
{
"mode": "posts",
"publications": ["https://www.lennysnewsletter.com", "astralcodexten.substack.com"],
"postedAfter": "2026-04-01",
"includeContent": true,
"maxPostsPerPublication": 500
}
```
#### Example input: a daily feed of new posts
```json
{
"mode": "posts",
"publications": ["https://www.noahpinion.blog", "https://www.thefp.com"],
"onlyNewPosts": true
}
```
Schedule it daily in Apify Console. The first run returns the newest posts up to your limit; later runs return only what was published since.
#### Example input: the bestsellers in two categories
```json
{
"mode": "searchPublications",
"categories": ["technology", "finance"],
"categoryRanking": "paid"
}
```
#### Example input: profiles of publications you already know
```json
{
"mode": "publicationDetails",
"publications": ["lenny", "https://newsletter.semianalysis.com", "https://substack.com/@petergyang"]
}
```
### Output
Each item has a `type` field: `post` or `publication`.
**Post** (Posts mode)
```json
{
"type": "post",
"postId": 215694124,
"title": "All of the Lenny & Friends Summit talks are now online!",
"subtitle": "Plus, some reflections and takeaways from the day",
"url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
"postedAt": "2026-09-29T13:15:57Z",
"postType": "newsletter",
"audience": "everyone",
"isPaywalled": false,
"wordCount": 1276,
"reactionCount": 291,
"commentCount": 7,
"restackCount": 5,
"authors": ["Lenny Rachitsky"],
"authorHandles": ["lenny"],
"tags": [],
"coverImageUrl": "https://substack-post-media.s3.amazonaws.com/public/images/9fed347c-0e02-4190-a590-1236719b99de_9213x6142.jpeg",
"publicationId": 10845,
"publicationName": "Lenny's Newsletter",
"publicationUrl": "https://www.lennysnewsletter.com"
}
```
**Post with content** (Full content mode, or Posts mode with *Include full post content*) adds:
```json
{
"title": "Open Thread 455",
"audience": "everyone",
"wordCount": 143,
"updatedAt": "2026-10-13T00:21:04Z",
"contentAccess": "full",
"contentText": "This is the weekly visible open thread. Post about anything you want, ask random questions, whatever. ACX has an unofficial subreddit, Discord, and bulletin board, and in-person meetups around the world. ...\n\n1: Meetup updates: ...",
"contentHtml": "This is the weekly visible open thread. ...
",
"contentWordCount": 141,
"links": ["https://www.reddit.com/r/slatestarcodex/", "https://discord.gg/RTKtdut"]
}
```
`wordCount` is Substack's count for the whole post; `contentWordCount` counts the text you received, so on a paid post's preview the two differ.
**Publication**
```json
{
"type": "publication",
"publicationId": 10845,
"name": "Lenny's Newsletter",
"url": "https://www.lennysnewsletter.com",
"subdomain": "lenny",
"customDomain": "www.lennysnewsletter.com",
"description": "Deeply researched product, growth, and career advice for product leaders, founders, and ambitious builders.",
"authorName": "Lenny Rachitsky",
"authorProfileUrl": "https://substack.com/@lenny",
"subscriberCount": 1200000,
"subscriberCountApprox": "1.2M+",
"subscriberCountLabel": "Over 1,200,000 subscribers",
"paidSubscriberLabel": "Tens of thousands of paid subscribers",
"bestsellerTier": 10000,
"monthlyPrice": 20.0,
"annualPrice": 200.0,
"foundingPrice": 400.0,
"priceCurrency": "USD",
"hasPodcast": true,
"aiCrawlersBlocked": false,
"sections": ["Community wisdom", "How I AI"],
"firstPostAt": "2019-06-14T15:00:48Z"
}
```
Leaderboard rows also carry `rank`, `category`, `categoryRanking` and `listTitle` (for example `"Bestsellers in Technology"`).
### Pricing
Pay only for results:
| Result | Price |
|---|---|
| Post (archive row) | $2.00 per 1,000 |
| Post with full content | $5.00 per 1,000 |
| Publication (info or leaderboard row) | $5.00 per 1,000 |
The default input (50 posts from one newsletter) costs about $0.10. Set a maximum spend per run in the run options and the Actor stops cleanly when it is reached.
### Paywalls, previews and AI opt-outs
The Actor sees what any logged-out reader sees. Free posts come back complete. For a paid post Substack serves the preview the writer chose, which can be a few paragraphs or nothing at all (common for paid discussion threads); those rows have `contentAccess` set to `preview` or `none` and `isPaywalled` set to `true`. There is no option to log in or unlock paid posts.
Writers can tell Substack to block AI crawlers. When they have, `aiCrawlersBlocked` is `true`. If you are building a training set or a retrieval index, filter on it.
### Limits to know about
These come from Substack, not from this Actor:
- **Posts in hidden sections are not listed.** Writers can hide a section's posts from the publication's archive (podcast summaries, community digests, for example). The Actor lists what the archive lists.
- **Subscriber numbers are what the writer shows.** Many publications hide exact counts; `subscriberCount` is then empty and `subscriberCountLabel` or `subscriberCountApprox` carries Substack's rounded wording when there is one. A young publication's ranking line is a launch date, which is not exported as a subscriber label.
- **Leaderboards are short.** When checked, the Technology Bestsellers list stopped at 100 publications, Rising at 75 and Top at 525. Leaderboards exist only for the top-level categories, not for subcategories such as "Artificial Intelligence".
- **No keyword search.** Substack's publication search answers requests from cloud servers, Apify's included, with an empty list, so publications are found through the category leaderboards instead.
- **Popular-first order cannot stop early** on a date filter, because dates are not in order; the Actor reads pages until your limit or the end of the archive.
- **Rate limits.** If Substack answers a burst with HTTP 429, the Actor waits and retries. Keep *Max parallel requests* at 3 or so for large runs.
- **Only new posts.** The Actor remembers the newest post it saved per publication. If a run stops early (max items or spending limit), older new posts it did not reach are not returned by later runs.
### FAQ
**Do I need a Substack account?** No. Everything here is public.
**Does it work with custom domains?** Yes. Use the domain the newsletter lives on, its `name.substack.com` address, or a profile link.
**Can I get the comments?** Not yet. Each post row carries the comment count.
**Can I scrape a paid newsletter I subscribe to?** No. The Actor never logs in, so paid posts come back as their public preview.
**How do I keep a dataset of a newsletter up to date?** Run Posts mode once with a high limit and *Include full post content*, then schedule the same input with *Only new posts since the last run* turned on.
### Feedback
Found a bug or need a field that is missing? Open an issue on the Actor's **Issues** tab.
# Actor input Schema
## `mode` (type: `string`):
Posts of a publication (newest first), full content of specific posts, publication info, or Substack's category leaderboards.
## `publications` (type: `array`):
For Posts and Publication info. A Substack address (lenny.substack.com or just lenny), a custom domain (www.lennysnewsletter.com), a profile link (substack.com/@lenny) or any post link from the publication.
## `postUrls` (type: `array`):
For Full content of specific posts. Links such as https://name.substack.com/p/post-slug, a custom-domain post link, or https://substack.com/home/post/p-123456.
## `categories` (type: `array`):
For Find publications. Substack's top-level categories by name, slug or ID, for example: technology, business, us-politics, finance, culture. Each category's leaderboard is listed in the order chosen below.
## `categoryRanking` (type: `string`):
Bestsellers ranks by paid subscriptions (up to 100). Rising lists fast-growing publications (up to 75). Top is Substack's broader 'Top in ' list (up to 525).
## `maxPostsPerPublication` (type: `integer`):
Posts mode only. The newest posts come first unless Order is changed.
## `sortBy` (type: `string`):
Posts mode. Newest first, or Substack's own 'top' order (most popular).
## `postedAfter` (type: `string`):
Posts mode. A date like 2026-01-31 (UTC). Older posts are left out.
## `postedBefore` (type: `string`):
Posts mode. A date like 2026-06-30 (UTC). Posts from this date on are left out.
## `keyword` (type: `string`):
Posts mode. Only posts that Substack's archive search matches for this word or phrase.
## `includeContent` (type: `boolean`):
Posts mode. Adds the post text and HTML (free posts in full, paid posts as the public preview). One extra request per post, billed as a post with content.
## `onlyNewPosts` (type: `boolean`):
Remembers the newest post saved for each publication and on later runs returns only newer ones. Schedule the Actor to get a feed of new posts and pay only for those.
## `maxPublicationsPerCategory` (type: `integer`):
Find publications mode. Bestsellers lists stop at 100, Rising at 75, Top at 525.
## `maxItems` (type: `integer`):
Stop the run after saving this many items. Leave empty for no limit.
## `maxConcurrency` (type: `integer`):
How many requests run at once. Substack rate-limits bursts; three is a safe pace.
## `proxyConfiguration` (type: `object`):
Not needed for most runs. Use it if Substack starts refusing requests from the run's server.
## Actor input object example
```json
{
"mode": "posts",
"publications": [
"https://www.lennysnewsletter.com"
],
"postUrls": [
"https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit"
],
"categories": [
"technology"
],
"categoryRanking": "paid",
"maxPostsPerPublication": 50,
"sortBy": "new",
"includeContent": false,
"onlyNewPosts": false,
"maxPublicationsPerCategory": 100,
"maxConcurrency": 3,
"proxyConfiguration": {
"useApifyProxy": false
}
}
```
# Actor output Schema
## `results` (type: `string`):
No description
# API
You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.
## JavaScript example
```javascript
import { ApifyClient } from 'apify-client';
// Initialize the ApifyClient with your Apify API token
// Replace the '' with your token
const client = new ApifyClient({
token: '',
});
// Prepare Actor input
const input = {
"publications": [
"https://www.lennysnewsletter.com"
],
"postUrls": [
"https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit"
],
"categories": [
"technology"
]
};
// Run the Actor and wait for it to finish
const run = await client.actor("winterlock/substack-scraper").call(input);
// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
console.dir(item);
});
// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs
```
## Python example
```python
from apify_client import ApifyClient
# Initialize the ApifyClient with your Apify API token
# Replace '' with your token.
client = ApifyClient("")
# Prepare the Actor input
run_input = {
"publications": ["https://www.lennysnewsletter.com"],
"postUrls": ["https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit"],
"categories": ["technology"],
}
# Run the Actor and wait for it to finish
run = client.actor("winterlock/substack-scraper").call(run_input=run_input)
# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
print(item)
# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start
```
## CLI example
```bash
echo '{
"publications": [
"https://www.lennysnewsletter.com"
],
"postUrls": [
"https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit"
],
"categories": [
"technology"
]
}' |
apify call winterlock/substack-scraper --silent --output-dataset
```
## MCP server setup
```json
{
"mcpServers": {
"apify": {
"type": "http",
"url": "https://mcp.apify.com/?tools=fetch-actor-details,winterlock/substack-scraper"
}
}
}
```
The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).
## OpenAPI specification
Download the OpenAPI definition: https://api.apify.com/v2/actors/JADQhOyYC3AabIV1D/builds/ci4aORckSlMIAS6Ob/openapi.json