# Substack Scraper — Posts, Newsletters, Authors & Free Content (`gazidev/substack-scraper`) Actor

Scrape any Substack newsletter (incl. custom domains): post titles, subtitles, authors, dates, free/paid audience, word count, likes, comment counts, cover images and full Markdown/HTML of free posts. Date filters, keyword publication search. Fast HTTP-only, $0.002 per post. Never bypasses paywalls.

- **URL**: https://apify.com/gazidev/substack-scraper.md
- **Developed by:** [Cemal Atakli](https://apify.com/gazidev) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.00 / 1,000 post scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper — Newsletter Posts, Authors, Likes & Free Content

Scrape any **Substack newsletter**, including publications on **custom domains**. Get every post's title, subtitle, author(s), publish date, free/paid audience, word count, likes (reactions), comment count, restacks, cover image and tags. Free posts also come with their **full text as Markdown, HTML or plain text**. You can also **search Substack publications by keyword**.

- ⚡ **Fast and cheap.** Plain HTTP requests to Substack's public JSON endpoints, with no browser. 100 posts take a few seconds.
- 💲 **$0.002 per post** plus a $0.005 start fee per run. You don't pay for failed inputs or for posts your filters drop.
- 🔒 **Paywall-safe.** Requests are anonymous and never unlock paid content. For paid posts you get metadata and Substack's public teaser text only.
- 🌐 **Works with custom domains** such as `www.lennysnewsletter.com` or `www.astralcodexten.com`, as well as `*.substack.com` addresses and bare subdomains like `lenny`.

### What can this Substack scraper do?

| Feature | |
|---|---|
| Scrape all posts of a publication (the whole archive or the newest N) | ✅ |
| Filter by date range (`2026-01-01`, or relative like `30 days`) | ✅ |
| Filter by audience (free / paid) and post type (newsletter / podcast / thread) | ✅ |
| Full post content for free posts: Markdown, HTML, both, or plain text | ✅ |
| Likes / reactions, comment count, restacks, word count | ✅ |
| Author names, handles, photos, guest-author flag | ✅ |
| Publication info: name, description, author, logo, language, custom domain, first post date | ✅ |
| Individual post URLs | ✅ |
| Keyword search for publications (best-effort, see FAQ) | ✅ |
| Paid-post full text | ❌ Never. We don't bypass paywalls. |

### Use cases

- **Newsletter research and competitor tracking.** See what top Substacks in your niche publish, how often, and which posts get the most likes and comments.
- **Content datasets for AI and RAG.** Clean Markdown of free posts with metadata, ready for embeddings or LLM summarisation.
- **Media monitoring.** Schedule the Actor daily with `dateFrom: "1 day"` to collect new posts from a list of newsletters.
- **Lead generation and influencer discovery.** Search publications by keyword, then rank them by engagement.
- **Archiving and backup** of your own publication's public posts.

### Input example

```json
{
  "publications": ["https://www.astralcodexten.com", "lenny", "noahpinion.substack.com"],
  "postUrls": ["https://www.slowboring.com/p/the-map-is-melting"],
  "maxPostsPerPublication": 20,
  "dateFrom": "2026-06-01",
  "audience": "all",
  "includeContent": true,
  "contentFormat": "markdown"
}
```

Keyword search:

```json
{ "searchQueries": ["climate", "product management"], "maxSearchResults": 20, "scrapeSearchResults": false }
```

| Field | Description |
|---|---|
| `publications` | A subdomain (`lenny`), a `https://x.substack.com` URL, or a custom domain. Post URLs pasted here are detected automatically. |
| `postUrls` | Specific posts (`https://<publication>/p/<slug>`). |
| `searchQueries` / `maxSearchResults` / `scrapeSearchResults` | Find publications by keyword, and optionally scrape their posts too. |
| `maxPostsPerPublication` | Newest first. `0` scrapes the whole archive. Default is 20. |
| `dateFrom` / `dateTo` | `YYYY-MM-DD` or relative dates (`30 days`, `6 months`). |
| `audience` | `all`, `free` or `paid`. |
| `postTypes` | `newsletter`, `podcast`, `thread`. Leave empty for all types. |
| `includeContent` / `contentFormat` | Full body of free posts as `markdown`, `html`, `both` or `text`. |
| `includePublicationInfo` | Adds publication fields to every post. |
| `maxConcurrency`, `proxyConfiguration` | Advanced. The defaults are fine; a proxy is rarely needed. |

### Output example

Each post is one dataset item. The example below is shortened:

```json
{
  "type": "post",
  "postId": 208730073,
  "title": "Announcing Lenny’s Jobs: The best place in the world to find, vet, and land your dream job",
  "subtitle": "Where product managers, engineers, designers, and growth/marketing professionals discover high-quality open roles",
  "slug": "announcing-lennys-jobs-the-best-place",
  "url": "https://www.lennysnewsletter.com/p/announcing-lennys-jobs-the-best-place",
  "postType": "newsletter",
  "publishedAt": "2026-08-18T15:40:06.921Z",
  "audience": "everyone",
  "isPaid": false,
  "authors": [{ "name": "Lenny Rachitsky", "handle": "lenny", "id": 1849774, "isGuest": false }],
  "authorNames": "Lenny Rachitsky",
  "wordCount": 933,
  "reactionCount": 379,
  "reactions": { "❤": 379 },
  "commentCount": 18,
  "restacks": 9,
  "coverImage": "https://substackcdn.com/image/fetch/...png",
  "tags": ["Career"],
  "previewText": "👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice...",
  "publicationName": "Lenny's Newsletter",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "publicationSubdomain": "lenny",
  "publicationCustomDomain": "www.lennysnewsletter.com",
  "publicationAuthor": "Lenny Rachitsky",
  "contentIncluded": true,
  "contentMarkdown": "My goal is to help you succeed in your career. ...",
  "scrapedAt": "2026-09-27T22:23:33+00:00"
}
```

For paid posts, `contentIncluded` is `false` and `contentNote` explains why. Only metadata and `previewText` are returned. Search results have `"type": "publication"` and include the publication name, URL, author, description and a subscriber-count label. A run summary, with counts and any failed inputs, is saved to the `OUTPUT` record of the key-value store. See `SAMPLE_OUTPUT.json` for complete items.

### Pricing

Pay per event. You pay only for results:

| Event | Price |
|---|---|
| Run start | $0.005 |
| Post scraped | $0.002 ($2 per 1,000 posts) |
| Publication search result | $0.002 |

Examples: 100 posts cost $0.205, and 10,000 posts cost about $20. Set a **maximum cost per run** in Apify. The Actor stops cleanly when it reaches the limit and never saves an item it can't charge for.

#### How it compares (Apify Store, September 2026)

| Actor | Monthly users | Start fee | Notes |
|---|---|---|---|
| **This Actor** | new | **$0.005** | $0.002/post, custom domains, date filters, keyword search, Markdown |
| automation-lab Substack scraper | 118 | see listing | |
| fatihtahta Substack scraper | 35 | see listing | |
| easyapi Substack scraper | 23 (rated 1.9★) | $0.09 | |

### Use with AI agents (Apify MCP)

This Actor works as a tool for Claude, ChatGPT, Cursor and other MCP clients through the Apify MCP server (`https://mcp.apify.com`). Add the Actor to your MCP configuration. An agent can then ask for things like *"get the last 10 free posts from Noahpinion as Markdown"* and receive structured JSON. The inputs are simple strings and numbers, so they're easy for an LLM to fill in. `contentFormat: "markdown"` gives compact, token-efficient text for summarising or RAG.

You can also call it from code with the Apify API or client (`apify-client` for Python/JS):

```python
from apify_client import ApifyClient
run = ApifyClient("<TOKEN>").actor("gazidev/substack-scraper").call(run_input={"publications": ["lenny"], "maxPostsPerPublication": 5})
```

### FAQ

**Does it scrape paid or paywalled posts?**
No. The Actor sends anonymous requests only, with no cookies or logins, and it never outputs the body of a post whose audience isn't `everyone`. For paid posts you get the metadata and Substack's public teaser text.

**Does it work with custom domains?**
Yes. Enter the domain (for example `https://www.astralcodexten.com`). `*.substack.com` addresses that redirect to a custom domain are followed automatically.

**How many posts can I get?**
Up to the whole public archive. Set `maxPostsPerPublication: 0` for no limit, and use `dateFrom`/`dateTo` to narrow the range.

**Are comments included?**
Comment **counts** and reaction counts are included. The text of individual comments isn't.

**How does keyword search work?**
It uses the same public search that substack.com uses. That endpoint is undocumented, so results and ranking may change. Substack's older dedicated publication-search endpoint now always returns empty results, so it isn't used.

**Some inputs failed. Was I charged?**
No. Errors such as a site not on Substack or a post returning 404 are logged and listed in the `OUTPUT` summary. They're never charged.

**Is scraping Substack legal?**
This Actor only reads publicly available data that any logged-out visitor can see. You're responsible for how you use it. Respect authors' copyright, and don't republish full articles without permission.

**Is it affiliated with Substack?**
No. This is an independent tool and isn't endorsed by Substack Inc.

# Actor input Schema

## `publications` (type: `array`):

Publications to scrape. Accepts a subdomain (<code>lenny</code>), a substack.com URL (<code>https://lenny.substack.com</code>) or a custom domain (<code>https://www.astralcodexten.com</code>).

## `postUrls` (type: `array`):

Optional list of specific post URLs (<code>https://\<publication>/p/\<slug></code>).

## `searchQueries` (type: `array`):

Optional. Find publications by keyword using Substack's public web search. Each found publication is returned as a <code>publication</code> record.

## `maxSearchResults` (type: `integer`):

Maximum publications returned for each search query.

## `scrapeSearchResults` (type: `boolean`):

If enabled, posts of every publication found by search are scraped too (subject to the post limits below).

## `maxPostsPerPublication` (type: `integer`):

Newest posts first. 0 = no limit (whole archive).

## `dateFrom` (type: `string`):

Only posts published on/after this date. <code>YYYY-MM-DD</code> or relative, e.g. <code>30 days</code>, <code>6 months</code>.

## `dateTo` (type: `string`):

Only posts published on/before this date (<code>YYYY-MM-DD</code>).

## `audience` (type: `string`):

Filter by who can read the post.

## `postTypes` (type: `array`):

Leave empty for all types.

## `includeContent` (type: `boolean`):

Adds the full post body for posts whose audience is <code>everyone</code>. Paywalled posts never include the body — only metadata and Substack's public teaser text.

## `contentFormat` (type: `string`):

Format of the free-post body.

## `includePublicationInfo` (type: `boolean`):

Adds publication name, description, author, logo, language, custom domain etc. to each post record.

## `maxConcurrency` (type: `integer`):

Keep low to stay polite to Substack (it rate-limits aggressive clients).

## `proxyConfiguration` (type: `object`):

Usually not needed. Enable Apify datacenter proxy only if you get blocked.

## Actor input object example

```json
{
  "publications": [
    "https://www.astralcodexten.com",
    "lenny"
  ],
  "maxSearchResults": 20,
  "scrapeSearchResults": false,
  "maxPostsPerPublication": 10,
  "audience": "all",
  "includeContent": true,
  "contentFormat": "markdown",
  "includePublicationInfo": true,
  "maxConcurrency": 4,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `posts` (type: `string`):

No description

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.astralcodexten.com",
        "lenny"
    ],
    "maxPostsPerPublication": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("gazidev/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": [
        "https://www.astralcodexten.com",
        "lenny",
    ],
    "maxPostsPerPublication": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("gazidev/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.astralcodexten.com",
    "lenny"
  ],
  "maxPostsPerPublication": 10
}' |
apify call gazidev/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,gazidev/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/ddpjJJpittnZkmzRa/builds/BOuMlzYMDVrUK27Vc/openapi.json