# Substack Scraper: Posts and Newsletter Finder (`nightbreaker_studio/substack-scraper`) Actor

Collect public Substack post metadata and available free text or paid previews. Find newsletters through public categories with subscriber-size labels. Includes filters, deduplication and explicit run summaries.

- **URL**: https://apify.com/nightbreaker_studio/substack-scraper.md
- **Developed by:** [NightBreaker Studio](https://apify.com/nightbreaker_studio) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper: Posts and Newsletter Finder

Two tools in one Actor:

1. **Posts**: collect posts exposed by a publication's public archive, including supported public HTTPS custom domains. Fields include title, subtitle, date, authors, tags, reactions and paywall status. When content is available, free posts return plain text and paid posts return only their public preview.
2. **Newsletter finder**: browse the categories currently exposed by Substack and collect available public metadata: subscriber size labels (e.g. `1.1M+`), paid tier labels, bestseller badge, author, language and launch date.

No login, browser or proxy is required. Availability, field coverage and directory rankings depend on Substack's public endpoints. Private network addresses and non-HTTPS custom domains are rejected.

### Who uses it

- **Sponsors and marketers**: build a list of newsletters in your niche, sized by audience, to buy sponsorships or pitch partnerships.
- **Writers and growth teams**: study what works for competitors: most popular posts, reactions, comments, length, free vs paid.
- **Researchers and AI builders**: collect clean text from free posts for analysis, summaries or RAG.
- **Investors and analysts**: track creator businesses and categories over time.

### How to use it

**Posts mode**

- Add publications: `https://www.lennysnewsletter.com`, `astralcodexten`, `https://newsletter.pragmaticengineer.com`.
- Choose **Newest** or **Most popular**, a max per publication, and optionally:
  - **Posted within the last N days**;
  - **Free posts only**;
  - **Keywords** in title or subtitle.

**Newsletter finder mode**

- Pick categories, e.g. `Technology`, `Finance`.
- Optionally narrow the list:
  - **Languages**, e.g. `en`;
  - **Minimum subscribers**, e.g. `10000`;
  - **Keywords** in the name or author bio.

Download as JSON, CSV or Excel, or connect via API, Zapier, Make, n8n. Schedule it to watch publications for new posts.

### Output examples

Illustrative records. Optional fields may be `null`; a post's `detailStatus` reports `available`, `unavailable` or `not_requested`.

```json
{
  "type": "post",
  "publication": "www.lennysnewsletter.com",
  "title": "All of the Lenny & Friends Summit talks are now online!",
  "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "postDate": "2026-09-29T13:15:57.512Z",
  "isPaywalled": false,
  "authors": ["Lenny Rachitsky"],
  "wordCount": 1276,
  "reactions": 292,
  "comments": 8,
  "restacks": 5,
  "content": "Full plain text of the post…"
}
```

```json
{
  "type": "newsletter",
  "category": "Finance",
  "name": "The Capitalist",
  "url": "https://www.thecapitalistmag.com",
  "author": "Matthew Miller",
  "subscribers": "740K+",
  "paidTier": "Hundreds of paid subscribers",
  "bestseller": "Bestseller 1000+",
  "hasPaidPlan": true,
  "language": "en",
  "createdAt": "2025-04-14T22:20:11.478Z"
}
```

### Pricing

See the Actor's **Pricing** tab for the current result price, any run-start fee and applicable platform charges. The `result` event is emitted only for unique matching posts or newsletters sent to the dataset, after filters and deduplication. Set a maximum total charge per run in Apify; the Actor stops saving after the SDK reports that limit.

### Run outcomes and limits

- `SUMMARY.status` is `complete`, `no_matching_results`, `partial`, `stopped_cost_limit` or `failed`. If every source is invalid or fails, the run fails. A valid empty archive or no matches is a successful zero-match run. `complete` means processing finished within your filters and caps, not that every publication or post was discovered.
- Inspect `failures` and `detailFailures` for partial data. A post can retain its basic archive metadata when the optional text request fails; the missing text is not claimed as complete.
- An active `postedWithinDays` filter excludes missing, invalid and future dates. `unknownDatesSkipped` and `futureDatesSkipped` report those exclusions. Sorting by newest does not guarantee all old or pinned posts precede every recent post, so filtered runs may scan much of an archive.
- Limits apply to unique saved results. The same newsletter in several selected categories is saved once under the first matching category. Deduplication is within a run only; subsequent scheduled runs can return and charge the same records again.
- Archive scans have a 1,000-page safeguard; category scans have a 100-page safeguard. Repeated pages and malformed responses produce explicit failures rather than false empty success. HTTP requests time out after 30 seconds and responses are capped at 16 MiB.

### Good to know

- Paid posts return only the public preview: the Actor never bypasses paywalls.
- Subscriber sizes are public labels or ranges, not exact audience measurements. A minimum subscriber filter compares the numeric lower bound; missing or unparseable values are excluded.
- The category directory is a subset selected by Substack, not a complete list of newsletters. Category availability can change.
- Only public endpoints are requested. Published articles and author bios can contain names or contact details. Review your intended reuse and permissions before exporting or republishing content.

# Actor input Schema

## `mode` (type: `string`):

Posts: articles from the publications you list. Newsletters: discover newsletters in Substack categories, with subscriber size and paid tier.

## `publications` (type: `array`):

Substack addresses or subdomains, e.g. astralcodexten or https://www.lennysnewsletter.com. Public HTTPS custom domains are supported when they expose the Substack endpoints.

## `sort` (type: `string`):

Newest first, or the publication's most popular posts first.

## `maxPostsPerPublication` (type: `integer`):

Maximum unique saved posts per publication. 0 = no output cap, subject to a 1,000-page archive safeguard.

## `postedWithinDays` (type: `integer`):

0 = any date. When enabled, missing, invalid and future dates are excluded and counted in SUMMARY.

## `onlyFree` (type: `boolean`):

Skip posts for paying subscribers.

## `includeContent` (type: `boolean`):

Available plain text for free posts; only the public preview for paid posts. Missing detail requests are disclosed in SUMMARY.

## `keywords` (type: `array`):

Posts mode: keep posts whose title or subtitle contains one of these. Newsletters mode: keep newsletters whose name or author bio contains one of these.

## `categoryNames` (type: `array`):

Substack categories, e.g. Technology, Business, Finance, Culture, Health & Wellness, Food & Drink, Science, Education, Crypto, Design, Travel, Sports, Music, News.

## `languages` (type: `array`):

Keep only newsletters in these language codes, e.g. en, es, de, it. Leave empty for all.

## `minSubscribers` (type: `integer`):

Uses the size Substack shows publicly (e.g. 10K+). 0 = no minimum. Newsletters that hide their size are dropped when this is set.

## `maxNewslettersPerCategory` (type: `integer`):

Maximum unique saved newsletters per category. 0 = no output cap, subject to a 100-page safeguard. Overlapping categories are deduplicated.

## Actor input object example

```json
{
  "mode": "posts",
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "sort": "new",
  "maxPostsPerPublication": 50,
  "postedWithinDays": 0,
  "onlyFree": false,
  "includeContent": true,
  "categoryNames": [
    "Technology"
  ],
  "minSubscribers": 0,
  "maxNewslettersPerCategory": 100
}
```

# Actor output Schema

## `results` (type: `string`):

All posts or newsletters found in this run.

## `summary` (type: `string`):

Results per publication or category, and anything that failed.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "https://www.lennysnewsletter.com"
    ],
    "categoryNames": [
        "Technology"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("nightbreaker_studio/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["https://www.lennysnewsletter.com"],
    "categoryNames": ["Technology"],
}

# Run the Actor and wait for it to finish
run = client.actor("nightbreaker_studio/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "https://www.lennysnewsletter.com"
  ],
  "categoryNames": [
    "Technology"
  ]
}' |
apify call nightbreaker_studio/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,nightbreaker_studio/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/jZelGMPWsiyOstkHJ/builds/pfwXpZvW4oGB2ID8X/openapi.json