# Substack Scraper \[$2.5/1k💰] | Newsletter Data | Engagement (`ahmed_jasarevic/substack-scraper`) Actor

Extract Substack newsletter data: posts, authors, likes, comments, restacks, and subscriber estimates from any public publication. Built for sponsorship prospecting, creator lead generation, competitive monitoring, and audience research — using Substack's own public JSON API with no browser needed.

- **URL**: https://apify.com/ahmed\_jasarevic/substack-scraper.md
- **Developed by:** [Ahmed Jasarevic](https://apify.com/ahmed_jasarevic) (community)
- **Categories:**
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $2.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Substack Scraper — Newsletter Posts, Engagement & Analytics Data

Extract structured post and engagement data from any public Substack newsletter — posts, authors, publish dates, likes, comments, restacks, subscriber estimates, and archive URLs — using Substack's own public archive JSON API. No browser, no login, no proxy needed.

### Use This Actor For

- **Newsletter sponsorship prospecting** — rank publications by engagement metrics (likes, comments, restacks) and filter free vs paid audiences before pitching
- **Creator-economy lead generation** — build author lists with handles, bios, bylines, and post history for outreach and PR campaigns
- **Competitive newsletter monitoring** — track competitor publications for new posts, publishing cadence, and engagement trends
- **Audience research** — analyze topics, posting frequency, and which posts resonate in any Substack niche
- **Newsletter market intelligence** — compile structured datasets of Substack publications for market sizing and category analysis
- **Content strategy research** — identify high-performing post formats, topics, and engagement patterns across newsletters

### How It Works

1. Point the actor at one or more Substack publication URLs (both `xxx.substack.com` subdomains and custom domains work).
2. It fetches the homepage to extract publication metadata, then pages through the internal archive endpoint to collect posts.
3. Each post is returned as a structured JSON record enriched with engagement metrics, author bylines, and publication context.
4. Optionally enable full body text extraction for free posts (adds one request per post).

Runs entirely on HTTP requests — no headless browser, no Cloudflare bypass, no DOM scraping. Uses the same JSON endpoints Substack's own web client calls.

### Extract Real Substack Newsletter Data for Sponsorship Research

| Field | Type | Required | Default | Notes |
|---|---|---|---|---|
| `publicationUrls` | `string[]` | Yes | — | Substack publication URLs. Works with `xxx.substack.com` subdomains and custom domains (e.g. `https://www.lennysnewsletter.com`). |
| `maxPostsPerPublication` | `integer` | No | `50` | Maximum number of posts to extract per publication. |
| `maxTotalPosts` | `integer` | No | `200` | Maximum total posts across all publications. Free users are capped at 10 posts per run. |
| `sortOrder` | `string` | No | `"new"` | Archive order: `new` (newest posts first) or `top` (most engaged posts first). |
| `searchQuery` | `string` | No | `""` | Optional keyword filter applied server-side to the publication archive (e.g. `"AI"`). Leave empty for all posts. |
| `audienceFilter` | `string` | No | `"all"` | Filter by paywall status: `all`, `free` (non-paywalled), or `paid` (paywalled posts). |
| `includePostBodies` | `boolean` | No | `false` | Fetch full body text for free (non-paywalled) posts. Costs one extra request per post. Paywalled bodies are never fetched. |
| `maxRequestsPerCrawl` | `integer` | No | `300` | Safety cap on total HTTP requests for the whole run. |
| `proxy` | `object` | No | `{"useApifyProxy":true}` | Proxy settings. Apify proxy is ON by default so your IP is never exposed. |

### Build Structured Substack Databases Without Official API Access

Each dataset item represents one post with the following fields:

| Field | Description |
|---|---|
| `publicationName` | Newsletter name (from embedded page state). |
| `publicationUrl` | Canonical publication URL. |
| `publicationTagline` | Publication tagline/description. |
| `publicationLogo` | Publication logo URL. |
| `publicationLanguage` | Publication language code. |
| `subscriberEstimate` | Raw "X subscribers / readers" hero copy where the publication shows it; `null` otherwise. |
| `archiveUrl` | Public archive URL of the publication. |
| `postId` | Substack internal post ID. |
| `title` | Post title. |
| `subtitle` | Post subtitle. |
| `slug` | URL slug. |
| `author` | Primary author name. |
| `authorHandle` | Author Substack handle. |
| `authorBio` | Author biography. |
| `authorPhoto` | Author profile photo URL. |
| `guestAuthors` | Array of guest author names (if multi-author post). |
| `publishDate` | ISO 8601 publish timestamp. |
| `audience` | Paywall status (`everyone` = free, `only_paid`). |
| `isPaid` | Boolean: true if post is behind a paywall. |
| `likes` | `reaction_count` from Substack. |
| `reactions` | Per-emoji reaction breakdown where present. |
| `comments` | `comment_count` + `child_comment_count`. |
| `restacks` | Restack (repost) count. |
| `wordCount` | Post word count. |
| `section` | Post section/category. |
| `description` | Post description/summary. |
| `coverImage` | Cover image URL. |
| `url` | Canonical post URL. |
| `type` | Post type (newsletter, podcast, thread, etc.). |
| `scrapedAt` | ISO 8601 timestamp of when the data was scraped. |

### Example Input

```json
{
  "publicationUrls": [
    "https://astralcodexten.substack.com",
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 50,
  "maxTotalPosts": 200,
  "sortOrder": "new",
  "audienceFilter": "all",
  "includePostBodies": false
}
```

### Example Output

```json
{
  "publicationName": "Astral Codex Ten",
  "publicationUrl": "https://www.astralcodexten.com",
  "author": "Scott Alexander",
  "authorHandle": "astralcodexten",
  "title": "Your Book Review: The Tale Of Genji",
  "publishDate": "2026-09-04T19:34:24.551Z",
  "audience": "everyone",
  "isPaid": false,
  "likes": 165,
  "comments": 75,
  "restacks": 6,
  "reactions": { "\u2764\uFE0F": 142, "\uD83D\uDE02": 23 },
  "subscriberEstimate": null,
  "archiveUrl": "https://www.astralcodexten.com/archive",
  "url": "https://www.astralcodexten.com/p/your-book-review-the-tale-of-genji",
  "wordCount": 3200,
  "type": "newsletter",
  "scrapedAt": "2026-09-06T18:30:00.000Z"
}
```

### Schedule Runs for Ongoing Newsletter Monitoring

- **API** — call the actor via the Apify REST API for programmatic data pipelines.
- **Webhooks** — connect to Zapier, Make, or n8n to trigger downstream workflows on new data.
- **Scheduling** — set up daily or weekly scheduled runs with a small `maxPostsPerPublication` to detect new posts cheaply.

Recurring usage improves Apify Store recommendation signals and keeps your newsletter monitoring data fresh.

### Related Actors

- [Substack Newsletter Scraper](https://apify.com/sourabhbgp/substack-scraper) — more comprehensive scraping with comments, Notes, and facepile data (50+ fields per post).
- [Substack Scraper](https://apify.com/bitofacoder/substack-scraper) — includes monitor/incremental mode and single-post detail mode.
- [Substack Lead Gen Scraper](https://apify.com/sourabhbgp/substack-lead-gen) — B2B lead generation focused on author profiles, subscriber counts, and sponsorship signals.
- [Substack Scraper by samwise.agency](https://apify.com/samwise.agency/substack-scraper) — lightweight scraper with engagement metrics and custom domain support.
- [Substack Newsletter & Creator API](https://apify.com/elegant_economy/substack-newsletter-scraper) — creator-focused with social link discovery and recent post metrics.

### FAQ

#### Why use this actor instead of the Substack API?

Substack does not publish an official public content API for third-party developers. This actor uses the same internal JSON endpoints (`/api/v1/archive`, embedded homepage JSON) that Substack's own web client calls — no browser rendering, no authentication, no Cloudflare bypass required. It handles pagination, rate limiting, data normalization, and custom domain redirects automatically.

#### What are alternatives to scraping Substack data?

Alternatives include: (1) Substack's built-in export feature for your own publication's subscriber and post data, (2) third-party analytics tools like StackStats or Metrisub that read your Substack CSV exports, (3) building a custom scraper using Substack's hidden JSON endpoints directly, or (4) using the Python `scrape-substack` package. This actor automates the extraction of *other publications'* public data at scale, which none of these alternatives do.

#### Can this actor scrape paywalled post content?

No. The actor only collects publicly visible metadata for paywalled posts (title, author, publish date, audience flag). Full body text is extracted only for free (non-paywalled) posts when `includePostBodies` is enabled. Substack's archive API returns only the free preview for paid posts.

#### Does this actor work with custom-domain Substack publications?

Yes. Both `xxx.substack.com` subdomain URLs and custom domain URLs (e.g. `https://www.lennysnewsletter.com`) are supported. The actor follows redirects from subdomain to custom domain automatically and builds API URLs from the canonical host.

#### How much does it cost to scrape Substack?

The actor is HTTP-only (no browser) and runs at 256 MB memory. A typical run uses ~1 request for the homepage plus ~1 request per 12 posts (Substack's page size), so 100 posts costs approximately 10 requests. A few thousand posts cost a fraction of a dollar in Apify compute. Free-plan users get a capped 10-post preview per run.

#### How does subscriberEstimate work?

Substack does not expose subscriber counts in any public JSON endpoint. The actor best-effort matches "X subscribers / readers" hero copy from the publication homepage HTML and returns `null` when the publication hides this information. Most publications do not show subscriber counts publicly.

#### Can I discover Substack publications by keyword or topic?

No. Substack's global search API requires a logged-in session. This actor covers public data only — you provide the publication URLs you want to analyze. For discovery, try the [Substack Lead Gen Scraper](https://apify.com/sourabhbgp/substack-lead-gen) which supports category-based discovery.

#### Can I filter posts by topic or keyword?

Yes. Use the `searchQuery` input field to apply a server-side keyword filter to a publication's archive. For example, set `searchQuery: "AI"` to only return posts matching that topic. This reduces request volume compared to fetching all posts and filtering client-side.

#### Can I sort by most-engaged posts?

Yes. Set `sortOrder: "top"` to retrieve posts sorted by engagement (most likes/comments first) instead of chronological order. This is useful for identifying top-performing content for sponsorship analysis or content strategy research.

### For AI Agents & LLM Apps

**Purpose:** Extract structured post-level data and engagement metrics from public Substack newsletters via Substack's own archive JSON API. Returns one JSON record per post with publication metadata, author bylines, engagement counts, and paywall status.

**Minimal working input:**

```json
{
  "publicationUrls": ["https://astralcodexten.substack.com"]
}
```

**Variant — topic-filtered extraction:**

```json
{
  "publicationUrls": ["https://astralcodexten.substack.com"],
  "searchQuery": "AI",
  "audienceFilter": "free",
  "sortOrder": "top"
}
```

**Output fields:** `publicationName`, `publicationUrl`, `author`, `authorHandle`, `authorBio`, `title`, `subtitle`, `publishDate`, `audience`, `isPaid`, `likes`, `comments`, `restacks`, `reactions`, `subscriberEstimate`, `archiveUrl`, `url`, `wordCount`, `type`, `scrapedAt`

**Behaviors an agent should know:**

- `publicationUrls` is the only required field — all other fields have sensible defaults.
- Free users are hard-capped at 10 posts per run by Substack, regardless of `maxPostsPerPublication` setting.
- `includePostBodies: true` roughly doubles request volume and cost — only enable when body text is needed.
- `subscriberEstimate` is `null` for most publications (Substack hides counts in public JSON).
- `sortOrder: "top"` may return fewer posts than `sortOrder: "new"` for some publications.
- `maxTotalPosts` is a hard ceiling across all publications; `maxPostsPerPublication` is per-publication.
- Custom domain publications must be called on their canonical host — the actor follows the redirect automatically.

**Billing model:** Pay-per-event. $0.005 per actor start + $0.0025 per result (per post returned).

### Legal & Compliance Disclaimer

This actor is an independent tool and is not affiliated with, endorsed by, or sponsored by Substack, Inc. It accesses only publicly available pages and uses Substack's own unauthenticated archive JSON API endpoints — no login bypass, no paywall circumvention, no CAPTCHA solving. Users are responsible for their own compliance with Substack's Terms of Service and applicable data-protection regulations (including GDPR and CCPA where relevant) when processing the extracted data. The engagement data returned (likes, comments, restacks) reflects publicly visible metrics at the time of scraping and should not be treated as private analytics.

### SEO Keywords

substack scraper, substack data extraction, substack newsletter data, substack api alternative, scrape substack newsletters, substack post analytics, substack engagement metrics, substack newsletter research, substack sponsorship prospecting, substack creator lead generation, newsletter data scraping, substack archive scraper, substack publication data, substack newsletter analytics, substack scraper apify, scrape substack api, substack newsletter database, substack content analysis, newsletter competitive intelligence, substack post archive, substack newsletter scraping tool, substack custom domain scraper, newsletter data extraction api, substack restacks comments likes

# Actor input Schema

## `publicationUrls` (type: `array`):

Substack publication URLs to scrape (works with both xxx.substack.com subdomains and custom domains). One post record is returned per post.

## `maxPostsPerPublication` (type: `integer`):

Maximum number of posts to extract from each publication.

## `maxTotalPosts` (type: `integer`):

Maximum total posts across all publications (paid users). Free users are capped at 10 posts per run.

## `sortOrder` (type: `string`):

Archive order: newest posts first, or top (most engaged) posts first.

## `searchQuery` (type: `string`):

Optional keyword filter applied server-side to the publication archive (e.g. "AI"). Leave empty for all posts.

## `audienceFilter` (type: `string`):

Filter posts by paywall status: all posts, free posts only, or paid (paywalled) posts only.

## `includePostBodies` (type: `boolean`):

Fetch full body text for free (non-paywalled) posts via the post detail endpoint. Costs one extra request per post. Paywalled bodies are never fetched.

## `maxRequestsPerCrawl` (type: `integer`):

Safety cap on total HTTP requests for the whole run.

## `proxy` (type: `object`):

Proxy settings. Apify proxy is ON by default so your IP is never exposed.

## Actor input object example

```json
{
  "publicationUrls": [
    "https://astralcodexten.substack.com",
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 50,
  "maxTotalPosts": 200,
  "sortOrder": "new",
  "searchQuery": "",
  "audienceFilter": "all",
  "includePostBodies": false,
  "maxRequestsPerCrawl": 300,
  "proxy": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publicationUrls": [
        "https://astralcodexten.substack.com",
        "https://www.lennysnewsletter.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("ahmed_jasarevic/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publicationUrls": [
        "https://astralcodexten.substack.com",
        "https://www.lennysnewsletter.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("ahmed_jasarevic/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publicationUrls": [
    "https://astralcodexten.substack.com",
    "https://www.lennysnewsletter.com"
  ]
}' |
apify call ahmed_jasarevic/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,ahmed_jasarevic/substack-scraper"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/nLWfUaBM8L91N2PeP/builds/NY4YDCsEdbLdgnLQ1/openapi.json