# Substack Scraper — Publication Feeds and Posts (`thescrapelab/substack-scraper`) Actor

Extract Substack feeds, bounded archive metadata and public year-sitemap links. Custom domains, explicit coverage and checkpoints.

- **URL**: https://apify.com/thescrapelab/substack-scraper.md
- **Developed by:** [Inus Grobler](https://apify.com/thescrapelab) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.15 / 1,000 saved results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

### What is Substack Scraper?

**Substack Scraper extracts newsletter posts and publication metadata into structured data you can export as CSV, Excel or JSON.** Paste newsletter URLs, choose your result limits, and collect titles, links, dates, public author names and optional excerpts. Substack subdomains and custom publication domains are supported.

Use it to research a newsletter shortlist, build a publication content index, or collect new and changed posts through repeat runs. Start with recent RSS/Atom posts; optional sources provide bounded older-post metadata or public post links by year.

- **Simple input:** paste URLs directly; no code is needed in Apify Console.
- **Useful filters:** select posts by date or a word or phrase.
- **Exportable results:** download data or retrieve it through the Apify API.
- **Incremental collection:** supply a saved checkpoint to skip unchanged records.
- **Visible coverage:** inspect source errors, missing values and the limits reached.

Notes, replies, private feeds and paid article bodies are unsupported. No source guarantees a complete archive. Use publications you are authorized to process.

### How to scrape Substack newsletters

1. Open [Substack Scraper in Apify Console](https://console.apify.com/actors/70ma24L3UhHbFgx3q) and select **Input**. The Actor is currently private, so your account needs access.
2. Paste publication links into **Newsletter or post URLs**. Use **Bulk edit** to paste several URLs, one per line. An empty list uses your existing API selection or, by default, the Simon Willison example.
3. Set **Total results** and **Results per newsletter**. Both default to 10. Raise both to collect more than 10 posts from one newsletter.
4. Set **Maximum cost per run** in the run options, then select **Start**. The default source collects recent posts with excerpts of up to 300 characters.
5. Open the dataset and export **JSON, CSV or Excel**. The Output tab also links to the coverage report and continuation checkpoint.

Leave the optional sections collapsed for your first run. **Filter posts** contains dates and keywords; **Content and output** changes text and record type; **Older posts and year indexes** selects other sources. **Advanced limits and continuation** controls source requests, work time and checkpoints.

### Substack Scraper pricing

**$0.15 per 1,000 saved results + $0.002 per start. Platform usage during the run is included.** The same rate applies to all Apify plans, with no Actor subscription or feature add-ons.

| Saved results | Charge with one start (USD) |
| --- | --- |
| 0 | $0.002 |
| 10 | $0.0035 |
| 100 | $0.017 |
| 1,000 | $0.152 |
| 10,000 | $1.502 |

These are price calculations, not guaranteed collection volumes. The formula is **$0.002 × starts + $0.00015 × saved results**. Resurrecting an interrupted run can incur another start fee. At the supported 128–256 MB memory, each start incurs one start event.

Each delivered post, publication record, archive metadata row or year-index link is one result. Updated records delivered later count again. Unchanged records skipped with a supplied checkpoint, suppressed duplicates, filtered items and source errors have no result charge. Empty or failed runs that start still incur the start fee. Independent runs without a checkpoint can return and bill the same records again.

Use **Maximum cost per run** to cap spending. The minimum selectable cap is $0.002; allow at least $0.00215 for a start and one result. A spending limit can return partial results and a checkpoint. Apify billing is authoritative, and platform interruption can prevent a final checkpoint save.

Long scans that produce few new results may stop early under the included-usage allowance. In **Coverage and errors**, `operating_cost_limit` identifies this stop; `operating_cost_unavailable` means usage could not be checked safely. Saved results remain available, and a checkpoint is saved when possible. Review the coverage and narrow the selection before continuing; every new start has its own start fee.

Your Apify account subscription, applicable taxes and storage retained after the run are governed separately by Apify. There is no automatic increase or expiry attached to this introductory Actor rate.

### What Substack data can you extract?

Choose **Where to find posts** to select a source:

| Source | What you receive | Coverage |
| --- | --- | --- |
| **Recent posts** (default) | Current RSS/Atom entries with post metadata and optional feed text | Current feed window only |
| **Older posts** (experimental) | Bounded archive metadata, optional public subtitles and observed engagement where supplied | Configured pages; publication sections may be omitted |
| **Post links by year** (advanced) | Titles and URLs from selected public year indexes | Selected indexes only; dates, authors, text and engagement remain unknown |

To return one publication metadata record per newsletter, choose **Content and output → Return → Newsletter details only** and keep **Recent posts** as the source. Publication records include the name, URL, description, available public author names and observed feed item count. Subscriber counts and subscription prices are unsupported and remain null.

The experimental archive source uses an undocumented public site route. This Actor runs through the **Apify API**; it is not an official Substack Developer API integration. Neither public access nor a successful run establishes commercial reuse permission.

#### Text, previews and engagement

**Short excerpt** uses the feed description or public archive subtitle. **Text supplied by the feed** uses text already present in RSS/Atom and works only with Recent posts. **No text** returns metadata. `maxTextCharacters` limits plain text; `textTruncated` reports clipping.

Feed text may be incomplete or a preview. `bodyText` stays null; feed `accessStatus` is `unknown` and `isPreviewOnly` is null. Archive audience labels can identify free or paid access, but do not establish how much text a visitor can read. Missing engagement values stay null. Observed reactions, comments and restacks are counts, not audience reach.

### Substack scraper input examples

This example requests up to 20 recent posts across two newsletters, with at most 10 from each:

```json
{
  "startUrls": [
    {"url": "https://simonw.substack.com"},
    {"url": "https://www.lennysnewsletter.com"}
  ],
  "maxItems": 20,
  "maxPostsPerPublication": 10
}
```

Publication homepages, `/feed`, `/archive` and `/p/post` URLs are accepted. For JSON/API input, `startUrls` accepts URL strings, bare names such as `simonw`, objects containing `url`, or a mixture. The Console form uses `newsletterUrls` with URL objects; a non-empty `newsletterUrls` list takes precedence over `startUrls`. Leave it empty or omit it to use `startUrls`. Add URLs directly; hosted URL-list files are unsupported. The `collection` setting selects the source; a URL ending in `/archive` does not switch it automatically.

These API inputs select the same newsletter:

```json
{"startUrls": ["https://simonw.substack.com"]}
```

```json
{"startUrls": ["simonw"]}
```

```json
{"startUrls": [{"url": "https://simonw.substack.com"}]}
```

#### Input settings reference

| JSON/API field | Default | Purpose |
| --- | --- | --- |
| `newsletterUrls` | `[]` | Console URL list as `{"url": "https://..."}` objects; up to 100; a non-empty list replaces `startUrls` |
| `startUrls` | Simon Willison example URL | API-compatible URL strings, bare names or URL objects; up to 100; used when `newsletterUrls` is empty |
| `maxItems` | `10` | Total dataset records; up to 10,000 |
| `maxPostsPerPublication` | `10` | Delivered posts per newsletter; up to 2,000 |
| `keyword` | Empty | Case-insensitive phrase in titles/descriptions; titles only for year indexes |
| `publishedAfter` | Unset | Inclusive start date or ISO timestamp, using UTC |
| `publishedBefore` | Unset | Exclusive end date or ISO timestamp, using UTC |
| `mode` | `"posts"` | Posts or `"publications"` metadata |
| `textMode` | `"excerpt"` | `"none"`, `"excerpt"` or `"feed"` |
| `maxTextCharacters` | `300` | Maximum plain-text characters per post |
| `collection` | `"feed"` | `"feed"`, `"archive"` or `"sitemap"` |
| `maxArchivePages` | `2` | Older-post pages per newsletter; up to 100 |
| `sitemapYears` | `[]` | One to five explicit years for year-index discovery |
| `maxRequests` | `20` | Source requests including redirects and retries; up to 120 |
| `maxRunSeconds` | `90` | Collection work-time budget, in seconds |
| `resumeState` | `{}` | Previous `CHECKPOINT` JSON for incremental collection |

Date and keyword filters apply to posts, not publication records. The end date excludes the selected day. Date-filtered posts with missing or invalid dates are excluded and reported. Exact date filters are unavailable for year indexes. Keyword filtering applies only to your supplied newsletters, not platform-wide search.

#### Collect older Substack post metadata

```json
{
  "startUrls": [{"url": "https://simonw.substack.com"}],
  "collection": "archive",
  "maxArchivePages": 2,
  "maxItems": 24,
  "maxPostsPerPublication": 24,
  "maxRequests": 3,
  "textMode": "none"
}
```

Each archive page requests up to 12 entries, plus one feed lookup per newsletter. Short pages do not prove that history has ended. Collection stops at an observed empty/repeated page, access block or configured limit. `archiveComplete` remains false. Older posts requires `mode: "posts"` and `textMode: "none"` or `"excerpt"`.

#### Discover Substack post links by year

```json
{
  "startUrls": [{"url": "https://www.lennysnewsletter.com"}],
  "collection": "sitemap",
  "sitemapYears": [2026],
  "maxItems": 10,
  "maxPostsPerPublication": 10,
  "maxRequests": 2,
  "textMode": "none"
}
```

Each selected year adds one request after the publication feed lookup. These results are labelled `contentCoverage: "index_only"`. The index year is not an inferred publication date. Use posts mode with none/excerpt text settings; this source provides no post text.

### Output: export Substack data to CSV, Excel or JSON

The default dataset contains your results. The Output tab also provides **Coverage and errors** (`OUTPUT`) and **Incremental checkpoint** (`CHECKPOINT`).

Post records include publication details, title, URL, source date, available author names and tags, observation time and coverage. Unsupported values remain null. Feed entries can include audio/video; non-post items and archive restacks are labelled separately. Notes and replies are not returned.

Abbreviated post metadata example with text omitted:

```json
{
  "recordType": "post",
  "publicationTitle": "Simon Willison’s Newsletter",
  "title": "I built a new feature for my blog using my voice",
  "url": "https://simonw.substack.com/p/i-built-a-new-feature-for-my-blog",
  "publishedAt": "2026-10-12T22:29:32Z",
  "bodyText": null,
  "accessStatus": "unknown",
  "isPreviewOnly": null,
  "coverage": "current_feed_only",
  "archiveComplete": false,
  "changeType": "new"
}
```

Check `OUTPUT.outcome` alongside the Apify run status. `COMPLETE` means the observed source windows were processed; it never proves a complete archive. `PARTIAL` means a limit, missing target or source issue left work unfinished. `FAILED` means source failures prevented a usable result; `CANCELLED` records a cooperative stop. A successful ten-result run can legitimately report `PARTIAL` because it reached your result limit.

### Run Substack Scraper through the API with Python

Use the [Apify Python client](https://docs.apify.com/api/client/python) with an Apify token that has access to this Actor. This example uses the version 3.3 client API and caps the Actor charge at $0.005:

```python
from decimal import Decimal

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("thescrapelab/substack-scraper").call(
    run_input={
        "startUrls": [{"url": "https://simonw.substack.com"}],
        "maxItems": 10,
    },
    max_total_charge_usd=Decimal("0.005"),
)
if run is None:
    raise RuntimeError("No Actor run returned")

for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item["title"], item["url"])
```

The dataset iterator handles pagination. Inspect the run's `OUTPUT` report for coverage and source errors before treating the exported data as complete.

### Collect new posts and resume partial runs

Copy the previous **CHECKPOINT** into **Continue from checkpoint** (`resumeState`) for the next run. Keep the collection source, output mode, filters and text settings the same, including selected sitemap years. Result, request, page and time limits may change. Use `{}` for a new selection configuration. Independent runs do not automatically share state.

Feed continuation rechecks incomplete feeds and can conditionally recheck completed feeds. Archive continuation rescans from the first page and skips unchanged records; increase `maxArchivePages` to reach further back. Year-index continuation rechecks the selected years. Rechecking can use requests even when no new rows are delivered.

Recovering the same run reconciles existing dataset records before adding rows. Check the dataset and last checkpoint after interruption: valid rows may exist without a final summary. A forced interruption can prevent cleanup. When combining feed, archive and year-index exports, match by post URL; record IDs differ between sources.

### FAQ and troubleshooting

#### Can I scrape every Substack post or Substack Notes?

Complete archives, all-section coverage, platform-wide search, Notes and replies are unsupported. A post absent from the selected source window is not classified as deleted. Publication domain changes may require record identity reconciliation.

#### Can this scraper read paid articles?

It does not log in, bypass paywalls or fetch paid article bodies. Feed excerpts and archive subtitles may be public previews. Access and preview flags remain unknown when the source does not establish them.

#### Why did I get fewer results, no results or a partial result?

Open **Coverage and errors**. Check result limits, filters, unchanged checkpoint records, unavailable post URLs, source errors and access blocks. Raise **Results per newsletter** as well as **Total results** when needed. A blocked source stops collection. Avoid repeatedly starting fresh runs without reviewing the report and checkpoint.

Collection uses one source request at a time, at least 1.1 seconds apart. Limits include 100 input URLs, 120 source requests, five MB per source response/decompression and ten MB of record output per run. Smaller configured limits take priority. The work budget defaults to 90 seconds, with a 35-second reserve before the Apify timeout for saving progress. Set the platform timeout long enough for both. Temporary connection failures may retry once; retries count toward the request limit. No identity rotation or challenge bypass is used.

#### What permission do I need to use the data?

Use sources you are authorized to process and follow applicable platform and publisher conditions for reuse, attribution and redistribution. Public visibility alone does not establish commercial permission. No Substack API account is required by this Actor, and its experimental archive route is not an official supported API.

#### How do I get support?

Use the Actor's **Issues** tab when available, or the support contact provided with your private access. Include the run link, build number, sanitized input and a small example of expected versus observed metadata. Never send tokens, private feed URLs, paid article bodies or subscriber lists. Support is best effort, without a fixed response-time SLA. Direct Apify account and invoice questions to [Apify support](https://help.apify.com/).

#### How are results retained or deleted?

Review your dataset and default key-value store in Apify Console under **Storage**. They can contain results, input, summaries and checkpoints. Export what you need before using Apify's normal deletion controls; deleting a checkpoint removes its continuation history. Copies and exports elsewhere need separate handling.

The Actor does not set a universal retention period, automatically delete exports or interpret a missing source item as a deletion request. Follow the retention, refresh, attribution and deletion conditions attached to your authorized use.

# Actor input Schema

## `newsletterUrls` (type: `array`):

Paste public newsletter or post links, one per line. Custom domains and /feed links work too. Leave empty to use existing API startUrls, or the Simon Willison example by default. When non-empty, this list replaces startUrls. Hosted URL-list files, private feeds and Notes are unsupported.

## `maxItems` (type: `integer`):

Maximum saved results across all newsletters in this run. Collection stops when this limit or another run limit is reached. Increase Results per newsletter too if you want more than 10 posts from one newsletter.

## `maxPostsPerPublication` (type: `integer`):

Maximum new or updated posts from each newsletter, also limited by Total results. Recent feeds may contain fewer posts. Ignored when returning newsletter details.

## `keyword` (type: `string`):

Optional word or phrase in the post title or description. Leave blank for all posts. Filters only your supplied newsletters; it does not search all of Substack.

## `publishedAfter` (type: `string`):

Include posts published on or after this date, in UTC. Leave blank for no start date. JSON/API input also accepts ISO timestamps. Exact date filters are unavailable for year-index links.

## `publishedBefore` (type: `string`):

Include posts published before this date, in UTC. The selected day is excluded. Leave blank for no end date. JSON/API input also accepts ISO timestamps.

## `mode` (type: `string`):

Choose newsletter posts, or one details record per newsletter. Newsletter details require Recent posts as the source.

## `textMode` (type: `string`):

Short excerpt uses the feed description or archive subtitle. Feed text uses text already supplied in RSS/Atom; it may be incomplete or a preview. Year indexes provide no text.

## `maxTextCharacters` (type: `integer`):

Maximum plain-text characters per post. The default is 300. Truncation is reported in the output; this setting has no effect when text is omitted.

## `collection` (type: `string`):

Recent posts uses the current RSS/Atom feed. Older posts uses an experimental, undocumented archive route with limited coverage. Links by year returns titles and URLs from public year indexes. Older posts and year indexes require Newsletter posts and cannot include feed text.

## `maxArchivePages` (type: `integer`):

Only for Older posts. Each page requests up to 12 metadata entries, plus one feed lookup per newsletter. Two pages is the default. Raise the result limits as well to return more posts. Continuing rescans these pages and skips unchanged records.

## `sitemapYears` (type: `array`):

Only for Post links by year: enter one to five years, such as \[2026]. These indexes return titles and URLs, without exact dates, text, authors or engagement. Leave empty for the other sources.

## `maxRequests` (type: `integer`):

Maximum source requests, including redirects, retries and publication lookups. The default is 20. Requests run one at a time, at least 1.1 seconds apart. Collection stops on denial, throttling or a challenge.

## `maxRunSeconds` (type: `integer`):

Maximum collection work time; the default is 90 seconds. The Actor also reserves 35 seconds before the Apify timeout to save progress. A limit can return partial results with a checkpoint. Set the Apify timeout long enough for collection and cleanup.

## `resumeState` (type: `object`):

Leave {} for a new selection. To collect new or changed records, paste CHECKPOINT JSON from an earlier run with the same source, filters, output and text settings. Limits may change. State is not automatically shared between runs.

## `startUrls` (type: `array`):

API-compatible array of publication URL strings, bare names such as simonw, objects containing only url, or mixed entries. Used when newsletterUrls is empty or omitted. Up to 100 entries.

## Actor input object example

```json
{
  "newsletterUrls": [],
  "maxItems": 10,
  "maxPostsPerPublication": 10,
  "keyword": "",
  "mode": "posts",
  "textMode": "excerpt",
  "maxTextCharacters": 300,
  "collection": "feed",
  "maxArchivePages": 2,
  "sitemapYears": [],
  "maxRequests": 20,
  "maxRunSeconds": 90,
  "resumeState": {},
  "startUrls": [
    {
      "url": "https://simonw.substack.com"
    }
  ]
}
```

# Actor output Schema

## `datasetItems` (type: `string`):

Structured records delivered to the default dataset.

## `runSummary` (type: `string`):

Request metrics, source coverage, limits and errors. Empty results are not evidence that an old post was deleted.

## `checkpoint` (type: `string`):

Copy this JSON into resumeState for a subsequent run with matching filters and text settings.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "newsletterUrls": [],
    "startUrls": [
        {
            "url": "https://simonw.substack.com"
        }
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("thescrapelab/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "newsletterUrls": [],
    "startUrls": [{ "url": "https://simonw.substack.com" }],
}

# Run the Actor and wait for it to finish
run = client.actor("thescrapelab/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "newsletterUrls": [],
  "startUrls": [
    {
      "url": "https://simonw.substack.com"
    }
  ]
}' |
apify call thescrapelab/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,thescrapelab/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/70ma24L3UhHbFgx3q/builds/xEsgWO0V5V1iUCQKs/openapi.json