# Substack Scraper - Posts, Markdown & New-Post Feed (`cedarfetch/substack-scraper`) Actor

Extract public posts from any Substack newsletter, custom domains included: title, date, authors, audience, paywall flag, likes, comment counts, podcast audio and the body as Markdown, text or HTML. Built-in new-posts feed between runs and a publication info row with plan prices.

- **URL**: https://apify.com/cedarfetch/substack-scraper.md
- **Developed by:** [Cedarfetch Labs](https://apify.com/cedarfetch) (community)
- **Features:** Limited permissions
- **Stats:** 1 total users, 0 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.00 / 1,000 posts

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper - Posts, Markdown & New-Post Feed

Extract the public posts of any Substack newsletter, on `name.substack.com` or a custom domain: title, subtitle, date, authors, post type, audience, paywall flag, likes, comment and restack counts, cover image, podcast audio URL and length, tags, section, and the post body as **Markdown**, plain text or the original HTML. Add one info row per newsletter with its description, author, language, logo, paid plan prices and post count.

Run it on a schedule with **change tracking** on and it becomes a feed: every row is labelled `new`, `updated`, `unchanged` or `removed`, and "Output only new and updated posts" delivers (and charges for) only what changed since the last run.

It reads only the public, per-newsletter endpoints that any visitor's browser loads (`/api/v1/archive`, `/api/v1/posts/<slug>`, the homepage and `/sitemap.xml`). No login, no cookies, no browser, no proxies, so runs are fast and cheap.

### What you can use it for

- **Building a RAG or AI knowledge base** from newsletters: clean Markdown with headings, links, lists, quotes, code and footnotes, and Substack's buttons and widgets removed.
- **Content and competitor research**: what a newsletter publishes, how often, how long, which posts get the most likes and comments.
- **Media monitoring**: a daily "new posts since yesterday" feed across many newsletters.
- **Sponsorship research**: which newsletters run paid plans and at what price, how many posts, podcast or not.
- **Archiving and analysis**: export a whole archive to JSON, CSV or Excel.

### Why this one

- **Markdown that is ready for an LLM.** Not a raw HTML dump: footnotes become `[^1]` references with definitions at the end, embeds become links (YouTube, X posts, linked Substack posts), image captions are kept, subscribe buttons and share widgets are dropped.
- **Honest paywall handling.** Paid and subscriber-only posts are never "unlocked". You get exactly what Substack shows an anonymous visitor (the free preview), clearly marked `isPaywalled: true` and `contentAvailable: "preview"`, plus the full post's word count. `onlyFree` skips them entirely.
- **Complete archives.** Many publishers hide whole sections (and often their podcast) from the main archive. By default the Actor also reads every hidden section; with `coverage: "everything"` it adds the remaining posts from the public sitemap. The run summary tells you when the sitemap shows more posts than the listings.
- **A new-posts feed built in.** Change tracking keeps a compact snapshot per newsletter in your account. `changesOnly` downloads bodies only for new and updated posts, so a daily feed of 50 newsletters costs a few requests each.
- **Clear failures.** Not a Substack newsletter, misspelled, deleted, blocked or rate-limited newsletters are reported per newsletter with a plain explanation, never as a silent empty dataset. If nothing at all could be extracted, the run fails with the reasons.
- **Polite by default.** One request at a time per newsletter with a 1 second pause, automatic slowdown when Substack answers "too many requests".

### Input

Only `publications` is required. Paste newsletters in any form:

| You paste | What is read |
|---|---|
| `simonw`, `simonw.substack.com`, `https://simonw.substack.com/archive` | The whole newsletter |
| `www.lennysnewsletter.com`, `https://www.noahpinion.blog/` | The whole newsletter (custom domain) |
| `https://www.noahpinion.blog/p/some-post` | Only that post |
| `https://open.substack.com/pub/simonw/p/some-post` | Only that post |

Links on `substack.com` itself (profiles like `substack.com/@name`, the app feed, search, Notes) are rejected with an explanation: use the newsletter's own address.

#### Example: the 20 latest posts of three newsletters as Markdown

```json
{
    "publications": ["simonw.substack.com", "www.lennysnewsletter.com", "www.noahpinion.blog"],
    "maxPostsPerPublication": 20
}
```

#### Example: everything published in the last 30 days, free posts only, plain text

```json
{
    "publications": ["garymarcus.substack.com", "www.dwarkesh.com"],
    "postedAfter": "30 days",
    "onlyFree": true,
    "contentFormat": "text"
}
```

#### Example: podcast episodes only, metadata without bodies

```json
{
    "publications": ["www.dwarkesh.com", "www.lennysnewsletter.com"],
    "postTypes": ["podcast"],
    "coverage": "everything",
    "includeContent": false
}
```

#### All options

| Field | Default | What it does |
|---|---|---|
| `publications` | (required) | Newsletters or post URLs. |
| `maxPostsPerPublication` | 0 (all) | Stop after this many matching posts per newsletter. |
| `sort` | `new` | `new`: newest first. `top`: most popular first, as Substack ranks them. |
| `coverage` | `archive_and_sections` | `archive_and_sections`: the main archive plus sections the publisher hid from it. `everything`: also any other post in the public sitemap (one request per such post). `archive`: the main archive only. |
| `postedAfter` | none | ISO date (`2026-11-01`) or relative (`30 days`, `2 weeks`, `24 hours`, `6 months`). With `sort: new`, reading stops at the first older post. |
| `postedBefore` | none | Same formats. |
| `includeContent` | true | Fetch each post's body (one request per post). Off: metadata only, about one request per 25 posts. |
| `contentFormat` | `markdown` | `markdown`, `text` or `html`. |
| `postTypes` | all | Any of `newsletter`, `podcast`, `thread`, `video`. |
| `onlyFree` | false | Skip paywalled and subscriber-only posts. |
| `includePublicationInfo` | true | One free `publication` row per newsletter. |
| `trackChanges` | false | Label every post `new` / `updated` / `unchanged` / `removed` against the previous run. |
| `changesOnly` | false | With tracking on, skip unchanged posts. |
| `snapshotStoreName` | `substack-scraper-snapshots` | Named key-value store for the snapshots. Use different names for independent feeds. |
| `concurrency` | 1 | Parallel requests per newsletter. |
| `requestDelayMs` | 1000 | Pause after each request to the same newsletter (adapts automatically after a rate limit). |
| `maxConcurrentPublications` | 3 | Newsletters read at the same time. |
| `requestTimeoutSecs` | 30 | Timeout for one request. |
| `maxRetries` | 4 | Retries for 429, 5xx and network errors. |

### Output

Rows go to the default dataset. Download them as JSON, CSV, Excel, XML or RSS from the Output tab or the API. The dataset has five views: **Posts**, **Content**, **Changes**, **Publications** and **Newsletters that could not be read**. Every row has a `recordType`.

#### Post row (`recordType: "post"`)

```json
{
    "recordType": "post",
    "publicationDomain": "www.lennysnewsletter.com",
    "publicationName": "Lenny's Newsletter",
    "publicationSubdomain": "lenny",
    "publicationId": 10845,
    "postId": 224206488,
    "slug": "state-of-the-tech-job-market-late",
    "url": "https://www.lennysnewsletter.com/p/state-of-the-tech-job-market-late",
    "title": "State of the tech job market: late 2026",
    "subtitle": "(the post's subtitle)",
    "type": "newsletter",
    "audience": "only_paid",
    "isPaywalled": true,
    "requiresPaidSubscription": true,
    "contentAvailable": "preview",
    "postDate": "2026-11-10T15:04:03.497Z",
    "updatedAt": "2026-11-10T15:09:41.796Z",
    "authors": [{ "name": "Lenny Rachitsky", "handle": "lenny" }],
    "wordcount": 2183,
    "reactionCount": 261,
    "commentCount": 24,
    "restackCount": 24,
    "coverImage": "https://substackcdn.com/image/fetch/...png",
    "podcastUrl": null,
    "podcastDurationSecs": null,
    "hasVideo": false,
    "tags": ["Career"],
    "section": null,
    "description": "(the post's description)",
    "previewText": "(the first lines, as Substack's archive shows them)",
    "language": "en",
    "contentFormat": "markdown",
    "content": "(the public preview as Markdown; the full body for free posts)",
    "source": "post",
    "scrapedAt": "2026-11-11T18:55:07.738Z"
}
```

(Values shortened.) Notes on the fields:

- `audience`: `everyone` (free), `only_subscribers` (free signup required), `only_paid`, `founding`. `isPaywalled` is true for anything but `everyone`; `requiresPaidSubscription` for `only_paid` and `founding`.
- `contentAvailable`: `full` (free post, body included), `preview` (gated post: only the public preview), `none` (content off, or the post has no body).
- `wordcount` is the full post's length as Substack reports it, also for paywalled posts, so you can see how much of a post the preview covers.
- `commentCount` is a number only. Comment text and commenters are never collected.
- `authors` holds the public byline names and handles shown on the post, nothing else.
- Podcast posts have `type: "podcast"`, `podcastUrl` (the audio file URL from the post's public payload) and `podcastDurationSecs`.
- `source`: `post` when the row came from the single-post endpoint (with `updatedAt`), `archive` when only listing data was needed.

#### Publication row (`recordType: "publication"`, free)

One per newsletter, before its posts:

```json
{
    "recordType": "publication",
    "publicationDomain": "www.lennysnewsletter.com",
    "publicationName": "Lenny's Newsletter",
    "publicationSubdomain": "lenny",
    "publicationId": 10845,
    "url": "https://www.lennysnewsletter.com",
    "customDomain": "www.lennysnewsletter.com",
    "description": "(the newsletter's tagline)",
    "authorName": "Lenny Rachitsky",
    "authorHandle": "lenny",
    "language": "en",
    "logoUrl": "https://substackcdn.com/image/fetch/...png",
    "type": "newsletter",
    "paidSubscriptionsEnabled": true,
    "plans": [
        { "planId": "monthly20usd", "name": "$20 a month", "interval": "month", "price": 20, "currency": "USD", "isFounding": false },
        { "planId": "yearly200usd", "name": "$200 a year", "interval": "year", "price": 200, "currency": "USD", "isFounding": false },
        { "planId": "founding40000usd", "name": "Insider Tier", "interval": "year", "price": 400, "currency": "USD", "isFounding": true }
    ],
    "monthlyPrice": 20,
    "annualPrice": 200,
    "foundingPrice": 400,
    "priceCurrency": "USD",
    "hasPodcast": true,
    "firstPostDate": "2019-06-14T15:00:48.833Z",
    "createdAt": "2019-06-01T15:35:37.885Z",
    "postCount": 1296,
    "scrapedAt": "2026-11-11T18:55:07.738Z"
}
```

`postCount` is the number of post URLs in the newsletter's public sitemap. Prices are the plan prices in the newsletter's own currency (`foundingPrice` is the minimum founding-member price per year). Subscriber counts are not included.

#### Newsletters that could not be read (`recordType: "publication_status"`, free)

| status | Meaning |
|---|---|
| `not_substack` | The site does not serve Substack's public endpoints (never was Substack, or moved off it). |
| `publication_not_found` | Substack has no newsletter at this address. |
| `blocked` | The site refused access (401/403) or showed a bot-protection page. Skipped, never worked around. |
| `rate_limited` | Substack kept answering 429 after polite retries and an automatic slowdown. Try later. |
| `unreachable` | The domain does not resolve, or TLS/network failed. |
| `invalid_input` | Not a usable newsletter or post URL (for example a substack.com profile link). |
| `no_posts` | A Substack newsletter with no public posts, or none of the requested post URLs exist. |

#### Run summary

The default key-value store record `OUTPUT` holds a summary: per newsletter the status, posts found / matched / delivered, paywalled count, which hidden sections were read, the sitemap post count, requests, retries and their reasons, rate-limit slowdowns, duration, warnings, and change counts. Use it to monitor scheduled runs.

### Recipe: a "new posts since last run" feed

1. Input:
   ```json
   {
       "publications": ["simonw.substack.com", "www.lennysnewsletter.com", "www.noahpinion.blog"],
       "trackChanges": true,
       "changesOnly": true,
       "postedAfter": "14 days"
   }
   ```
2. Run it once. Every post of the last 14 days arrives labelled `new` (this is the baseline).
3. Create a schedule (Schedules -> daily). From then on each run delivers only posts that are new or whose title, subtitle, audience or length changed, and you pay only for those. A day with nothing new delivers zero post rows and succeeds.
4. Connect the dataset to Slack, email, Make, Zapier or a webhook through Apify integrations, or poll `.../runs/last/dataset/items?view=changes`.

`postedAfter` keeps each daily run to a few requests per newsletter. Removed posts are only reported by runs that read the whole archive (no `postedAfter`, no cap); see below.

### Recipe: build a RAG corpus from newsletters

1. Input:
   ```json
   {
       "publications": ["simonw.substack.com", "www.astralcodexten.com"],
       "onlyFree": true,
       "contentFormat": "markdown",
       "includePublicationInfo": false
   }
   ```
2. Download with the **Content** view (`?view=content`) or take the full rows from the API.
3. Use one document per row: `content` as the text; `title`, `url`, `postDate`, `authors` and `publicationName` as metadata for citations. Split on Markdown headings (`##`), which the converter keeps.
4. Keep it fresh: add `trackChanges` + `changesOnly` and schedule it; re-embed rows with `changeType` `new` or `updated`, delete `removed` ones.

If you include paywalled posts, filter on `contentAvailable == "full"` so previews do not end up in the corpus as if they were whole posts.

### Change tracking

Turn on **Track changes between runs** and schedule the Actor. Each run compares every post with the snapshot from the previous run of the same newsletter and adds:

| Field | Meaning |
|---|---|
| `changeType` | `new`, `updated`, `unchanged` or `removed`. |
| `changes` | Everything that applies: `new`, `title_changed`, `subtitle_changed`, `audience_changed` (for example a post moved behind the paywall), `slug_changed`, `wordcount_changed`, and `edited` (the post's updated time moved, only seen when the body is fetched). |
| `previousTitle`, `previousAudience`, `previousWordcount`, `previousUpdatedAt` | Last run's values. |

Rules, so you can rely on the labels:

- **The first tracked run labels everything `new`.** It creates the baseline.
- **`updated`** means the title, subtitle, audience, slug or word count changed. An edit that changes none of them shows `edited` in `changes` while `changeType` stays `unchanged`. Likes and comment counts are not treated as changes.
- **`removed`** rows (with the last known title and URL) are emitted only after a **complete** read: no cap, no `postedAfter` stop, no fetch errors, not stopped by the charge limit, and the same `coverage` as the previous run. If more than half of the known posts vanished at once, the Actor assumes a glitch and reports nothing; the next complete run decides.
- **Nothing is lost on a budget stop.** The snapshot is updated only for rows actually delivered, so a change that did not fit in this run is reported in the next one.
- Single-post inputs are tracked in their own snapshot.
- Snapshots live in the named key-value store `snapshotStoreName` in your account (compressed, one record per newsletter). Delete it, or pick a new name, to start over.

### Pricing

Pay per event: a small fee per run start and per post row delivered to the dataset (`post` event). See the Pricing tab for current prices.

- One price per post row, with or without the body.
- `removed` rows from change tracking count as post rows; with `changesOnly`, unchanged posts are not delivered and not charged.
- Publication info rows and rows for newsletters that could not be read are free.
- Set **Maximum cost per run** in the run options to cap spending. The Actor stops cleanly at the cap, keeps everything delivered so far, does not download bodies it cannot deliver, and notes it in `OUTPUT`.

### Use from an AI agent or via API

Run synchronously and get the rows back in one HTTP call:

```bash
curl -X POST "https://api.apify.com/v2/acts/<username>~substack-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"publications":["simonw.substack.com"],"maxPostsPerPublication":10,"onlyFree":true}'
```

JavaScript:

```js
import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('<username>/substack-scraper').call({ publications: ['www.noahpinion.blog'], postedAfter: '7 days' });
const { items } = await client.dataset(run.defaultDatasetId).listItems();
```

Python:

```python
from apify_client import ApifyClient
client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("<username>/substack-scraper").call(run_input={"publications": ["www.dwarkesh.com"], "postTypes": ["podcast"]})
items = client.dataset(run["defaultDatasetId"]).list_items().items
```

The Actor is also available to agents through the Apify MCP server, which exposes Store Actors as tools. Tips for agents: keep `maxPostsPerPublication` small for exploratory calls, use `includeContent: false` when you only need titles and dates, read `recordType` to tell post rows from `publication` and `publication_status` rows, and check `contentAvailable` before treating `content` as a whole post.

### Limits and honest caveats

- **Paywalled posts are preview only.** The Actor never logs in and never tries to get around a paywall. For paid and subscriber-only posts you get what an anonymous visitor sees: usually the opening of the post, sometimes nothing beyond the title and description.
- **No comments, no subscriber or reader data.** Comment counts only. Subscriber counts, reader lists, emails and commenter details are not collected.
- **No discovery or search.** You tell it which newsletters to read. Substack's global search, category, leaderboard and profile pages live on substack.com and are not used (they refuse anonymous API clients).
- **Notes, chats and live streams** are not posts and are not included.
- **Hidden posts.** Posts a publisher hides from every listing are only found with `coverage: "everything"`, which costs one request per such post. On some newsletters (often those with a podcast) that is hundreds of extra requests.
- **Rate limits.** Substack throttles clients that read fast. The defaults stay under it in testing; if you raise `concurrency` or lower `requestDelayMs`, the Actor slows itself down on the first 429, and a newsletter that stays throttled is reported as `rate_limited` with everything read so far delivered. Large archives with bodies take time: about 1.4 seconds per post at the defaults.
- **`sort: "top"`** is Substack's own popularity order. With `postedAfter` it cannot stop early, so it reads the whole archive listing (still about one request per 25 posts).
- Explicit post URLs bypass `postedAfter`, `postedBefore`, `postTypes`, `onlyFree` and the cap: you asked for that post.
- Markdown conversion handles what Substack's editor produces. Unusual embeds (polls, audio players, paid-only widgets) are dropped or reduced to a link.

### FAQ

**Does it work with custom domains?** Yes. Paste the domain; the Actor detects Substack from the public API and response headers. A site that is not (or no longer) on Substack is reported as `not_substack` after one request and is not charged.

**Can I get the full text of paid posts if I am a subscriber?** No. The Actor does not accept logins or cookies and only reads public content.

**How do I get only new posts every day?** See the feed recipe above: `trackChanges` + `changesOnly` + a schedule.

**Why are there more posts in `postCount` than rows?** `postCount` counts the public sitemap. Posts hidden from every listing need `coverage: "everything"`. The `OUTPUT` summary warns when the gap is large.

**How fast is it?** Metadata only: about 25 posts per request, a 900-post archive (plus its hidden sections) in about a minute. With bodies: one request per post at a polite pace.

**Is this legal?** The Actor collects publicly available newsletter content and public byline names, nothing personal beyond that. You are responsible for how you use the data, for respecting copyright and each newsletter's terms. Republishing other people's posts usually needs their permission. If in doubt, ask a lawyer.

### Support

Found a newsletter that behaves oddly? Open an issue on the Issues tab with the newsletter address and the run ID; the `OUTPUT` record usually shows what happened.

### More Actors from Cedarfetch Labs

Built the same way: public data only, typed output, per-source error reporting, change tracking for scheduled runs.

- [Shopify Store Scraper](https://apify.com/cedarfetch/shopify-store-scraper): full product catalogs from any Shopify store, with price and stock change tracking.
- [ATS Jobs Scraper](https://apify.com/cedarfetch/ats-jobs-scraper): open jobs from Greenhouse, Lever, Ashby, SmartRecruiters, Workable, Recruitee and Personio boards, with new and removed postings flagged.
- [Remote Jobs Scraper](https://apify.com/cedarfetch/remote-jobs-scraper): remote jobs from Remote OK, We Work Remotely and five more boards in one deduplicated feed.
- [Google Trending Searches](https://apify.com/cedarfetch/google-trending-searches): what is trending on Google right now, per country, with alerts on new trends.

# Changelog

This Actor's version history is a separate document: https://apify.com/cedarfetch/substack-scraper/changelog.md

# Actor input Schema

## `publications` (type: `array`):

One per line. Accepts a Substack subdomain (simonw.substack.com, or just simonw), a custom domain (www.lennysnewsletter.com), a full URL (https://www.noahpinion.blog/archive), or a post URL (https://simonw.substack.com/p/some-post or open.substack.com/pub/simonw/p/some-post) to read only that post. Links on substack.com itself (profiles, the app feed, search) are not supported.

## `maxPostsPerPublication` (type: `integer`):

Stop after this many matching posts per newsletter. 0 means the whole archive. Keeps test runs and samples cheap.

## `sort` (type: `string`):

"new": newest first (fastest with a postedAfter date, because reading stops at the first older post). "top": the newsletter's most popular posts first, as Substack ranks them.

## `coverage` (type: `string`):

Publishers can hide posts from their main archive (whole sections, or podcast episodes). "archive_and_sections" (default): the main archive plus every section hidden from it; cheap, 1 request per 25 posts. "everything": also every other post in the public sitemap that no listing shows (often podcast episodes); 1 extra request per such post, so slower on big newsletters. "archive": only what the main /archive page lists.

## `postedAfter` (type: `string`):

Optional. Keep only posts published on or after this moment. An ISO date (2026-11-01 or 2026-11-01T12:00:00Z) or a relative value such as "30 days", "2 weeks", "24 hours" or "6 months".

## `postedBefore` (type: `string`):

Optional. Keep only posts published before this moment. Same formats as "Only posts published after".

## `includeContent` (type: `boolean`):

Fetch each post's body (one extra request per post). Free posts come in full. Paywalled posts contain only the public preview Substack shows to visitors, marked isPaywalled: true and contentAvailable: "preview". Turn off for a fast metadata-only list (about one request per 25 posts).

## `contentFormat` (type: `string`):

"markdown": headings, links, lists, quotes, code, images and footnotes as Markdown, Substack's buttons and widgets removed (best for AI and RAG). "text": plain text. "html": the body exactly as Substack serves it.

## `postTypes` (type: `array`):

Optional. Keep only these types as Substack labels them: newsletter (regular posts), podcast, thread, video. Empty means all types.

## `onlyFree` (type: `boolean`):

Skip posts that are paywalled or require a subscription, so every delivered row has the full body.

## `includePublicationInfo` (type: `boolean`):

Add one free row per newsletter (recordType "publication") with its name, description, author name, language, logo, whether paid subscriptions are on, public plan prices and the number of posts in its sitemap. Costs one extra request per newsletter.

## `trackChanges` (type: `boolean`):

Compare every post with the previous run and add changeType (new, updated, unchanged, removed), the list of changes, and previous values. The snapshot is kept in a named key-value store in your account, so it survives between runs. Schedule the Actor to get a daily feed.

## `changesOnly` (type: `boolean`):

Requires change tracking. Skips unchanged posts, so you receive (and pay for) only new, updated and removed posts: a "new posts since last run" feed. Bodies are downloaded only for those posts.

## `snapshotStoreName` (type: `string`):

Name of the key-value store that holds the change-tracking snapshots. Use a different name to run independent feeds on the same newsletters. Lowercase letters, digits and hyphens.

## `concurrency` (type: `integer`):

How many requests run at once against one newsletter. Substack rate-limits busy clients (HTTP 429): 1 request at a time with a 1 s pause ran clean in testing, 2 at a time with 0.5 s hit rate limits after about 100 requests. On a 429 the Actor slows itself down automatically.

## `requestDelayMs` (type: `integer`):

Pause after each request to the same newsletter, with small random jitter. Doubled automatically (up to 10 s) after a rate-limit answer and eased back after 20 clean answers.

## `maxConcurrentPublications` (type: `integer`):

How many different newsletters are read at the same time.

## `requestTimeoutSecs` (type: `integer`):

Give up on a single request after this many seconds (it is then retried).

## `maxRetries` (type: `integer`):

Retries for rate-limit (429), server-error (5xx) and network failures, with jittered exponential backoff that honours Retry-After. 401/403 and bot-protection pages are never retried.

## Actor input object example

```json
{
  "publications": [
    "simonw.substack.com",
    "www.lennysnewsletter.com",
    "https://www.noahpinion.blog/p/normal-americans-punished-maga"
  ],
  "maxPostsPerPublication": 100,
  "sort": "new",
  "coverage": "archive_and_sections",
  "postedAfter": "30 days",
  "postedBefore": "2026-01-01",
  "includeContent": true,
  "contentFormat": "markdown",
  "postTypes": [],
  "onlyFree": false,
  "includePublicationInfo": true,
  "trackChanges": false,
  "changesOnly": false,
  "snapshotStoreName": "substack-scraper-snapshots",
  "concurrency": 1,
  "requestDelayMs": 1000,
  "maxConcurrentPublications": 3,
  "requestTimeoutSecs": 30,
  "maxRetries": 4
}
```

# Actor output Schema

## `posts` (type: `string`):

Post rows, one publication info row per newsletter, and publication_status rows for newsletters that could not be read.

## `content` (type: `string`):

The same rows in the content view (title, URL, date, paywall flag, body).

## `changes` (type: `string`):

The rows shown with the change-tracking columns.

## `summary` (type: `string`):

Per-publication status (ok, partial, blocked, not_substack, publication_not_found, unreachable...), counts, request totals and change counts.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publications": [
        "simonw.substack.com"
    ],
    "maxPostsPerPublication": 5
};

// Run the Actor and wait for it to finish
const run = await client.actor("cedarfetch/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "publications": ["simonw.substack.com"],
    "maxPostsPerPublication": 5,
}

# Run the Actor and wait for it to finish
run = client.actor("cedarfetch/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publications": [
    "simonw.substack.com"
  ],
  "maxPostsPerPublication": 5
}' |
apify call cedarfetch/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,cedarfetch/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/fgzibT0CDTi4BRaY5/builds/5WzAglmITh02sTsGa/openapi.json