Substack Scraper - Posts, Markdown & New-Post Feed

Extract the public posts of any Substack newsletter, on name.substack.com or a custom domain: title, subtitle, date, authors, post type, audience, paywall flag, likes, comment and restack counts, cover image, podcast audio URL and length, tags, section, and the post body as Markdown, plain text or the original HTML. Add one info row per newsletter with its description, author, language, logo, paid plan prices and post count.

Run it on a schedule with change tracking on and it becomes a feed: every row is labelled new, updated, unchanged or removed, and "Output only new and updated posts" delivers (and charges for) only what changed since the last run.

It reads only the public, per-newsletter endpoints that any visitor's browser loads (/api/v1/archive, /api/v1/posts/<slug>, the homepage and /sitemap.xml). No login, no cookies, no browser, no proxies, so runs are fast and cheap.

What you can use it for

• Building a RAG or AI knowledge base from newsletters: clean Markdown with headings, links, lists, quotes, code and footnotes, and Substack's buttons and widgets removed.
• Content and competitor research: what a newsletter publishes, how often, how long, which posts get the most likes and comments.
• Media monitoring: a daily "new posts since yesterday" feed across many newsletters.
• Sponsorship research: which newsletters run paid plans and at what price, how many posts, podcast or not.
• Archiving and analysis: export a whole archive to JSON, CSV or Excel.

Why this one

• Markdown that is ready for an LLM. Not a raw HTML dump: footnotes become [^1] references with definitions at the end, embeds become links (YouTube, X posts, linked Substack posts), image captions are kept, subscribe buttons and share widgets are dropped.
• Honest paywall handling. Paid and subscriber-only posts are never "unlocked". You get exactly what Substack shows an anonymous visitor (the free preview), clearly marked isPaywalled: true and contentAvailable: "preview", plus the full post's word count. onlyFree skips them entirely.
• Complete archives. Many publishers hide whole sections (and often their podcast) from the main archive. By default the Actor also reads every hidden section; with coverage: "everything" it adds the remaining posts from the public sitemap. The run summary tells you when the sitemap shows more posts than the listings.
• A new-posts feed built in. Change tracking keeps a compact snapshot per newsletter in your account. changesOnly downloads bodies only for new and updated posts, so a daily feed of 50 newsletters costs a few requests each.
• Clear failures. Not a Substack newsletter, misspelled, deleted, blocked or rate-limited newsletters are reported per newsletter with a plain explanation, never as a silent empty dataset. If nothing at all could be extracted, the run fails with the reasons.
• Polite by default. One request at a time per newsletter with a 1 second pause, automatic slowdown when Substack answers "too many requests".

Input

Only publications is required. Paste newsletters in any form:

You paste	What is read
simonw, simonw.substack.com, https://simonw.substack.com/archive	The whole newsletter
www.lennysnewsletter.com, https://www.noahpinion.blog/	The whole newsletter (custom domain)
https://www.noahpinion.blog/p/some-post	Only that post
https://open.substack.com/pub/simonw/p/some-post	Only that post

Links on substack.com itself (profiles like substack.com/@name, the app feed, search, Notes) are rejected with an explanation: use the newsletter's own address.

Example: the 20 latest posts of three newsletters as Markdown

{
    "publications": ["simonw.substack.com", "www.lennysnewsletter.com", "www.noahpinion.blog"],
    "maxPostsPerPublication": 20
}

Example: everything published in the last 30 days, free posts only, plain text

{
    "publications": ["garymarcus.substack.com", "www.dwarkesh.com"],
    "postedAfter": "30 days",
    "onlyFree": true,
    "contentFormat": "text"
}

Example: podcast episodes only, metadata without bodies

{
    "publications": ["www.dwarkesh.com", "www.lennysnewsletter.com"],
    "postTypes": ["podcast"],
    "coverage": "everything",
    "includeContent": false
}

All options

Field	Default	What it does
publications	(required)	Newsletters or post URLs.
maxPostsPerPublication	0 (all)	Stop after this many matching posts per newsletter.
sort	new	new: newest first. top: most popular first, as Substack ranks them.
coverage	archive_and_sections	archive_and_sections: the main archive plus sections the publisher hid from it. everything: also any other post in the public sitemap (one request per such post). archive: the main archive only.
postedAfter	none	ISO date (2026-11-01) or relative (30 days, 2 weeks, 24 hours, 6 months). With sort: new, reading stops at the first older post.
postedBefore	none	Same formats.
includeContent	true	Fetch each post's body (one request per post). Off: metadata only, about one request per 25 posts.
contentFormat	markdown	markdown, text or html.
postTypes	all	Any of newsletter, podcast, thread, video.
onlyFree	false	Skip paywalled and subscriber-only posts.
includePublicationInfo	true	One free publication row per newsletter.
trackChanges	false	Label every post new / updated / unchanged / removed against the previous run.
changesOnly	false	With tracking on, skip unchanged posts.
snapshotStoreName	substack-scraper-snapshots	Named key-value store for the snapshots. Use different names for independent feeds.
concurrency	1	Parallel requests per newsletter.
requestDelayMs	1000	Pause after each request to the same newsletter (adapts automatically after a rate limit).
maxConcurrentPublications	3	Newsletters read at the same time.
requestTimeoutSecs	30	Timeout for one request.
maxRetries	4	Retries for 429, 5xx and network errors.

Output

Rows go to the default dataset. Download them as JSON, CSV, Excel, XML or RSS from the Output tab or the API. The dataset has five views: Posts, Content, Changes, Publications and Newsletters that could not be read. Every row has a recordType.

Post row (recordType: "post")

{
    "recordType": "post",
    "publicationDomain": "www.lennysnewsletter.com",
    "publicationName": "Lenny's Newsletter",
    "publicationSubdomain": "lenny",
    "publicationId": 10845,
    "postId": 224206488,
    "slug": "state-of-the-tech-job-market-late",
    "url": "https://www.lennysnewsletter.com/p/state-of-the-tech-job-market-late",
    "title": "State of the tech job market: late 2026",
    "subtitle": "(the post's subtitle)",
    "type": "newsletter",
    "audience": "only_paid",
    "isPaywalled": true,
    "requiresPaidSubscription": true,
    "contentAvailable": "preview",
    "postDate": "2026-11-10T15:04:03.497Z",
    "updatedAt": "2026-11-10T15:09:41.796Z",
    "authors": [{ "name": "Lenny Rachitsky", "handle": "lenny" }],
    "wordcount": 2183,
    "reactionCount": 261,
    "commentCount": 24,
    "restackCount": 24,
    "coverImage": "https://substackcdn.com/image/fetch/...png",
    "podcastUrl": null,
    "podcastDurationSecs": null,
    "hasVideo": false,
    "tags": ["Career"],
    "section": null,
    "description": "(the post's description)",
    "previewText": "(the first lines, as Substack's archive shows them)",
    "language": "en",
    "contentFormat": "markdown",
    "content": "(the public preview as Markdown; the full body for free posts)",
    "source": "post",
    "scrapedAt": "2026-11-11T18:55:07.738Z"
}

(Values shortened.) Notes on the fields:

• audience: everyone (free), only_subscribers (free signup required), only_paid, founding. isPaywalled is true for anything but everyone; requiresPaidSubscription for only_paid and founding.
• contentAvailable: full (free post, body included), preview (gated post: only the public preview), none (content off, or the post has no body).
• wordcount is the full post's length as Substack reports it, also for paywalled posts, so you can see how much of a post the preview covers.
• commentCount is a number only. Comment text and commenters are never collected.
• authors holds the public byline names and handles shown on the post, nothing else.
• Podcast posts have type: "podcast", podcastUrl (the audio file URL from the post's public payload) and podcastDurationSecs.
• source: post when the row came from the single-post endpoint (with updatedAt), archive when only listing data was needed.

Publication row (recordType: "publication", free)

One per newsletter, before its posts:

{
    "recordType": "publication",
    "publicationDomain": "www.lennysnewsletter.com",
    "publicationName": "Lenny's Newsletter",
    "publicationSubdomain": "lenny",
    "publicationId": 10845,
    "url": "https://www.lennysnewsletter.com",
    "customDomain": "www.lennysnewsletter.com",
    "description": "(the newsletter's tagline)",
    "authorName": "Lenny Rachitsky",
    "authorHandle": "lenny",
    "language": "en",
    "logoUrl": "https://substackcdn.com/image/fetch/...png",
    "type": "newsletter",
    "paidSubscriptionsEnabled": true,
    "plans": [
        { "planId": "monthly20usd", "name": "$20 a month", "interval": "month", "price": 20, "currency": "USD", "isFounding": false },
        { "planId": "yearly200usd", "name": "$200 a year", "interval": "year", "price": 200, "currency": "USD", "isFounding": false },
        { "planId": "founding40000usd", "name": "Insider Tier", "interval": "year", "price": 400, "currency": "USD", "isFounding": true }
    ],
    "monthlyPrice": 20,
    "annualPrice": 200,
    "foundingPrice": 400,
    "priceCurrency": "USD",
    "hasPodcast": true,
    "firstPostDate": "2019-06-14T15:00:48.833Z",
    "createdAt": "2019-06-01T15:35:37.885Z",
    "postCount": 1296,
    "scrapedAt": "2026-11-11T18:55:07.738Z"
}

postCount is the number of post URLs in the newsletter's public sitemap. Prices are the plan prices in the newsletter's own currency (foundingPrice is the minimum founding-member price per year). Subscriber counts are not included.

Newsletters that could not be read (recordType: "publication_status", free)

status	Meaning
not_substack	The site does not serve Substack's public endpoints (never was Substack, or moved off it).
publication_not_found	Substack has no newsletter at this address.
blocked	The site refused access (401/403) or showed a bot-protection page. Skipped, never worked around.
rate_limited	Substack kept answering 429 after polite retries and an automatic slowdown. Try later.
unreachable	The domain does not resolve, or TLS/network failed.
invalid_input	Not a usable newsletter or post URL (for example a substack.com profile link).
no_posts	A Substack newsletter with no public posts, or none of the requested post URLs exist.

Run summary

The default key-value store record OUTPUT holds a summary: per newsletter the status, posts found / matched / delivered, paywalled count, which hidden sections were read, the sitemap post count, requests, retries and their reasons, rate-limit slowdowns, duration, warnings, and change counts. Use it to monitor scheduled runs.

Recipe: a "new posts since last run" feed

1. Input:

   {
       "publications": ["simonw.substack.com", "www.lennysnewsletter.com", "www.noahpinion.blog"],
       "trackChanges": true,
       "changesOnly": true,
       "postedAfter": "14 days"
   }
   

2. Run it once. Every post of the last 14 days arrives labelled new (this is the baseline).
3. Create a schedule (Schedules -> daily). From then on each run delivers only posts that are new or whose title, subtitle, audience or length changed, and you pay only for those. A day with nothing new delivers zero post rows and succeeds.
4. Connect the dataset to Slack, email, Make, Zapier or a webhook through Apify integrations, or poll .../runs/last/dataset/items?view=changes.

postedAfter keeps each daily run to a few requests per newsletter. Removed posts are only reported by runs that read the whole archive (no postedAfter, no cap); see below.

Recipe: build a RAG corpus from newsletters

1. Input:

   {
       "publications": ["simonw.substack.com", "www.astralcodexten.com"],
       "onlyFree": true,
       "contentFormat": "markdown",
       "includePublicationInfo": false
   }
   

2. Download with the Content view (?view=content) or take the full rows from the API.
3. Use one document per row: content as the text; title, url, postDate, authors and publicationName as metadata for citations. Split on Markdown headings (##), which the converter keeps.
4. Keep it fresh: add trackChanges + changesOnly and schedule it; re-embed rows with changeType new or updated, delete removed ones.

If you include paywalled posts, filter on contentAvailable == "full" so previews do not end up in the corpus as if they were whole posts.

Change tracking

Turn on Track changes between runs and schedule the Actor. Each run compares every post with the snapshot from the previous run of the same newsletter and adds:

Field	Meaning
changeType	new, updated, unchanged or removed.
changes	Everything that applies: new, title_changed, subtitle_changed, audience_changed (for example a post moved behind the paywall), slug_changed, wordcount_changed, and edited (the post's updated time moved, only seen when the body is fetched).
previousTitle, previousAudience, previousWordcount, previousUpdatedAt	Last run's values.

Rules, so you can rely on the labels:

• The first tracked run labels everything new. It creates the baseline.
• updated means the title, subtitle, audience, slug or word count changed. An edit that changes none of them shows edited in changes while changeType stays unchanged. Likes and comment counts are not treated as changes.
• removed rows (with the last known title and URL) are emitted only after a complete read: no cap, no postedAfter stop, no fetch errors, not stopped by the charge limit, and the same coverage as the previous run. If more than half of the known posts vanished at once, the Actor assumes a glitch and reports nothing; the next complete run decides.
• Nothing is lost on a budget stop. The snapshot is updated only for rows actually delivered, so a change that did not fit in this run is reported in the next one.
• Single-post inputs are tracked in their own snapshot.
• Snapshots live in the named key-value store snapshotStoreName in your account (compressed, one record per newsletter). Delete it, or pick a new name, to start over.

Pricing

Pay per event: a small fee per run start and per post row delivered to the dataset (post event). See the Pricing tab for current prices.

• One price per post row, with or without the body.
• removed rows from change tracking count as post rows; with changesOnly, unchanged posts are not delivered and not charged.
• Publication info rows and rows for newsletters that could not be read are free.
• Set Maximum cost per run in the run options to cap spending. The Actor stops cleanly at the cap, keeps everything delivered so far, does not download bodies it cannot deliver, and notes it in OUTPUT.

Use from an AI agent or via API

Ready-to-run Python, Node.js and cURL examples: github.com/cedarfetch/substack-scraper-examples.

Run synchronously and get the rows back in one HTTP call:

curl -X POST "https://api.apify.com/v2/acts/<username>~substack-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"publications":["simonw.substack.com"],"maxPostsPerPublication":10,"onlyFree":true}'

JavaScript:

import { ApifyClient } from 'apify-client';
const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('<username>/substack-scraper').call({ publications: ['www.noahpinion.blog'], postedAfter: '7 days' });
const { items } = await client.dataset(run.defaultDatasetId).listItems();

Python:

from apify_client import ApifyClient
client = ApifyClient("<APIFY_TOKEN>")
run = client.actor("<username>/substack-scraper").call(run_input={"publications": ["www.dwarkesh.com"], "postTypes": ["podcast"]})
items = client.dataset(run["defaultDatasetId"]).list_items().items

The Actor is also available to agents through the Apify MCP server, which exposes Store Actors as tools. Tips for agents: keep maxPostsPerPublication small for exploratory calls, use includeContent: false when you only need titles and dates, read recordType to tell post rows from publication and publication_status rows, and check contentAvailable before treating content as a whole post.

Limits and honest caveats

• Paywalled posts are preview only. The Actor never logs in and never tries to get around a paywall. For paid and subscriber-only posts you get what an anonymous visitor sees: usually the opening of the post, sometimes nothing beyond the title and description.
• No comments, no subscriber or reader data. Comment counts only. Subscriber counts, reader lists, emails and commenter details are not collected.
• No discovery or search. You tell it which newsletters to read. Substack's global search, category, leaderboard and profile pages live on substack.com and are not used (they refuse anonymous API clients).
• Notes, chats and live streams are not posts and are not included.
• Hidden posts. Posts a publisher hides from every listing are only found with coverage: "everything", which costs one request per such post. On some newsletters (often those with a podcast) that is hundreds of extra requests.
• Rate limits. Substack throttles clients that read fast. The defaults stay under it in testing; if you raise concurrency or lower requestDelayMs, the Actor slows itself down on the first 429, and a newsletter that stays throttled is reported as rate_limited with everything read so far delivered. Large archives with bodies take time: about 1.4 seconds per post at the defaults.
• sort: "top" is Substack's own popularity order. With postedAfter it cannot stop early, so it reads the whole archive listing (still about one request per 25 posts).
• Explicit post URLs bypass postedAfter, postedBefore, postTypes, onlyFree and the cap: you asked for that post.
• Markdown conversion handles what Substack's editor produces. Unusual embeds (polls, audio players, paid-only widgets) are dropped or reduced to a link.

FAQ

Does it work with custom domains? Yes. Paste the domain; the Actor detects Substack from the public API and response headers. A site that is not (or no longer) on Substack is reported as not_substack after one request and is not charged.

Can I get the full text of paid posts if I am a subscriber? No. The Actor does not accept logins or cookies and only reads public content.

How do I get only new posts every day? See the feed recipe above: trackChanges + changesOnly + a schedule.

Why are there more posts in postCount than rows? postCount counts the public sitemap. Posts hidden from every listing need coverage: "everything". The OUTPUT summary warns when the gap is large.

How fast is it? Metadata only: about 25 posts per request, a 900-post archive (plus its hidden sections) in about a minute. With bodies: one request per post at a polite pace.

Is this legal? The Actor collects publicly available newsletter content and public byline names, nothing personal beyond that. You are responsible for how you use the data, for respecting copyright and each newsletter's terms. Republishing other people's posts usually needs their permission. If in doubt, ask a lawyer.

Support

Found a newsletter that behaves oddly? Open an issue on the Issues tab with the newsletter address and the run ID; the OUTPUT record usually shows what happened.

More Actors from Cedarfetch Labs

Built the same way: public data only, typed output, per-source error reporting, change tracking for scheduled runs.

• Shopify Store Scraper: full product catalogs from any Shopify store, with price and stock change tracking.
• ATS Jobs Scraper: open jobs from Greenhouse, Lever, Ashby, SmartRecruiters, Workable, Recruitee and Personio boards, with new and removed postings flagged.
• Remote Jobs Scraper: remote jobs from Remote OK, We Work Remotely and five more boards in one deduplicated feed.
• Google Trending Searches: what is trending on Google right now, per country, with alerts on new trends.