Substack Posts Scraper

Substack Posts Scraper is a Substack scraper that exports every post of any Substack newsletter to JSON, CSV or Excel: title, subtitle, authors, publish date, post type, paywall status, likes (reactions), comments, restacks, podcast audio and, optionally, the full post body (HTML and plain text). Give it a publication URL, a subdomain such as lenny or a custom domain such as newsletter.pragmaticengineer.com, and get one row per post - a whole newsletter archive in a spreadsheet, with no code and no login.

The Actor uses Substack's public JSON endpoints (the same ones the Substack website uses) instead of parsing HTML pages, so it is fast, light on Substack's servers and does not break when the page layout changes. No login is required and no private data is accessed.

What can you do with it

• Export a Substack archive to CSV or Excel: every post of a newsletter (maxPostsPerPublication: 0) in one file, for backups, migrations or a reading list.
• Monitor competitor newsletters: track what a set of Substacks publish, how often, and which posts get the most likes, comments or restacks.
• Find what resonates in a niche: sort a publication's posts by reactions to see which topics and formats perform best before you write your own.
• Feed AI and search tools: pull the full text of free posts for summaries, topic modelling, RAG pipelines or an internal search index.
• Study paywall strategy: see which posts a publication paywalls, how long they are and who writes them.
• Collect podcast episodes: audio URLs and durations from Substack podcasts.

How to use

1. Open Substack Posts Scraper and click Try for free (or Start).
2. Paste one or more publications (lenny, https://www.lennysnewsletter.com, a custom domain), set how many posts you want (0 = whole archive) and tick Include post body if you need the text.
3. Click Start, then download the results as CSV, Excel or JSON from the Output tab, or fetch them through the API.

Sample output (selected columns)

title	publication.name	publishedAt	paywalled	reactionCount	commentCount	wordCount
How to tell if a startup has a moat	Lenny's Newsletter	2026-10-13	true	375	2	4153

The full record (25+ fields) is shown under Output below.

Input

Field	Type	Default	Description
publications	array of strings	required	Substack publications to scrape: full URLs (https://www.lennysnewsletter.com), subdomains (lenny, lenny.substack.com) or custom domains (newsletter.pragmaticengineer.com). Publications that moved to a custom domain are followed automatically.
maxPostsPerPublication	integer	100	Posts to save per publication. 0 = the whole archive.
sort	new or top	new	new = newest first (chronological). top = Substack's own popularity ranking (not chronological, so it cannot be combined with publishedAfter).
includePostBody	boolean	false	Fetch each post individually and add bodyHtml and bodyText. Free posts get the full body; paywalled posts get the public preview only. Roughly doubles the number of requests.
publishedAfter	string	-	Only keep posts published on or after this date and stop paging there. Absolute (2026-01-31, 2026-01-31T10:00:00Z, UTC) or relative (7 days, 2 weeks, 3 months). Only with sort: new.
postTypes	array	all	Keep only these types: newsletter, podcast, video, thread. Empty = all.
proxyConfiguration	object	Apify datacenter proxies	Proxy settings. Datacenter proxies are enough for normal volumes; switch to residential if you see repeated 403/429 errors in the log.

Example input:

{
    "publications": ["lenny", "newsletter.pragmaticengineer.com"],
    "maxPostsPerPublication": 50,
    "sort": "new",
    "includePostBody": true,
    "publishedAfter": "30 days",
    "postTypes": ["newsletter", "podcast"]
}

Output

One dataset item per post. The fields are stable and documented in the dataset schema; bodyHtml and bodyText appear only when includePostBody is enabled.

{
    "publication": {
        "subdomain": "lenny",
        "name": "Lenny's Newsletter",
        "url": "https://www.lennysnewsletter.com",
        "customDomain": "www.lennysnewsletter.com"
    },
    "postId": 219482066,
    "slug": "how-to-tell-if-a-startup-has-a-moat",
    "url": "https://www.lennysnewsletter.com/p/how-to-tell-if-a-startup-has-a-moat",
    "canonicalUrl": "https://www.lennysnewsletter.com/p/how-to-tell-if-a-startup-has-a-moat",
    "title": "How to tell if a startup has a moat",
    "subtitle": "A framework for assessing whether the company you're considering joining is built to last",
    "description": "A framework for assessing whether the company you're considering joining is built to last",
    "type": "newsletter",
    "audience": "only_paid",
    "paywalled": true,
    "publishedAt": "2026-10-13T12:45:41.160Z",
    "authors": [{ "name": "Dan Hockenmaier", "handle": "danhock" }],
    "coverImageUrl": "https://substackcdn.com/image/fetch/...",
    "wordCount": 4153,
    "reactionCount": 375,
    "commentCount": 2,
    "restackCount": 22,
    "podcastUrl": null,
    "podcastDurationSec": null,
    "bodyHtml": "<p><em>Hey there, I'm Lenny. ...</em></p>...",
    "bodyText": "Hey there, I'm Lenny. ...",
    "scrapedAt": "2026-10-29T12:00:00.000Z"
}

Field	Meaning
publication.subdomain	Substack subdomain (lenny). null when the publication uses a custom domain and Substack did not expose the subdomain.
publication.name, publication.url, publication.customDomain	Name and home page (current domain) of the publication.
postId, slug, url, canonicalUrl	Substack's numeric ID, the URL slug, the post URL on the current domain and the canonical URL reported by Substack (differs for cross-posts).
title, subtitle, description	Post texts; subtitle/description are null when empty.
type	newsletter, podcast, video or thread.
audience	Substack's audience value: everyone, only_paid, only_free (free subscribers) or founding.
paywalled	true when the full post needs a paid subscription (only_paid or founding).
publishedAt	ISO 8601 (UTC).
authors	Public bylines, name and handle only.
coverImageUrl	Cover image, or null.
wordCount	Length of the full post as reported by Substack (also for paywalled posts).
reactionCount, commentCount, restackCount	Engagement counters at scrape time.
podcastUrl, podcastDurationSec	Audio file URL and duration in seconds for podcast posts, otherwise null.
bodyHtml, bodyText	Post body (only with includePostBody). For paywalled posts this is the public preview. null if the post could not be fetched.
scrapedAt	When the item was collected.

The Overview tab in the run shows title, publication, date, paywall flag, reactions and URL; the full items are available as JSON, CSV, Excel or via the API like any Apify dataset.

Pricing

This Actor uses Apify's pay-per-event model, so you only pay for what you get:

• $0.0015 per saved post (one dataset item), and
• $0.005 per run start.

1,000 posts therefore cost about $1.50 + $0.005, whichever options you choose; enabling includePostBody does not change the price per post. The exact prices are shown on the Pricing tab of this page. Platform usage (compute, proxies) is covered by these prices and is not billed separately. Set a maximum cost per run in the run options if you want a hard cap: the Actor stops requesting more posts as soon as the cap is reached and everything saved until then stays in the dataset.

How it works

For every publication the Actor:

1. reads the publication's RSS feed once (/feed) to learn its name and current domain (publications that moved from name.substack.com to a custom domain are followed);
2. pages through the public archive endpoint (/api/v1/archive) 50 posts at a time, newest first, until the limit, the publishedAfter cut-off or the end of the archive;
3. with includePostBody, fetches each selected post (/api/v1/posts/<slug>) for its body.

Politeness: at most 3 requests in flight, 300-800 ms between requests to the same publication, exponential backoff on 429 / 5xx responses (up to 5 retries), no login and no cookies. A publication that fails (typo, removed, blocked) is reported in the log and the run continues with the next one; the run only fails when every publication failed. If two entries turn out to be the same publication (for example lenny and https://www.lennysnewsletter.com), it is scraped once and the second entry is reported as skipped.

The log ends with a summary: publications processed, posts saved, paywalled count and the error of every publication that could not be scraped.

Limitations

• Paywalled posts only contain the public preview in bodyHtml/bodyText (plus all metadata, including the full wordCount). The Actor never logs in and cannot bypass paywalls.
• Posts restricted to free subscribers (audience: only_free) are not marked paywalled, but their body may also be a preview.
• sort: top is Substack's own ranking, which is not strictly by reaction count and cannot be combined with publishedAfter.
• Comments, subscriber counts, Notes and recommendations are not collected (they are not posts).
• Very large archives (thousands of posts) with includePostBody take longer because every post is one request; expect roughly 2-4 posts per second.
• Substack can rate-limit an IP that sends too many requests. The Actor backs off automatically; if a run still ends with 429/403 errors, run fewer publications at once or choose residential proxies in proxyConfiguration.

Legal, privacy and robots.txt

The Actor collects only publicly available data from publication pages that Substack's robots.txt allows for all crawlers (/api/, /feed and /p/ are not disallowed; the Actor does not touch /feed/private, comment pages or any sign-in flows). It does not use accounts, cookies or paid content.

Personal data is limited to what Substack publishes on every post: the author's display name and handle. Emails, photos, bios and subscriber status that Substack includes in its responses are discarded. If you process this data in the EU you remain responsible for having a lawful basis (for example legitimate interest for research) and for honouring data-subject requests; see Apify's notes on GDPR and web scraping. Respect the publication's terms when republishing content.

Integrations

• Make, Zapier, n8n and other automation tools: add the Apify integration (the official "Apify" app/node), choose Run an Actor with this Actor and your input, and use Get dataset items (or the "Actor run finished" trigger) to pass the records on - for example to a Google Sheet of new posts or a Slack message for every new issue of a newsletter. Each tool also accepts Apify webhooks: Console -> Actor -> Integrations -> Webhooks, event Run succeeded.
• AI agents (MCP): Apify's MCP server (https://mcp.apify.com, or the @apify/actors-mcp-server package) exposes this Actor as a tool to Claude, ChatGPT, Cursor and other MCP clients: the agent calls substack-posts-scraper with an input such as {"publications": ["lenny"], "maxPostsPerPublication": 20} and reads the dataset items back.
• API and SDKs: POST https://api.apify.com/v2/acts/davidts~substack-posts-scraper/run-sync-get-dataset-items with the input as JSON body returns the records directly; the Apify client libraries for JavaScript and Python wrap the same call.
• Schedules: Console -> Schedules runs the Actor daily or weekly with a saved input and, combined with a webhook or integration, delivers only what is new.

Troubleshooting

Log message	What it means
Publication not found (HTTP 404 ...)	The subdomain/URL does not exist. Check the spelling; use the URL shown in the browser.
did not return an RSS feed ... Either this is not a Substack publication or it is private.	The domain is not a Substack site, the publication is private, or Substack served a block page.
Substack rate-limited the requests (HTTP 429) / blocked the requests (HTTP 403)	Too many requests from one IP. Try again later, scrape fewer publications per run, or switch proxyConfiguration to residential proxies.
publishedAfter can only be used with sort "new"	Remove publishedAfter or switch sort to new.
The archive is empty	The publication has no public posts yet.

FAQ

How much does it cost to scrape 1,000 Substack posts? About $1.50 ($0.0015 per saved post) plus $0.005 per run, with or without the full post body. Platform usage is included. You can cap the cost of any run in the run options.

Is it legal to scrape Substack? What about robots.txt and GDPR? The Actor reads only public data from paths that Substack's robots.txt allows, never logs in and cannot see paywalled text. The only personal data kept is the author's public display name and handle. Scraping public data is generally lawful, but you are responsible for how you use it: in the EU you need a lawful basis (for example legitimate interest for research), and republishing someone's posts needs their permission. See Legal, privacy and robots.txt above. This is not legal advice.

Can I export Substack posts to CSV, Excel or Google Sheets? Yes. Every run's dataset downloads as CSV, Excel, JSON, XML or HTML from the Output tab. For Google Sheets, import the CSV, use the Google Sheets Import & Export Actor from the Apify Store or a Make/Zapier scenario, or point =IMPORTDATA() at the API URL of the last run's dataset (keep in mind the URL contains your API token).

Can I get new posts automatically every day or week? Yes. Create a schedule in Apify Console (Schedules) with sort: "new" and publishedAfter: "7 days" (or 1 day) so each run only saves recent posts. Add a webhook or a Make/Zapier/n8n integration to send them to Slack, e-mail or a sheet.

Can I call it from my code or an AI agent? Yes. POST https://api.apify.com/v2/acts/davidts~substack-posts-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN with the input as JSON returns the posts directly; the Apify JavaScript and Python clients and Apify's MCP server (for Claude, ChatGPT, Cursor and other agents) work too. See Integrations.

Does it scrape paywalled posts, comments or subscriber counts? No. Paywalled posts come with full metadata and the public preview only. Comments, Notes, subscriber counts and recommendations are not collected.

If this Actor saves you time, a short review on its Store page helps other people find it - thank you! Found a bug or need a field? Open an issue on the Issues tab and I will look at it.

Changelog

The full history is in the Actor's CHANGELOG.md (also shown in the Information tab in Apify Console).

• 0.1 (2026-10) - first release: archive paging, custom-domain handling, paywall detection, optional post bodies, podcast fields, publishedAfter cut-off, type filter, pay-per-event pricing.