Iβm excited to share that Climate Money is launching paid subscriptions tomorrow, October 22. ...
...", "comments": [ { "commentId": 367620777, "parentId": null, "depth": 0, "authorName": "(commenter's display name)", "authorHandle": "(commenter's handle)", "isPostAuthor": false, "body": "The free and paid boundary is clearest when readers can tell what remains dependable on each side. ...", "date": "2026-10-22T17:34:07.205Z", "editedAt": null, "reactionCount": 1, "restacks": 0 } ], "discoveredBy": "input", "scrapedAt": "2026-11-02T11:11:16.316Z" } ``` See [samples/sample-output.json](samples/sample-output.json) for 8 real items (2 publications, 4 free and 2 paywalled posts; commenter names are replaced with placeholders in that file). Notes on the fields: - `postId` is stable. Use it to de-duplicate. - `isPaywalled` is `true` when the full text is not public, i.e. `audience` is anything other than `everyone` (in testing: `only_paid`; Substack also has founding-member and subscribers-only posts). For those posts `bodyText`, `bodyHtml` and `comments` are `null`, because Substack only shows them to subscribers. `previewText`, `wordcount` and the engagement counts are still there. - `type` is Substack's post type: `newsletter`, `podcast`, `thread` and so on. Podcast posts also have `podcastUrl` and `podcastDurationSec`. - `bodyText` and `bodyHtml` are only present when **Include full text** is on, and `comments` only when **Include comments** is on. - Restacks of other publications' posts that appear in an archive are skipped (and counted in the run summary). They belong to the other publication. - `discoveredBy` says where a publication came from: `input`, `keyword: ...` or `category: ...`. With **Include publication info** on, each publication also gets one item with `itemType: "publication"` and `type: "publication"`: ```json { "itemType": "publication", "type": "publication", "publicationId": 10845, "name": "Lenny's Newsletter", "subdomain": "lenny", "customDomain": "www.lennysnewsletter.com", "url": "https://www.lennysnewsletter.com", "description": "Deeply researched product, growth, and career advice for product leaders, founders, and ambitious builders.", "language": "en", "authorName": "Lenny Rachitsky", "authorHandle": "lenny", "logoUrl": "https://substackcdn.com/image/fetch/...", "createdAt": "2019-06-01T15:35:37.885Z", "firstPostDate": "2019-06-14T15:00:48.833Z", "subscriberCountApprox": "1.2M+", "subscriberCountText": "Millions of subscribers", "paidSubscriberCountText": "Tens of thousands of paid subscribers", "paidSubscriptionsEnabled": true, "hasPodcast": true, "twitterHandle": "lennysan", "discoveredBy": "input", "sourceInput": "https://www.lennysnewsletter.com", "scrapedAt": "2026-11-02T11:11:07.509Z" } ``` Subscriber counts are the rounded figures Substack shows publicly (e.g. "1.2M+"). Exact counts are not public. A `RUN_SUMMARY` record in the run's key-value store lists, for every publication: posts output, posts listed, paywalled posts, posts skipped (too old, not new, restacks), failed requests, errors and duration, plus the results of each keyword and category search. ### How much does it cost to scrape Substack? This Actor uses **pay-per-event** pricing. You pay **$1.00 per 1,000 posts** ($0.001 per post) on the Free plan, and less on paid Apify plans. Full text and comments are included. Publication profile items, which are optional, cost $0.002 each. There is no start fee. | Posts | Price (Free plan) | |---|---| | 100 | $0.10 | | 1,000 | $1.00 | | 10,000 | $10.00 | Posts removed by your filters, posts already output in incremental mode, skipped restacks and failed publications cost nothing. To cap spending, set **Maximum cost per run** in the run options. The scraper stops cleanly when the budget is reached and says so in the status message. ### Scheduling and incremental monitoring of new posts Use this to get **alerts for new posts** from a list of newsletters, or to keep a database in sync: 1. Turn on **Only new posts** and give the monitor a name in **Monitor name** (e.g. `ai-newsletters`). 2. Save the configuration as a **Task** and add a **Schedule** (e.g. every morning). 3. The first run outputs the current posts (up to **Max posts per publication**) and records its start time as the baseline. If the first run is cut short (by **Maximum cost per run**, an error or an abort), the baseline is not set yet: the next run continues the first run and skips the posts already output. 4. Every later run outputs **only posts published after the baseline** that haven't been output yet. Posts are listed newest-first and the scraper stops at the first archive page older than the baseline, so a run that finds nothing new costs two requests per publication. Older posts never show up in later runs, even if the first run stopped at **Max posts per publication**. The memory is kept per publication in a named key-value store (`substack-scraper-state` by default), under the key you choose. Different monitor names are fully independent. Don't run two runs with the same monitor name at the same time. ### Discovering newsletters - **By keyword.** For each keyword the scraper runs Substack's post search and collects the publications behind the matching posts, in search order, up to **Max publications per keyword or category**. This finds publications that *write about* the topic, which is usually what you want for research, outreach or sponsorships. - **By category.** The scraper takes the publications Substack ranks highest in the category (the same list as the category pages on substack.com). - To get only a list of newsletters, turn **Include posts** off and **Include publication info** on. Substack's dedicated publication-name search returns no results without a login, so it isn't used. ### Integrations and API - **API.** Start runs and fetch results with the [Apify API](https://docs.apify.com/api/v2), or the JavaScript and Python clients: ```bash curl -X POST "https://api.apify.com/v2/acts/rjmacarthy~substack-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \ -H "Content-Type: application/json" \ -d '{"publications":["https://www.lennysnewsletter.com"],"maxPostsPerPublication":20}' ``` - **Webhooks.** Get notified (or trigger your pipeline) when a run finishes. - **Zapier, Make, n8n.** Send new posts to Google Sheets, Airtable, Slack, email or your CRM with no code. - **Google Sheets, Slack, Airbyte, Keboola and more** through Apify integrations. - **AI agents.** Use the Actor as a tool through the Apify MCP server, e.g. to summarise this week's posts from a list of newsletters. ### FAQ **Do I need a Substack account, login, cookies or a proxy?** No. The scraper uses the public endpoints that Substack's own pages use. A proxy option exists in case Substack blocks or rate-limits the requests. **Can it get the full text of paid posts?** No. Substack only serves a preview of paid posts to non-subscribers, and this Actor doesn't log in. Paid posts are output with their metadata and `isPaywalled: true`, and `bodyText` is `null`, so you never mistake a preview for the full article. **Why are comments `null` on paywalled posts?** Substack only shows comments on paid posts to subscribers; the API returns none to everyone else. The `commentCount` is still correct. **Does it collect subscriber lists or email addresses?** No. It only outputs what anyone can see on a publication's public pages: posts, public bylines, public comments and the publication profile. It never outputs subscriber lists or emails. **Are there any limits on the free plan?** No. Free-plan users get the same features. You pay only per item, and your Apify free monthly credit covers it. **How fast is it?** A metadata-only scrape needs one request per 23β50 posts, so whole archives take seconds. Full text needs one extra request per free post, and comments one per post that has comments. At the default pacing of at most 4 requests per second, that is roughly 200 full-text posts per minute. The scraper slows down automatically if Substack answers with HTTP 429. **What happens with an invalid input?** That publication is reported as failed with a clear reason, for example "publication not found", "the domain does not exist" or "does not look like a Substack publication", and the others continue. You're not charged for it. **Is scraping Substack legal?** The scraper only collects publicly available content that publishers make available to everyone. It doesn't collect personal data beyond public bylines and public comment display names. Always check that your use complies with the applicable terms and laws, including copyright: the full text of posts belongs to their authors. ### Limits - Paid and subscriber-only posts: metadata, preview text and counts only (see above). - Keyword discovery finds publications through matching *posts*, not by publication name. Each keyword reads up to 10 search pages. - Profile URLs (`substack.com/@name`), Notes and Chat are not supported. - Restacks of other publications' posts are skipped. - Substack's "top" ranking is Substack's own and can change between runs. - Incremental state lives in a named key-value store in your account, created by the Actor itself. Deleting that store resets the monitor. If you already have an unrelated store with the same name, set a different **State key-value store name**. ### Ready-to-run examples - [Find AI newsletters on Substack and scrape their posts](https://apify.com/rjmacarthy/substack-scraper/examples/ai-newsletters-on-substack) - [Export every post from a Substack newsletter](https://apify.com/rjmacarthy/substack-scraper/examples/export-substack-newsletter-posts) ### More scrapers by the same developer - [Telegram Channel Scraper](https://apify.com/rjmacarthy/telegram-channel-scraper) - [Google Ads Transparency Scraper](https://apify.com/rjmacarthy/google-ads-transparency-scraper) - [ATS Jobs Scraper: Greenhouse, Lever, Ashby & Workday](https://apify.com/rjmacarthy/ats-jobs-scraper) - [Dice Jobs Scraper](https://apify.com/rjmacarthy/dice-jobs-scraper) - [SEEK Jobs Scraper](https://apify.com/rjmacarthy/seek-jobs-scraper) - [Workday Jobs Scraper](https://apify.com/rjmacarthy/workday-jobs-scraper) Browse all my Actors on [my Apify Store profile](https://apify.com/rjmacarthy). ### Support Found a publication that doesn't work, or need an extra field? Open an issue on the **Issues** tab with the URL. Fixes are usually quick. # Actor input Schema ## `publications` (type: `array`): Substack publications to scrape. Any of these work: a subdomain ("lenny"), a substack.com address ("https://lenny.substack.com"), a custom domain ("www.lennysnewsletter.com"), or a single post URL ("https://example.substack.com/p/post-slug"). Profile pages (substack.com/@name) are not supported. ## `searchKeywords` (type: `array`): Optional. For each keyword, the scraper searches Substack's posts and takes the publications that wrote them (e.g. "climate tech", "product management"). Those publications are then scraped like the ones above. ## `categories` (type: `array`): Optional. Top publications of a Substack category, in Substack's own ranking. Use the category name or slug, e.g. "technology", "business", "finance", "culture", "us-politics", "science", "climate", "health", "food", "sports". Subcategories also work. An unknown value is reported in the log together with the valid list. ## `maxPublicationsPerSearch` (type: `integer`): How many publications to take from each keyword or category. ## `maxPostsPerPublication` (type: `integer`): Stop after this many posts from each publication (in the chosen sort order). 0 = no limit (the whole archive). ## `sort` (type: `string`): "new" = newest first. "top" = the publication's most popular posts first (Substack's own ranking). ## `postedAfter` (type: `string`): Only output posts published after this date (e.g. 2026-10-01) or within this period (e.g. "7 days", "3 months"). Leave empty for no date filter. ## `includeFullText` (type: `boolean`): Add the full post body as plain text (bodyText) and HTML (bodyHtml). Only free posts have public full text: paywalled posts are flagged with isPaywalled: true and get bodyText: null (every post still has previewText). Costs one extra request per free post; turn off for a faster metadata-only scrape. ## `includeComments` (type: `boolean`): Add each post's public comments (author display name, text, date, likes, replies as a flat list with parentId). One extra request per post that has comments. ## `maxCommentsPerPost` (type: `integer`): Limit the comments added to each post. 0 = all. ## `includePosts` (type: `boolean`): Output one item per post. Turn off (and turn on "Include publication info") to get only a list of publications, e.g. for newsletter discovery. ## `includePublicationInfo` (type: `boolean`): Also output one item per publication (type: "publication"): name, URL, description, author, language, approximate subscriber count, paid subscriptions on/off, podcast on/off, creation date. ## `onlyNewPosts` (type: `boolean`): Remember what was already output and only output (and charge for) posts published since the previous run. The first run outputs the matching posts and sets the baseline; later runs output only posts published after the first run. Posts are always listed newest-first in this mode. ## `stateKey` (type: `string`): Name of this monitor's memory. Runs with the same name share the list of already-output posts (per publication). Use different names for independent monitors. ## `stateStoreName` (type: `string`): Named key-value store in your account where incremental state is kept (letters, digits and dashes). ## `maxConcurrency` (type: `integer`): Maximum parallel HTTP requests. Substack serves every publication from the same backend, so keep this low. ## `requestIntervalMs` (type: `integer`): Minimum time between the start of two requests, across all publications. The default (250 ms, at most 4 requests per second) is polite and fast enough for thousands of posts per run. ## `proxyConfiguration` (type: `object`): Not needed for Substack's public API in normal use. Enable it only if Substack blocks or rate-limits the requests (HTTP 403 or 429 in the log). ## Actor input object example ```json { "publications": [ "https://www.lennysnewsletter.com", "climatemoney" ], "maxPublicationsPerSearch": 10, "maxPostsPerPublication": 10, "sort": "new", "includeFullText": true, "includeComments": false, "maxCommentsPerPost": 0, "includePosts": true, "includePublicationInfo": false, "onlyNewPosts": false, "stateKey": "default", "stateStoreName": "substack-scraper-state", "maxConcurrency": 3, "requestIntervalMs": 250, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `posts` (type: `string`): One item per post: title, publication, date, authors, type, paywall flag, word count, likes, comments, restacks and URL. ## `postsFull` (type: `string`): Every field of every item, including full text (bodyText, bodyHtml), comments and publication-info items. ## `publications` (type: `string`): Publication-info items (only when "Include publication info" is on). ## `summary` (type: `string`): Per-publication status, counts, discovery results and any errors. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '