# Substack Newsletter Scraper (`bigibyte/substack-newsletter-scraper`) Actor Substack scraper that exports newsletter posts, full articles, comments, and author leads, so you can collect subscriber ranges and social links without a login or an API key. - **URL**: https://apify.com/bigibyte/substack-newsletter-scraper.md - **Developed by:** [BigiByte](https://apify.com/bigibyte) (community) - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing Pay per usage This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Substack Scraper: Export Newsletter Posts, Comments, and Author Leads This Substack scraper reads public newsletters and saves posts, comments, and publication details to an Apify dataset. You can download JSON, CSV, or Excel, or hook the dataset into Google Sheets, Make, Zapier, or n8n. There is no login and no Substack API key, because Substack does not offer a public API for this. It follows redirects, so a `name.substack.com` address and a custom domain like `https://www.lennysnewsletter.com` land on the same publication. Paywalled posts are flagged. The body you get for those is whatever Substack returns without a session, which may be a preview rather than the full paid text. ### Use cases - Sponsorship outreach: open a category leaderboard, then sort publications by the subscriber range Substack prints and by the author's Twitter handle. - Newsletter directory: search a keyword, or walk categories such as business, technology, finance, and crypto, and keep the publication URL plus author handle. - Reading and research: export free articles as Markdown, plain text, or HTML when you want a pile of posts instead of clicking through the archive. - Comment review: flatten replies on a post, with parent id and depth, when you care what readers actually said. - Competitor watch: rerun the same publication URL on a schedule and keep new posts from the archive. ### How to run it 1. Choose a mode: posts, post details, comments, publication and author info, or discover. 2. Paste publication URLs, post URLs, or `@handles`. In discover mode, add category slugs and/or search phrases instead. 3. Set a max result count. The free cap is 1,000 rows per run. 4. Start the run and export the dataset as JSON, CSV, or Excel. A bad URL does not stop the run. The actor writes an error row and moves on. ### Input | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `mode` | string | `posts` | `posts`, `post_details`, `comments`, `publication_info`, or `discover`. | | `startUrls` | string\[] | `https://www.lennysnewsletter.com` | Publication URLs, custom domains, `https://substack.com/@handle`, `@handle`, or `/p/{slug}` post URLs. | | `sort` | string | `new` | Posts mode. `new` or `top`. | | `searchInArchive` | string | empty | Posts mode. Keyword passed to Substack's archive search. | | `includeFullContent` | boolean | `false` | Fetch each post body. One extra request per post. | | `contentFormat` | string | `markdown` | `markdown`, `text`, or `html`. | | `dateFrom` | string | empty | `YYYY-MM-DD`. With `sort=new`, paging stops once posts are older. With `sort=top`, older posts are skipped and paging continues. | | `includeAuthorProfile` | boolean | `true` | In publication and discover modes, add followers, subscriber count, bio, and social links from the public profile. | | `categories` | string\[] | `business` | Discover mode. Category slug or numeric id, such as `technology` or `4`. | | `leaderboardType` | string | `all` | `all` or `paid`. | | `searchQueries` | string\[] | empty | Discover mode. Global Substack search. | | `maxItems` | integer | `100` | Cap for the whole run. Maximum 1000. | | `maxItemsPerSource` | integer | `50` | Cap per publication, category, or search phrase. | | `maxConcurrency` | integer | `5` | Simultaneous requests. Maximum 5. | | `timeoutSecs` | integer | `300` | Stop the run after this many seconds and keep rows already saved. | | `requestTimeoutSecs` | integer | `30` | Per-request timeout. | | `proxyConfiguration` | object | No proxy | Optional Apify proxy. Leave it off unless you hit 429 or 403. | #### Posts ```json { "mode": "posts", "startUrls": ["https://www.lennysnewsletter.com", "https://garymarcus.substack.com"], "sort": "new", "includeFullContent": false, "maxItems": 100 } ``` #### Single post ```json { "mode": "post_details", "startUrls": ["https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit"], "contentFormat": "markdown" } ``` #### Comments ```json { "mode": "comments", "startUrls": ["https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit"], "maxItems": 200 } ``` #### Publication and author leads ```json { "mode": "publication_info", "startUrls": ["https://www.lennysnewsletter.com", "@lenny"], "includeAuthorProfile": true } ``` #### Discover ```json { "mode": "discover", "categories": ["business", "technology"], "leaderboardType": "all", "searchQueries": ["product management"], "includeAuthorProfile": true, "maxItems": 100, "maxItemsPerSource": 25 } ``` ### What data does this actor extract? Every row has `mode`, `scrapedAt`, and `sourceInput`. Failures use `error` and `statusCode` instead of a result. Codes: `NOT_SUBSTACK`, `NOT_FOUND`, `HTTP_`, `PARSE_ERROR`, `INVALID_INPUT`. The dataset table uses `postTitle`, `postType`, `summary`, and `heroText` because Apify's dataset schema rejects the field names `title`, `type`, and `description`. The JSON row still includes `title`, `type`, and `description` with the same values. #### Posts and post details ```json { "mode": "posts", "postId": 215694124, "publicationId": 10845, "publicationHost": "www.lennysnewsletter.com", "title": "All of the Lenny & Friends Summit talks are now online!", "postTitle": "All of the Lenny & Friends Summit talks are now online!", "slug": "all-of-the-lenny-and-friends-summit", "url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit", "publishedAt": "2026-09-29T13:15:57.512Z", "type": "newsletter", "postType": "newsletter", "audience": "everyone", "isPaywalled": false, "bodyMayBeTruncated": false, "wordCount": 1276, "reactionCount": 287, "commentCount": 7, "authors": [{"id": 1849774, "name": "Lenny Rachitsky", "handle": "lenny"}], "scrapedAt": "2026-10-12T00:00:00Z", "sourceInput": "https://www.lennysnewsletter.com" } ``` | Field | Type | Description | |-------|------|-------------| | `postId`, `publicationId`, `publicationHost` | integer, integer, string | Ids, plus the host after redirects. | | `title`, `postTitle`, `subtitle`, `description`, `summary` | string | Title and deck. `postTitle` and `summary` mirror `title` and `description`. | | `slug`, `url`, `publishedAt` | string | Slug, canonical URL, publish time. | | `type`, `postType`, `audience` | string | Post type (`newsletter`, and others such as `podcast` when Substack sends them) and audience (`everyone` or `only_paid`). | | `isPaywalled`, `bodyMayBeTruncated` | boolean | Paywalled when audience is not `everyone`. The stored body may be incomplete. | | `wordCount`, `reactionCount`, `commentCount`, `restackCount` | integer | Counts from the archive or post payload. | | `coverImage`, `section`, `tags`, `authors` | string, string, string\[], object\[] | Cover, section name, tag names, and bylines (`id`, `name`, `handle`). | | `podcastUrl`, `podcastDurationSec` | string, number | Present when the post has podcast fields. | | `preview` | string | Short plain-text preview. | | `content` | string | Body in the requested format. Omitted unless you asked for full content, or the mode is `post_details`. | #### Comments ```json { "mode": "comments", "commentId": 349085941, "postId": 215694124, "parentId": null, "depth": 0, "authorName": "Product Mindset", "authorHandle": "productmindset", "authorUserId": 8822099, "body": "Thank you Lenny for putting this together...", "date": "2026-09-29T23:34:47.245Z", "reactionCount": 3, "replyCount": 0 } ``` | Field | Type | Description | |-------|------|-------------| | `commentId`, `postId`, `postUrl` | integer, integer, string | Comment, post, and the post URL. | | `parentId`, `depth` | integer | Parent comment. Depth `0` is a top-level comment. Nested replies are flattened. | | `authorName`, `authorHandle`, `authorUserId` | string, string, integer | Who wrote it. | | `body`, `date`, `editedAt` | string | Text and timestamps. | | `reactionCount`, `restackCount`, `replyCount` | integer | Reactions, restacks, and Substack's reply count. | | `isDeleted`, `isPinned` | boolean | Deleted flag, and pinned when Substack set `pinned_by_user_id`. | A post with zero comments produces no comment rows. That is not an error. #### Publication info and leaderboard ```json { "mode": "publication_info", "publicationId": 10845, "name": "Lenny's Newsletter", "url": "https://www.lennysnewsletter.com", "subdomain": "lenny", "customDomain": "www.lennysnewsletter.com", "language": "en", "createdAt": "2019-06-01T15:35:37.885Z", "isPaid": true, "paidSubscribersRange": "Tens of thousands of paid subscribers", "freeSubscriberCount": 1200000, "freeSubscriberCountRaw": "1,200,000", "authorName": "Lenny Rachitsky", "authorHandle": "lenny", "authorFollowers": 1383847, "authorSubscriberCount": 1200000, "twitter": "lennysan", "socialLinks": [ {"type": "twitter", "url": "https://x.com/lennysan"}, {"type": "youtube", "url": "https://www.youtube.com/c/LennysPodcast"} ] } ``` | Field | Type | Description | |-------|------|-------------| | `publicationId`, `name`, `url`, `subdomain`, `customDomain` | mixed | Identity and the public URL (custom domain when there is one). | | `description`, `heroText`, `summary` | string | Hero text. The same string is stored under all three names. | | `language`, `createdAt`, `firstPostDate` | string | Language and dates. First-post date is filled from the leaderboard only. | | `isPaid` | boolean | True when `payments_state` is `enabled`. | | `paidSubscribersRange`, `totalSubscribersRange` | string | Labels such as "Tens of thousands of paid subscribers". The total range comes from the leaderboard only. | | `freeSubscriberCount`, `freeSubscriberCountRaw` | integer, string | Parsed count, plus the original string (`1,200,000`). Null when Substack does not send one. | | `logoUrl`, `hasPodcast`, `podcastFeedUrl`, `supportEmail` | mixed | Podcast and support email are leaderboard fields. They stay empty on a plain publication lookup. | | `category`, `leaderboardRank` | string, integer | Set in discover mode. Rank is the position while paging that leaderboard. | | `authorName`, `authorHandle`, `authorBio` | string | Author fields. | | `authorFollowers`, `authorSubscriberCount` | integer | From the public profile when enrichment is on. | | `twitter`, `socialLinks` | string, object\[] | Screen name and links (`type`, `label`, `url`). | Search rows add `resultType`: `profile`, `post`, or `comment`. #### Errors | Field | Type | Description | |-------|------|-------------| | `sourceInput` | string | The URL, handle, category, or query that failed. | | `error` | string | `NOT_SUBSTACK`, `NOT_FOUND`, `HTTP_`, `PARSE_ERROR`, or `INVALID_INPUT`. | | `statusCode` | integer | HTTP status when there was one. | `NOT_SUBSTACK` is for hosts that are not Substack publications. `platformer.news`, for example, is on Ghost now, and its archive route returns 404. ### Limits and notes - 1,000 results per run. `maxItems` stops the run mid-page once that many rows are stored. Duplicate post, comment, and publication ids are skipped and do not count. - Archive requests never ask for more than 50 posts. Substack returns 400 above that. - Paid posts (`audience` `only_paid`) set `isPaywalled` and `bodyMayBeTruncated`. Do not treat `content` as the full paid article. - Subscriber numbers are whatever Substack shows publicly: a range label, and sometimes a free-subscriber count. This actor does not estimate a number Substack did not send. - This is not an official Substack product and it is not affiliated with Substack. Use it within Substack's terms and privacy rules. It does not log in, and it does not try to unlock paid posts. ### FAQ **Is there an official Substack API?** No public API covers archive posts, comments, and leaderboards. This actor calls the same public JSON routes the site uses in the browser. **Can this Substack scraper read custom-domain newsletters?** Yes. `https://www.lennysnewsletter.com` and `https://lenny.substack.com` both work. Publications that moved to a custom domain redirect, and the row records the final host. **Can it export paid Substack posts?** It requests the post without a login. If Substack marks the audience as `only_paid`, the row is flagged and the body may be only the public preview. **How do I get a Substack subscriber count?** Publication rows include `paidSubscribersRange` (text such as "Tens of thousands of paid subscribers") and `freeSubscriberCount` when Substack sends a figure like `1,200,000`. Leaderboard rows can also include `totalSubscribersRange`. There is no exact paid-subscriber integer in these public responses. **How many posts, comments, or newsletters can one run collect?** `maxItems` caps the run at 1,000. `maxItemsPerSource` caps each publication, category, or search. Comments are a separate row per comment, including nested replies. ### Related tools If you are cleaning the export afterward, a contact extractor and an HTML-to-Markdown pass are the usual next steps. BigiByte also builds scrapers for other publishing platforms, including Medium and Ghost, when you need them as a separate job. Need bulk Substack data, a scheduled monitor, or a custom pipeline? BigiByte builds custom scrapers and APIs: https://bigibyte.com ### Trademark Notice Substack and related names, logos, and trademarks belong to their respective owners. Their use in this documentation is solely to identify the source of the data and does not imply affiliation, sponsorship, endorsement, or partnership. ### Disclaimer This Actor is an independent tool and is not affiliated with, endorsed by, sponsored by, or otherwise officially associated with Substack. ### Is web scraping legal? Collecting information that is already public is generally legal. Scraping is not the same thing as hacking, and there is no single law that bans it. The limits are the usual ones: do not break into accounts, do not take private or confidential data, and be careful with personal data and copyrighted works. A site's terms can also restrict how you access it. Facts such as a public name, price, or rating are not the same thing as republishing someone's article or photo. This actor reads public pages or official public APIs. It does not log in and it does not bypass access controls. This is not legal advice. For the longer explanation, see Apify's article [Is web scraping legal?](https://blog.apify.com/is-web-scraping-legal/). # Actor input Schema ## `mode` (type: `string`): Choose the data you want. ## `startUrls` (type: `array`): Publication URLs (https://name.substack.com, custom domains like https://www.lennysnewsletter.com), profile handles (https://substack.com/@lenny or @lenny), or post URLs (/p/slug). ## `sort` (type: `string`): Posts mode only. ## `searchInArchive` (type: `string`): Posts mode only. Uses Substack's archive search. ## `includeFullContent` (type: `boolean`): Fetch each post's body (slower: 1 extra request per post). ## `contentFormat` (type: `string`): How to store the article body when full content is requested. Markdown, plain text, or the original HTML. ## `dateFrom` (type: `string`): YYYY-MM-DD. Posts mode with sort=new stops paging once older. ## `includeAuthorProfile` (type: `boolean`): publication_info and discover modes. ## `categories` (type: `array`): Discover mode. Category slugs or ids, e.g. business, technology, finance, crypto. ## `leaderboardType` (type: `string`): Discover mode. Use the full category leaderboard or the paid-subscriptions leaderboard. ## `searchQueries` (type: `array`): Discover mode. Substack global search (newsletters, authors, posts). ## `maxItems` (type: `integer`): Total cap for the whole run (free version max 1000). ## `maxItemsPerSource` (type: `integer`): Cap for each publication, category, or search phrase. The run also stops at max results. ## `maxConcurrency` (type: `integer`): How many requests can run at once. ## `timeoutSecs` (type: `integer`): Stop the run after this many seconds and keep the rows already saved. ## `requestTimeoutSecs` (type: `integer`): Per-request timeout. ## `proxyConfiguration` (type: `object`): Not needed normally. Enable only if you see 429/403. ## Actor input object example ```json { "mode": "posts", "startUrls": [ "https://www.lennysnewsletter.com" ], "sort": "new", "searchInArchive": "pricing", "includeFullContent": false, "contentFormat": "markdown", "includeAuthorProfile": true, "categories": [ "business", "technology" ], "leaderboardType": "all", "searchQueries": [ "product management" ], "maxItems": 100, "maxItemsPerSource": 50, "maxConcurrency": 5, "timeoutSecs": 300, "requestTimeoutSecs": 30, "proxyConfiguration": { "useApifyProxy": false } } ``` # Actor output Schema ## `overview` (type: `string`): Newsletter posts with title, URL, date, audience, reactions, comments, and word count. ## `comments` (type: `string`): Flattened post comments with author, body, date, reactions, and post URL. ## `leads` (type: `string`): Publications and authors, including subscriber ranges, Twitter, support email, and language. ## `errors` (type: `string`): Inputs that failed, with the error code and HTTP status. ## `dataset` (type: `string`): Every row from the run, without a table view applied. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://www.lennysnewsletter.com" ], "categories": [ "business" ], "proxyConfiguration": { "useApifyProxy": false } }; // Run the Actor and wait for it to finish const run = await client.actor("bigibyte/substack-newsletter-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://www.lennysnewsletter.com"], "categories": ["business"], "proxyConfiguration": { "useApifyProxy": False }, } # Run the Actor and wait for it to finish run = client.actor("bigibyte/substack-newsletter-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://www.lennysnewsletter.com" ], "categories": [ "business" ], "proxyConfiguration": { "useApifyProxy": false } }' | apify call bigibyte/substack-newsletter-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,bigibyte/substack-newsletter-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/eNgsvVV0RyyFAGDdd/builds/TUGojipg826cPkash/openapi.json