# Substack Scraper: Posts, Comments & Newsletter Profiles (`s-r/substack-scraper`) Actor Collect Substack newsletter posts, writer and publication profiles, subscriber estimates and comment threads. Search by keyword, filter by date, and optionally export public article text. Paid content is limited to its public preview. - **URL**: https://apify.com/s-r/substack-scraper.md - **Developed by:** [SR](https://apify.com/s-r) (community) - **Categories:** Social media, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $1.00 / 1,000 posts This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Substack Scraper Substack Scraper collects newsletter posts, public author profiles, publication information, and comments for newsletter research, creator discovery, and content monitoring. It works as a no-code Substack API alternative: you get structured rows for Substack newsletters, writers, and discussions without building your own integration. Start with a newsletter URL, a direct article link, an author profile, or a keyword. Export the resulting dataset to JSON, CSV, or a spreadsheet and join records using the stable identifiers returned with each row. ### What Substack Scraper collects A run can contain three kinds of records. Post records describe articles with their title, subtitle, publication date, author, newsletter, engagement counts, and whether the article is available to everyone or intended for paying subscribers. Profile records describe either a person or a publication, including the public biography, external links, and visible audience estimates. Comment records contain individual comments and replies, with identifiers connecting each reply to its parent and each comment to its post. This structure lets you compare newsletters without placing comment threads inside article cells. Filter the dataset by `record_type` to work with articles, people, publications, or discussions separately. Every row also carries `_type`, `input`, and `fetched_at`, so you can identify its kind, the input that produced it, and when it was collected. ### Substack Scraper use cases - **Newsletter research.** Pull the archive of several Substack newsletters, then compare publishing frequency, likes, comments, and restacks per post to see which topics and formats resonate. - **Creator outreach and partnerships.** Search for writers by keyword, collect their public bios, links, and audience estimates, and build a shortlist of Substack writers for sponsorships, guest posts, or podcast invitations. - **Content monitoring.** Schedule a run with a recent date range to track new posts from competitors, industry analysts, or your own portfolio of newsletters, and feed them into a spreadsheet, Slack alert, or database. - **Audience and sentiment analysis.** Collect comment threads as separate rows with their parent identifiers to study how readers react to a topic, which questions they ask, and which posts start discussions. - **Text analysis and AI pipelines.** Enable public article text to build a corpus of free Substack posts for summarisation, topic modelling, or retrieval, with paid previews clearly labelled. ### Input Provide at least one value in `urls` or `keywords`. The `urls` list accepts newsletter homepages, Substack subdomains, bare subdomain names, custom newsletter domains, direct `/p/` article links, public author handles prefixed with `@`, and profile URLs on Substack. Links in the form `open.substack.com/pub/.../p/...` and `substack.com/home/post/p-...` are also supported. An unrecognized entry produces a warning while valid entries continue. A completely invalid list returns an input error. Set `maxPostsPerNewsletter` to control the number of matching posts collected from each newsletter. The default is 20, with a maximum of 5,000. Choose `sort: "new"` for recent posts or `sort: "top"` for the order Substack supplies as its top archive. Use `startDate` and `endDate` in `YYYY-MM-DD` form to select an inclusive UTC date range. Set `onlyFree: true` to exclude posts for paying subscribers. `contentType` accepts `all`, `newsletter`, `podcast`, or `thread`; availability depends on what the publication has posted. Keyword searches use `searchType: "posts"` by default. Choose `profiles` to find people, or `publications` to discover newsletters associated with matching author profiles. Publication discovery is derived from those author matches, rather than an exhaustive catalogue of newsletters containing the search term. Set `maxSearchResultsPerKeyword` to limit primary results from each keyword; the default is 20. Additional publication profiles are separate records. Optional article text is disabled by default. Enable `includeContent` to receive `body_html` and plain `text`. Enable `includeComments` to collect comments as separate rows, and use `maxCommentsPerPost` to set the limit, including replies. The default comment limit is 20. `includePublicationInfo` defaults to true and adds one profile per distinct publication encountered during the run. ### Output Posts use `post_id` and `post_url` as stable join keys. They include `title`, `subtitle`, `posted_at`, `author_id`, `author_handle`, `publication_id`, `publication_name`, `publication_url`, `likes`, `comments_count`, `shares`, `tags`, and public content when requested. `shares` reports Substack restacks. Archive posts and direct article links carry the newsletter's name and URL, including articles by guest authors and when `includePublicationInfo` is off. That option controls the additional publication profile record. If publication information cannot be retrieved, a warning explains the missing information while available post metadata is retained. Without requested content, `text`, `body_html`, and `content_status` are null; a short excerpt is stored separately and is never presented as full article text. For content requests, `content_status: "full"` means the public response included the article text for a free article. `content_status: "preview"` means a paid article returned only its publicly accessible preview. This actor does not obtain subscriber-only text. A missing content response generates a warning and keeps the available article metadata. Comments include `comment_id`, `post_id`, `parent_comment_id`, `text`, `posted_at`, author information, `likes`, and `depth`. A parent id lets you reconstruct a discussion tree. Deleted comments retain their identifier and may still have replies, but their text is null. Comments are limited in tree order, including descendants before the next root discussion. Some posts, including paid articles, show a comment count while their comments are not publicly available; the actor reports this in a warning when no comments can be returned. Profiles have `profile_kind: "user"` or `"publication"`, `profile_url`, `profile_id`, `name`, `bio`, `links`, and visible audience information. User `followers` and publication `subscriber_count` are different measurements. Publication `followers` repeats its subscriber estimate because newsletters have no separate follower count. Subscriber figures are rounded public estimates supplied by Substack; hidden counts remain null. They are suitable for audience comparisons, not precise subscriber accounting. #### Example output A post record from a real run with `urls: ["https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit"]` and `includeContent: true`, shortened to the main fields (the full record also contains `body_html`, `text`, `bylines`, `cover_image`, and more): ```json { "record_type": "post", "post_id": "215694124", "post_url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit", "title": "All of the Lenny & Friends Summit talks are now online!", "subtitle": "Plus, some reflections and takeaways from the day", "posted_at": "2026-09-29T13:15:57.512Z", "audience": "everyone", "is_paid": false, "wordcount": 1276, "likes": 277, "comments_count": 6, "shares": 4, "author_name": "Lenny Rachitsky", "author_handle": "lenny", "publication_name": "Lenny's Newsletter", "publication_url": "https://www.lennysnewsletter.com", "content_status": "full" } ``` A paid article written by guest authors, from a run with a direct article link, `includeContent: true`, and `includePublicationInfo: false`. The newsletter's name and URL are still filled, and `content_status` shows that only the public preview was returned: ```json { "record_type": "post", "post_id": "216168140", "post_url": "https://www.lennysnewsletter.com/p/advanced-evals-how-to-find-and-fix", "title": "Advanced evals: How to find (and fix) hidden AI failures in your product", "posted_at": "2026-09-22T12:45:14.998Z", "audience": "only_paid", "is_paid": true, "likes": 347, "comments_count": 4, "shares": 20, "author_name": "Hamel Husain", "author_handle": "hamelhusain", "publication_id": "10845", "publication_name": "Lenny's Newsletter", "publication_url": "https://www.lennysnewsletter.com", "content_status": "preview" } ``` A publication profile from a run with `urls: ["garymarcus"]`: ```json { "record_type": "profile", "profile_kind": "publication", "profile_id": "888615", "profile_url": "https://garymarcus.substack.com", "handle": "garymarcus", "name": "Marcus on AI", "subscriber_count": 119000, "subscriber_count_text": "Hundreds of thousands of subscribers", "paid_subscribers_text": "Thousands of paid subscribers", "author_name": "Gary Marcus", "created_at": "2022-05-14T14:09:01.902Z", "language": "en", "payments_enabled": true } ``` ### Substack Scraper compared with other options - **Copying by hand** works for one or two newsletters but does not scale to archives of hundreds of posts, and engagement counts change after publication. - **RSS feeds** list recent posts of one newsletter, but they carry no like, comment, or restack counts, no author or publication profiles, and no comment threads. This actor returns all of those as structured rows and can go back up to 5,000 posts per newsletter. - **Open-source libraries for the unofficial Substack API** need code, hosting, and maintenance whenever the site changes. This actor runs in the cloud, exports to JSON, CSV, or Excel, and plugs into Apify schedules, webhooks, and integrations. - **Other Substack scrapers in the Store** differ in scope. This one combines posts, comments, writer profiles, publication profiles, keyword search, date filters, and optional public text in one run, separates each record type with `record_type`, and marks paid previews explicitly instead of presenting them as full articles. ### Substack Scraper pricing Charges correspond to delivered records, with separate events for posts, comments, profiles, and full public article content. A paid article preview does not count as full content. The run-start event is charged once per actor run, including a run that finds no records. Empty results carry no record or content charges. Consult the actor's current pricing panel for event rates. Non-paying users receive at most ten dataset rows in total. Posts, profiles, and comments share that allowance. A newsletter profile and its discussion can therefore use part of the allowance before ten posts have been returned. Disable optional comments or publication information when you want the allowance to focus on articles. ### Larger collections and monitoring The actor follows continuation automatically to collect larger requests. If the run time limit interrupts a collection, the output reports that it is partial and includes the continuation needed to resume with the same input. For scheduled monitoring, use a narrow date range and store `post_id` or `comment_id` in your destination to detect new records across separate runs. A run removes repeated post and profile identifiers within its own collection; it does not remember your previous runs. Public URLs can disappear or become unavailable. Warnings identify entries or optional phases that did not complete while other records remain available. A run with no matching public data finishes with an empty result status. The actor returns only public source data and never substitutes fabricated profiles or article text for an unavailable source. ### FAQ #### Can I collect paid newsletter articles? You can collect their public metadata and public preview. Subscriber-only text is outside the available output. The `is_paid` and `content_status` fields make that distinction explicit in each post record. #### Does publication search find every matching newsletter? No. It derives publication records from matching author profiles. A newsletter whose author does not match the keyword may be absent. Use its direct publication URL when you already know the newsletter you need. #### Are subscriber numbers exact? No. Substack publishes rounded estimates and sometimes hides them. The actor preserves the visible estimate and its descriptive text, and returns null when the count is unavailable. #### Can I combine several newsletters and keywords? Yes. Include multiple entries in the two input lists. Limits apply per publication or keyword, while repeated post and profile records are removed across the whole run. The originating input remains on each delivered row. #### Is this the official Substack API? No. This actor is not affiliated with Substack. It reads the same public pages and data a visitor sees in a browser and returns them as structured records, so it only covers public information and needs no Substack account. #### How do I find Substack newsletters about a topic? Add a keyword and set `searchType` to `publications` to discover newsletters through matching writers, or `posts` to find articles mentioning the term. Then pass the newsletters you want into `urls` to collect their full archive. #### Can I export the results to CSV or Excel? Yes. Every run stores its rows in an Apify dataset that you can download as JSON, CSV, Excel, or XML, or read through the Apify API and integrations. # Actor input Schema ## `urls` (type: `array`): Publication URLs or subdomains, post URLs, and author profiles. Public custom domains are supported. ## `keywords` (type: `array`): Search terms for posts, profiles, or publications. ## `searchType` (type: `string`): Publication search derives newsletters from matching author profiles; coverage depends on the author match. ## `sort` (type: `string`): Archive order. Top uses the order provided by Substack. ## `contentType` (type: `string`): Filter posts by their Substack content type. ## `maxPostsPerNewsletter` (type: `integer`): Maximum matching posts per publication. ## `maxSearchResultsPerKeyword` (type: `integer`): Maximum matching primary results per keyword; extra publication rows do not count toward this limit. ## `maxCommentsPerPost` (type: `integer`): Maximum comments per post, including replies in tree order. ## `includeContent` (type: `boolean`): Include publicly available HTML and plain text. Paid posts return a preview only. ## `includeComments` (type: `boolean`): Include comments as separate rows, including replies. ## `includePublicationInfo` (type: `boolean`): Include one publication profile per distinct newsletter. ## `onlyFree` (type: `boolean`): Keep posts available to everyone; exclude paid posts. ## `startDate` (type: `string`): Inclusive start date in YYYY-MM-DD (UTC). ## `endDate` (type: `string`): Inclusive end date in YYYY-MM-DD (UTC). ## Actor input object example ```json { "urls": [ "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit" ], "searchType": "posts", "sort": "new", "contentType": "all", "maxPostsPerNewsletter": 20, "maxSearchResultsPerKeyword": 20, "maxCommentsPerPost": 20, "includeContent": false, "includeComments": false, "includePublicationInfo": true, "onlyFree": false } ``` # Actor output Schema ## `results` (type: `string`): Public data rows distinguished by record_type. ## `output` (type: `string`): Run counts, warnings, continuation and status. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "urls": [ "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit" ] }; // Run the Actor and wait for it to finish const run = await client.actor("s-r/substack-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "urls": ["https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit"] } # Run the Actor and wait for it to finish run = client.actor("s-r/substack-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "urls": [ "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit" ] }' | apify call s-r/substack-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,s-r/substack-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/1zf6EHGYaT6fqlNqu/builds/TvIeTfXw0B8CGxgVj/openapi.json