# Substack Scraper - Newsletter Posts, Full Text & RAG Chunks (`scrapewise/substack-scraper`) Actor Scrape Substack newsletters: posts with likes, comments, paywall flag and authors; full text as markdown with RAG chunks; publication search, post search and category rankings with subscriber counts. Only new posts on daily runs. Error rows are free. - **URL**: https://apify.com/scrapewise/substack-scraper.md - **Developed by:** [Scrapewise Data](https://apify.com/scrapewise) (community) - **Categories:** Social media, AI, News - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.68 / 1,000 post delivereds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Substack Scraper: newsletter posts, full text as markdown, RAG chunks and rankings Scrape [Substack](https://substack.com) newsletters **without an account or cookies**. Give a publication (on substack.com or on its own domain), a post link, an author's @handle, a search term or a category, and get every post with title, subtitle, date, authors, likes, comments, restacks, word count, cover image, audience and a clear **paywall flag**. Turn on full text to get each public post as **clean markdown**, or ask for **RAG chunks** ready for embeddings. Built for **daily monitoring**: schedule the same newsletters every morning and get **only the posts published since the last run**. A quiet day costs nothing. ### At a glance - **Price per 1,000 posts, Free plan:** US$ 0.80 (down to US$ 0.68 on Gold and above) - **Full post text as markdown (optional):** US$ 1.067 per 1,000, on top; paid-post previews are free - **Publication rows (subscribers, prices, rank):** US$ 1.067 per 1,000 - **Fee per run start:** None - **Publications on substack.com and on their own domains:** Yes - **Post search and publication search across all of Substack:** Yes - **Category rankings (bestsellers, rising, all):** Yes - **Only new posts since the last run (`onlyNew`):** Yes, free when nothing is new - **RAG chunks with section headings and overlap:** Yes, same price as the text - **Paid content:** Never; paid posts come with the public preview only, flagged `isPaywalled` - **Error rows (bad URL, unknown publication, no results):** Free, with an `errorCode` ### One real row From a test run on 2026-10-21 (a Noahpinion post link, `chunkSizeWords: 300`), shortened; this post has 28 chunks: ```json { "type": "post", "id": 221088139, "title": "Can a Million AI Agents Create Superintelligence?", "subtitle": "Ramez Naam gives a skeptic’s take on agent swarms.", "url": "https://www.noahpinion.blog/p/can-a-million-ai-agents-create-superintelligence", "publishedAt": "2026-10-19T04:33:30Z", "postType": "newsletter", "audience": "everyone", "isPaywalled": false, "authors": [{ "name": "Ramez Naam", "handle": "rameznaam", "profileUrl": "https://substack.com/@rameznaam" }], "reactionCount": 241, "commentCount": 17, "restacks": 24, "wordCount": 7178, "coverImage": "https://substackcdn.com/image/fetch/...6335ee9e-3d13-4a47-9213-3c1d01cd8eff_1075x717.jpeg", "publicationName": "Noahpinion", "publicationUrl": "https://www.noahpinion.blog", "markdown": "![](https://substackcdn.com/image/fetch/...)\n\n*Art by GPT-6*\n\n*Ramez Naam is, in my opinion, one of the best futurists alive. But although [he has a Substack](https://www.rameznaam.com/) (which you should definitely follow), he doesn’t write nearly enough...*", "textWordCount": 7061, "textIsPreview": false, "chunks": [ { "index": 0, "text": "*Art by GPT-6*\n\n*Ramez Naam is, in my opinion, one of the best futurists alive. But although [he has a Substack](https://www.rameznaam.com/) (which you should definitely follow), he doesn’t write nearly enough...*", "wordCount": 328, "section": null, "postId": 221088139, "postTitle": "Can a Million AI Agents Create Superintelligence?", "postUrl": "https://www.noahpinion.blog/p/can-a-million-ai-agents-create-superintelligence" } ] } ``` And the publication row of the same newsletter (`includePublicationDetails`): ```json { "type": "publication", "id": 35345, "name": "Noahpinion", "url": "https://www.noahpinion.blog", "subdomain": "noahpinion", "authorName": "Noah Smith", "subscriberCount": 460000, "subscriberCountText": "Over 460,000 subscribers", "paidSubscribersText": "Tens of thousands of paid subscribers", "hasPaidSubscriptions": true, "monthlyPrice": 10.0, "yearlyPrice": 100.0, "priceCurrency": "USD", "hasPodcast": true } ``` ### What you can do with it - **AI and RAG:** feed the newsletters you trust into a vector database. Full text in markdown, chunks with section headings and overlap, every chunk linked back to its post. - **Newsletter monitoring:** follow competitors, analysts or your industry's top writers every morning and send only the new posts to Slack, e-mail, Notion or Google Sheets (`onlyNew`). - **Market research and sponsorships:** find newsletters on a topic with subscriber counts and prices, or pull a category's bestseller and rising rankings to pick where to advertise or whom to pitch. - **Content and trend analysis:** which posts get the most likes, comments and restacks, how often a publication writes, how much of it is paywalled. - **Media and PR:** search all of Substack for posts that mention your brand, product or topic. ### Input | Field | What it does | |---|---| | `startUrls` | Publications (`https://www.noahpinion.blog`, `https://name.substack.com`, `oneusefulthing`), post links (`…/p/post-name`, `substack.com/@handle/p-123`, `open.substack.com/pub/name/p/post`), or `@handle` | | `searchPosts` | Terms searched across every Substack post (`AI agents`, `interest rates`) | | `searchPublications` | Terms searched across Substack publications (`climate`, `venture capital`) | | `categories` | Substack categories (`Technology`, `Business`, `Finance`, `Culture`, `U.S. Politics`, `Science`...) | | `categoryRanking` | `bestsellers` (default), `rising` or `all` | | `maxPostsPerPublication` | Posts per publication, default 20; 0 = the whole archive (up to 10,000) | | `maxResultsPerSearch` | Results per search or category, default 20 | | `postsPerFoundPublication` | Latest posts of every publication found by a search or category, default 0 | | `sortBy` | `newest` (default) or `top` (the publication's most popular posts) | | `includePublicationDetails` | One publication row per publication in `startUrls` | | `includeFullText` | Full text of each public post as markdown | | `chunkSizeWords`, `chunkOverlapWords` | RAG chunks (turns on the full text); 0 = off | | `onlyNew` | Deliver only posts not delivered in earlier runs | | `proxyConfiguration` | Default Apify Proxy | Example: new posts of three AI newsletters every morning, with markdown and chunks for a RAG pipeline: ```json { "startUrls": ["https://www.oneusefulthing.org", "https://www.noahpinion.blog", "https://www.astralcodexten.com"], "maxPostsPerPublication": 10, "chunkSizeWords": 300, "onlyNew": true } ``` Or the 50 bestsellers in Technology with their three latest posts: ```json { "categories": ["Technology"], "categoryRanking": "bestsellers", "maxResultsPerSearch": 50, "postsPerFoundPublication": 3 } ``` ### How daily monitoring works With `onlyNew` the Actor remembers what it delivered for each publication (by Substack's own publication id, so `noahpinion`, `@noahpinion` and `https://www.noahpinion.blog` share one memory), each search and each category, in a named key-value store in your own account, `substack-scraper-state`. - The first run delivers up to the limit and marks everything it read as seen. - Later runs read each publication newest first and stop when they reach posts already delivered. A day with nothing new delivers nothing and costs nothing. - Searches and categories deliver only results not delivered before. - Delete the key-value store to start over. ### Paywalls The Actor reads only what Substack shows to anyone without an account. Paid posts (`audience: "only_paid"` or `"founding"`) come with `isPaywalled: true` and every field of the post; with full text on, you get the **public preview** that Substack shows above the paywall, marked `textIsPreview: true`, and the text event is **not charged**. Nothing behind the paywall is read, and no login is used. ### Output Every post row: `id`, `title`, `subtitle`, `url`, `slug`, `publishedAt`, `updatedAt`, `postType` (`newsletter`, `podcast`, `thread`, `video`), `audience` (`everyone`, `only_free`, `only_paid`, `founding`), `isPaywalled`, `authors` (name, handle, profile link), `authorNames`, `reactionCount`, `reactions`, `commentCount`, `restacks`, `wordCount`, `coverImage`, `previewText`, `language`, `sectionName`, `tags`, `podcastDurationSeconds`, `publicationId`, `publicationName`, `publicationUrl`, `position`, `source` (the input it came from), `scrapedAt`. With full text: `markdown`, `textWordCount`, `textIsPreview`, and `chunks` (`index`, `text`, `wordCount`, `section`, `postId`, `postTitle`, `postUrl`) when chunking is on. If a text cannot be read, the post still comes with `textError` and only the post is charged. Every publication row: `id`, `name`, `url`, `subdomain`, `customDomain`, `description`, `authorName`, `authorHandle`, `authorProfileUrl`, `logo`, `language`, `publicationType`, `createdAt`, `firstPostAt`, `subscriberCount` (when the publication shows it), `subscriberCountText`, `paidSubscribersText`, `bestsellerTier`, `hasPaidSubscriptions`, `monthlyPrice`, `yearlyPrice`, `foundingMemberPrice`, `priceCurrency`, `hasPodcast`, `podcastTitle`, `category`, `categoryRanking`, `position` (rank in the search or category), `source`, `scrapedAt`. Dataset views: **Posts**, **Full text**, **Publications** and **RAG chunks** (one row per chunk). People data is limited to the author's public name and handle, which are the byline of the post. No e-mails, photos, bios or social accounts. ### Error codes (never charged) | `errorCode` | Meaning | |---|---| | `INVALID_INPUT` | A value the Actor does not understand (unknown category, a substack.com page that is not a publication, post or profile, a bad option) | | `NOT_FOUND` | No Substack publication at this address or name (including sites that left Substack), or the post was removed | | `NO_RESULTS` | A search or category with no results, or a publication with no public posts | | `PUBLICATION_INFO_UNAVAILABLE` | The publication details could not be read; its posts still come | | `BLOCKED` | Substack refused every attempt; run again | | `NOT_REACHED` | The run timeout arrived before this input | | `TIMEOUT_TOO_SHORT` | The run timeout is too short to read even one page; raise it | | `UNEXPECTED` | A page came in a shape we did not expect; the rest of the run goes on | ### Pricing | Event | Per 1,000 | |---|---| | Post delivered | US$ 0.80 | | Full post text (public posts only, with `includeFullText` or chunks) | US$ 1.067 | | Publication delivered (searches, categories, `includePublicationDetails`) | US$ 1.067 | A daily `onlyNew` run on 20 newsletters that finds 15 new posts costs US$ 0.012; with full text, about US$ 0.03. The first run of a 500-post archive with full text costs about US$ 0.93. Error rows, duplicates, posts already delivered and paid-post previews are free, and there is no start fee. ### Limitations - Paid content is never read: paid posts come with the public preview only. - Subscriber counts appear only when the publication shows them on Substack; otherwise `subscriberCount` is empty and `subscriberCountText` or `paidSubscribersText` may still carry Substack's rounded label. - Substack's post search returns the posts it finds most relevant, not every post that mentions the term. - Comment texts are not collected, only counts. - A site that moved off Substack (to its own platform) is reported as `NOT_FOUND`, not filled with its old archive. ### FAQ **Do I need a Substack account?** No. No login and no cookies; the default proxy works. **Does it work with custom domains?** Yes. `https://www.noahpinion.blog`, `noahpinion.blog`, `NOAHPINION.BLOG`, `https://noahpinion.substack.com`, `@noahpinion` and `noahpinion` all reach the same publication. **Can I get the newsletter as text for ChatGPT, Claude or my own RAG app?** Yes. Turn on `includeFullText` for markdown, or set `chunkSizeWords` (300 is a good start) for chunks with section headings, ready for embeddings. **Can I use it from n8n, Make, Zapier or an AI agent?** Yes, through the Apify app or the Apify MCP server. Error rows are free, so an agent can explore cheaply. **What if a run is cut by its timeout?** The Actor stops shortly before the limit and ends successfully with what it delivered, telling you in the status message how to get the rest. ### Changelog - **0.1 (2026-10-07):** first version. Publications by URL, custom domain, @handle or name; post links; post search; publication search; category rankings; full text as markdown; RAG chunks; paywall flag with free previews; `onlyNew` for daily schedules; free error rows. # Actor input Schema ## `startUrls` (type: `array`): Publications or posts, one per line: a publication address on substack.com or on its own domain (for example https://www.noahpinion.blog, https://www.oneusefulthing.org or https://name.substack.com), a post link (.../p/post-name), an author's @handle, or just the newsletter name. Upper case, missing https and extra paths are fine. Each publication returns its latest posts (title, date, likes, comments, restacks, paywall flag). Leave every list empty to run an example publication. Also accepts 'urls', 'publications', 'publicationUrls' or 'postUrls'. ## `searchPosts` (type: `array`): Search terms, one per line (for example AI agents, interest rates, college football). Searches posts across all of Substack, most relevant first, up to 'Max results per search or category'. Each row keeps the term it came from. Also accepts 'postSearch' or 'searchTerms'. ## `searchPublications` (type: `array`): Search terms, one per line (for example climate, venture capital, personal finance). Returns matching newsletters with author, subscriber count when public and paid plan prices, charged as publication rows. Add 'Latest posts of each publication found' to also get their posts. Also accepts 'publicationSearch'. ## `categories` (type: `array`): Substack categories, one per line, as named on Substack: Technology, Business, Finance, Culture, U.S. Politics, World Politics, Science, Health & Wellness, Sports, Food & Drink, Crypto, History, Fiction, Music, Education and more. Returns the category ranking chosen below with each publication's rank, charged as publication rows. Also accepts 'category'. ## `categoryRanking` (type: `string`): Which list to read for each category: Bestsellers (default, Substack's paid leaderboard), Rising (the trending list) or All (every publication Substack lists in the category). Used only with Categories. ## `maxPostsPerPublication` (type: `integer`): Stop each publication after this many posts (default 20). 0 means the whole archive, up to 10,000 posts. You pay per post delivered, so start low for large archives. Also accepts 'maxPosts' or 'maxItems'. ## `maxResultsPerSearch` (type: `integer`): How many results to take from each search or category (1 to 1,000, default 20): posts for a post search, publications for a publication search or category. ## `postsPerFoundPublication` (type: `integer`): For publication searches and categories: also delivers this many latest posts of every publication found (for example 5). 0 (default) returns publications only. Each post is charged as a post, so 50 publications with 5 posts each adds 250 posts. ## `sortBy` (type: `string`): Order of each publication's posts: Newest (default, newest first) or Top (the publication's most popular posts). With a post limit it decides which posts you get. 'Only new posts' always reads newest first. Also accepts 'sort'. ## `includePublicationDetails` (type: `boolean`): Also delivers one row per publication given in the URLs above with its name, description, author, subscriber count when public, paid plan prices and podcast. Charged as a publication row. Off by default. Also accepts 'includePublication'. ## `includeFullText` (type: `boolean`): Also delivers the full text of every public post as clean Markdown (headings, lists, links, images), with its word count. One extra request per post, charged as a separate 'Full post text' event. Paid posts come with the free public preview only, marked textIsPreview, and are not charged for text. Off by default. Also accepts 'fullText'. ## `chunkSizeWords` (type: `integer`): 0 (default) means off. Above 0, each post also gets a 'chunks' array ready for embeddings and RAG: pieces of about this many words (up to 5,000), cut only between paragraphs (or sentences in a long paragraph), each with its section heading, post id, title and URL. Turns on the full text (and its charge). 300 is a good start. The 'RAG chunks' dataset view shows one row per chunk. ## `chunkOverlapWords` (type: `integer`): Words repeated from the end of each chunk at the start of the next one (default 50), so an idea cut in two keeps its context. Capped at half the chunk size. 0 for no overlap. Used only when 'RAG chunks: words per chunk' is above 0. ## `onlyNew` (type: `boolean`): For scheduled runs. Remembers which posts each publication (and each search or category) already delivered, in a named key-value store in your account ('substack-scraper-state'), and delivers only what was not delivered before. The first run delivers up to the limit and marks what it read as seen. Seen items are not delivered and not charged. Publications are read newest first. Delete the store to start over. Also accepts 'onlyNewPosts'. ## `proxyConfiguration` (type: `object`): The default Apify Proxy works. Keep the default. ## Actor input object example ```json { "startUrls": [ "https://www.noahpinion.blog", "https://www.oneusefulthing.org" ], "categoryRanking": "bestsellers", "maxPostsPerPublication": 20, "maxResultsPerSearch": 20, "postsPerFoundPublication": 0, "sortBy": "newest", "includePublicationDetails": false, "includeFullText": false, "chunkSizeWords": 0, "chunkOverlapWords": 50, "onlyNew": false, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `results` (type: `string`): No description ## `resultsCsv` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://www.noahpinion.blog", "https://www.oneusefulthing.org" ] }; // Run the Actor and wait for it to finish const run = await client.actor("scrapewise/substack-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": [ "https://www.noahpinion.blog", "https://www.oneusefulthing.org", ] } # Run the Actor and wait for it to finish run = client.actor("scrapewise/substack-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://www.noahpinion.blog", "https://www.oneusefulthing.org" ] }' | apify call scrapewise/substack-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,scrapewise/substack-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/goNOXIDOU4es0kShK/builds/dVXGEVkC8EzyuzFur/openapi.json