# Substack Posts & Newsletters Scraper (`thisisb3/substack-scraper`) Actor Scrape Substack posts and find newsletters by category leaderboard: titles, dates, likes, comments, free-post text, subscriber counts, prices, author social links and recommendations. - **URL**: https://apify.com/thisisb3/substack-scraper.md - **Developed by:** [ThisIsB3](https://apify.com/thisisb3) (community) - **Categories:** News, Lead generation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing from $0.80 / 1,000 posts This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Substack Posts & Newsletters Scraper Scrape **Substack posts** and **find Substack newsletters** by category leaderboard. Get post titles, dates, likes, comments, restacks, word counts and the full text of free posts, plus publication details nobody else gives you in one place: **leaderboard rank, subscriber count, subscription prices, authors' social links and each newsletter's recommendations**. No login, no browser, no code. Export to JSON, CSV or Excel, or use it through the Apify API. ### What does Substack Posts & Newsletters Scraper do? It reads Substack's public data the same way the Substack website does, with three ways in: - **Publications**: paste `name.substack.com` or a custom domain such as `www.astralcodexten.com`. You get the latest posts (as many as you want, newest first) and one row with the publication's details. - **Categories**: pick Substack categories such as Technology, Business or Finance. You get the **Bestsellers leaderboard** for each (ranked), or every newsletter in the category, as publication rows ready for outreach or research. - **Post URLs**: paste links to individual posts, including links copied from the Substack app. Every row has a `type` field (`post` or `publication`), so one run can mix both. The dataset has a **Posts** view and a **Publications** view. ### Why use it? - **Newsletter discovery**: list the top newsletters in any category with their rank, size, price and authors. Turn on recommendations to follow each newsletter's recommendation network and find the ones around it. - **Sponsorship and partnership research**: compare subscriber counts, paid-subscriber bands and prices, then reach authors through the public social links on their Substack profile (X/Twitter, LinkedIn, YouTube, Bluesky, websites). - **Competitor monitoring**: schedule a daily or weekly run on the newsletters you follow and track what they publish, how often, and how readers react (likes, comments, restacks). - **Media research**: build a dataset of posts on a beat, with dates, sections, tags, podcast episodes and word counts, plus the text of free posts. It is built for reliability: it uses Substack's public JSON (no fragile page parsing, no browser), slows down automatically if Substack asks it to, keeps partial results when a single request fails, and is checked every day by an automated test. ### How to use it 1. Click **Try for free**. 2. Add newsletters to **Publications**, pick **Categories**, or paste **Post URLs** (any combination). 3. Choose what you want: posts per publication, full text, publication details, recommendations. 4. Click **Start** and download your data when the run finishes (usually seconds to a few minutes). You can also call it from the Apify API, schedule it, or connect it to Make, Zapier, Google Sheets and other integrations. ### How much does it cost? You pay only for the rows you receive. There is no monthly fee. | Row | Event | Price | | ------------------------------------------------------------------- | ---------------- | ------------------------------------------------ | | Post (all fields except the text) | `post` | **$1.00 per 1,000** | | Post with full text (free posts only) | `post-full-text` | **$2.00 per 1,000**, charged *instead of* `post` | | Publication (details, rank, authors, social links, recommendations) | `publication` | **$3.00 per 1,000** | Examples: the latest 50 posts of 20 newsletters cost about $1.06 (1,000 posts plus 20 publication rows). The Bestsellers top 25 of 10 categories costs $0.75. Apify's free plan includes monthly platform credit, enough for thousands of posts every month. Set a maximum cost per run in the run options and the scraper stops exactly at that budget. ### Input | Field | Description | | ------------------------------- | ------------------------------------------------------------------------------------------------------------ | | Publications | Newsletters to scrape: `name.substack.com`, a custom domain, or any link to the publication. | | Categories | Substack categories to discover newsletters in (publication rows only). | | Post URLs | Individual posts, e.g. `https://name.substack.com/p/post-title` or `https://substack.com/@name/p-123456789`. | | Max posts per publication | Latest posts per publication (default 20). 0 = publication rows only. | | Include full text of free posts | Adds plain text and HTML for free, public posts (default off). | | Include publication details | One row per publication in Publications (default on). | | Include recommended newsletters | Adds each publication's recommendations to its row (default off). | | Category list | `bestsellers` (paid leaderboard, ranked; default) or `all` (every newsletter in the category). | | Max publications per category | How many publications per category, from the top (default 25). | | Maximum results | Stop after this many rows in total (default 1,000; 0 = no limit). | | Max concurrency | Parallel requests, also the most started per second (default 4). | Example input: ```json { "publications": ["https://on.substack.com", "www.astralcodexten.com"], "categories": ["technology", "business"], "maxPostsPerPublication": 50, "includeFullText": false, "includeRecommendations": true, "maxPublicationsPerCategory": 25 } ``` ### Output #### Post rows | Field | Description | | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | | `title`, `subtitle`, `url`, `slug` | The post and its link. | | `publishedAt` | Publish date (ISO 8601). | | `audience`, `isPaid` | `free` (public), `subscribers` (free sign-up), `paid` or `founding`; `isPaid` is true for paywalled posts. | | `postType`, `isPodcast` | `newsletter`, `podcast`, `thread` or `video`. | | `reactionCount`, `restackCount`, `commentCount` | Likes, restacks and comments. | | `wordCount`, `tags`, `sectionName`, `coverImageUrl` | Length, tags, section and cover image. | | `authors` | Bylines (`id`, `name`, `handle`). | | `publicationId`, `publicationName`, `publicationSubdomain`, `publicationUrl` | The newsletter the post belongs to. | | `fullText`, `fullTextHtml` | Text of free posts when "Include full text" is on; otherwise `null`. | A real post row (from the default input): ```json { "type": "post", "id": 218520648, "title": "A CEO tried to bully a journalist into silence. He failed.", "subtitle": "This week, we notched another win for free speech", "url": "https://on.substack.com/p/a-ceo-tried-to-bully-a-journalist", "slug": "a-ceo-tried-to-bully-a-journalist", "publishedAt": "2026-10-02T17:10:42.907Z", "audience": "free", "isPaid": false, "postType": "newsletter", "isPodcast": false, "reactionCount": 3731, "restackCount": 587, "commentCount": 0, "wordCount": 323, "tags": [], "sectionName": "News & Views", "coverImageUrl": "https://substackcdn.com/image/fetch/$s_!qRDf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F4b63deba-7410-45fd-a672-d7a41e2e3924_3240x2160.png", "authors": [], "publicationId": 1, "publicationName": "On Substack", "publicationSubdomain": "on", "publicationUrl": "https://on.substack.com", "fullText": null, "fullTextHtml": null, "scrapedAt": "2026-10-07T16:37:42.305Z" } ``` #### Publication rows | Field | Description | | ----------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | `name`, `subdomain`, `customDomain`, `url` | The newsletter and its addresses. | | `description`, `logoUrl`, `language` | Tagline, logo and language. | | `category`, `leaderboardRank` | Substack category and position on its Bestsellers leaderboard, when known. | | `subscriberCount` | Subscribers as rounded and shown publicly by Substack (when the publisher shows it). | | `paidSubscribersEstimate` | Substack's public paid-subscriber band, e.g. "Thousands of paid subscribers". | | `hasPaidSubscriptions`, `monthlyPrice`, `annualPrice`, `currency` | Whether it sells subscriptions, and the cheapest plans. | | `createdAt`, `firstPostAt`, `hasPodcast` | Age and podcast. | | `authors` | Owner first, up to 3 authors: `name`, `handle`, `bio`, `photoUrl`, `profileUrl`, `isOwner`, `socialLinks` (`platform`, `url`, `title`). | | `recommendations` | Newsletters it recommends (`id`, `name`, `url`, `hasPaidSubscriptions`, `description`), when turned on. | A real publication row from a Technology category run (social links and recommendations shortened): ```json { "type": "publication", "id": 458709, "name": "The Pragmatic Engineer", "subdomain": "pragmaticengineer", "customDomain": "newsletter.pragmaticengineer.com", "url": "https://newsletter.pragmaticengineer.com", "description": "Big Tech and startups, from the inside. Highly relevant for software engineers, AI engineers and engineering leaders, useful for those working in tech.", "logoUrl": "https://substackcdn.com/image/fetch/$s_!eod5!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F27076b05-e3f4-4d05-9621-3da4f7f189cd_300x300.png", "category": "Technology", "leaderboardRank": 3, "hasPaidSubscriptions": true, "subscriberCount": 1100000, "paidSubscribersEstimate": "Tens of thousands of paid subscribers", "monthlyPrice": 15, "annualPrice": 150, "currency": "USD", "language": "en", "createdAt": "2021-08-25T13:08:12.798Z", "firstPostAt": "2021-08-30T11:24:02.082Z", "hasPodcast": true, "authors": [ { "name": "Gergely Orosz", "handle": "pragmaticengineer", "bio": "Big Tech and startups from the inside. Especially relevant for software engineers / AI engineers, useful for anyone working in tech.", "photoUrl": "https://substack-post-media.s3.amazonaws.com/public/images/58fed27c-f331-4ff3-ba47-135c5a0be0ba_400x400.png", "profileUrl": "https://substack.com/@pragmaticengineer", "isOwner": true, "socialLinks": [ { "platform": "twitter", "url": "https://x.com/GergelyOrosz", "title": "GergelyOrosz" }, { "platform": "linkedin", "url": "https://www.linkedin.com/in/gergelyorosz/", "title": null }, { "platform": "youtube", "url": "https://www.youtube.com/@pragmaticengineer", "title": null } ] } ], "recommendations": [ { "id": 2935724, "name": "charity.wtf", "subdomain": "charitydotwtf", "customDomain": "charity.wtf", "url": "https://charity.wtf", "hasPaidSubscriptions": false, "description": null } ], "scrapedAt": "2026-10-07T16:37:56.734Z" } ``` ### Tips - Start with the defaults to preview the data, then raise **Max posts per publication** for full archives. - For a lead list, use **Categories**: they return publication rows only. For your own list of newsletters without posts, set **Max posts per publication** to 0. - Want posts from newsletters you discovered? Copy their `url` values into **Publications** and run again. - Schedule weekly runs on your competitor list to track new posts and engagement over time. - Bare custom domains (`example.com`) are retried on `www.example.com` automatically. ### FAQ **Is it legal to scrape Substack?** This scraper only reads public data that any logged-out visitor can see, through the same public endpoints the Substack website uses. It never logs in and never bypasses a paywall. Publication and author details are what publishers choose to show publicly. You are responsible for how you use the data: respect authors' copyright (don't republish their posts), and check the laws that apply to you, such as GDPR, before contacting people. **What safeguards are built in?** - **Paywalled posts never include text.** Full text is only returned for free, public posts. For paid, founding-member and subscriber-only posts you get the metadata (title, date, likes...) but never the body, even if Substack sends a preview. - **No email addresses.** Email links are dropped and anything that looks like an email address is replaced with `[email removed]` in every field. - **Polite by default.** A few requests per second at most, and it pauses automatically when Substack signals rate limiting. **Is there an official Substack API?** Substack has no official public API for this data. This scraper gives you a stable, documented output format instead, and it is fixed quickly if Substack changes something. **Can I search newsletters by keyword?** Not at the moment: Substack's keyword search does not return results to logged-out visitors. Use **Categories** to discover newsletters, and turn on **recommendations** to explore the network around the ones you find. **Why is `subscriberCount` empty for some newsletters?** Some publishers hide their subscriber count. The scraper never guesses; it returns `null`. **The scraper returned fewer results than expected.** Check that the publication or post opens in a normal browser without logging in. If the site changed, open an issue on the **Issues** tab and it will be fixed quickly. **Can I get other fields, such as comments?** Open an issue on the **Issues** tab with the field you need. # Actor input Schema ## `publications` (type: `array`): Newsletters to scrape. Paste the address as you see it: name.substack.com, a custom domain such as www.astralcodexten.com, or any link to the publication. You get its latest posts and, unless you switch it off below, one row with the publication's details. ## `categories` (type: `array`): Discover newsletters by Substack category. Each category returns its top publications as publication rows: name, leaderboard rank, subscriber count, prices, authors and their social links. Posts are not scraped for these; paste the ones you like into Publications. ## `postUrls` (type: `array`): Individual posts to scrape, e.g. https://name.substack.com/p/post-title. Links copied from the Substack app (substack.com/@name/p-123456789) work too. ## `maxPostsPerPublication` (type: `integer`): Latest posts to scrape from each publication in Publications, newest first. Set to 0 to skip posts and get only the publication rows. ## `includeFullText` (type: `boolean`): Adds the post text (plain text and HTML) for free, public posts. Paywalled and subscriber-only posts never include text. Rows with text are billed as "post with full text". ## `includePublicationInfo` (type: `boolean`): Adds one row per publication in Publications: name, category, leaderboard rank, subscriber count, prices, authors and their public social links. ## `includeRecommendations` (type: `boolean`): Adds the newsletters each publication recommends (its recommendation network) to its publication row. One extra request per publication. ## `categoryRanking` (type: `string`): Bestsellers is Substack's paid leaderboard for the category (ranked, with the leaderboard rank in each row). All newsletters lists every publication in the category, including free ones, without a leaderboard rank. ## `maxPublicationsPerCategory` (type: `integer`): How many publications to return from each category, from the top of the list. Substack's leaderboard pages show the top 25. ## `maxItems` (type: `integer`): Stop after this many rows in total (posts plus publications). Set to 0 for no limit. You are only charged for rows you receive. ## `maxConcurrency` (type: `integer`): How many requests run in parallel; also the most requests started per second. The default is deliberately polite, and the scraper slows down automatically if Substack asks it to. ## Actor input object example ```json { "publications": [ "https://on.substack.com" ], "maxPostsPerPublication": 20, "includeFullText": false, "includePublicationInfo": true, "includeRecommendations": false, "categoryRanking": "bestsellers", "maxPublicationsPerCategory": 25, "maxItems": 1000, "maxConcurrency": 4 } ``` # Actor output Schema ## `results` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "publications": [ "https://on.substack.com" ] }; // Run the Actor and wait for it to finish const run = await client.actor("thisisb3/substack-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "publications": ["https://on.substack.com"] } # Run the Actor and wait for it to finish run = client.actor("thisisb3/substack-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "publications": [ "https://on.substack.com" ] }' | apify call thisisb3/substack-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,thisisb3/substack-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/CHi4hfXwagiZar7vG/builds/5eZJcMmYmozHUZ0QL/openapi.json