# Substack Scraper: Newsletters, Posts, Subscribers & Authors (`murch1k/substack-scraper`) Actor Scrape Substack newsletters and posts without login: subscriber counts, paid plan prices, leaderboard rank, author X/LinkedIn/website links, plus posts with likes, comments and full article text. Find newsletters by keyword or category leaderboard. Fast JSON API, no browser. Pay per result. - **URL**: https://apify.com/murch1k/substack-scraper.md - **Developed by:** [Murch1k](https://apify.com/murch1k) (community) - **Categories:** AI, Marketing, Lead generation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks - **User rating**: No ratings yet ## Pricing $3.00 / 1,000 result scrapeds This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event ## What's an Apify Actor? An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md. ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run. For examples already wired to this Actor's own input schema, see the [API](#api) section below. Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`). # README ## Substack Scraper — Newsletters, Subscribers, Posts & Full Article Text Scrape **any Substack newsletter, post or author** — no login, no browser, no API key. Find newsletters **by keyword** or straight from **Substack category leaderboards**, and get clean JSON with **subscriber counts, paid plan prices, leaderboard rank, author X/LinkedIn/YouTube/website links**, plus every post with **likes, comments, restacks, word count, paywall status and (optionally) the full article text**. Export to JSON, CSV or Excel, or send it to Google Sheets, Make, Zapier, n8n, a vector database or your AI agent. ### Who uses it - **Sponsorship & influencer marketing**: build lists of newsletters in your niche with audience size, price and author contacts. - **B2B lead generation**: writers with 10k+ subscribers, their websites and social profiles. - **Content & competitor research**: which posts get the most likes and comments, how often competitors publish, what they put behind the paywall. - **AI / RAG datasets**: full text of free posts as clean HTML and plain text. - **Investors & analysts**: track leaderboard rankings and subscriber growth over time with scheduled runs. ### What you get | Record | Fields | |---|---| | **Newsletter** (`type: "newsletter"`) | name, URL, custom domain, description, language, **subscriber count**, paid-subscriber tier, **monthly / annual price**, founding plan, **leaderboard category & rank**, bestseller tier, podcast & RSS feed, **author name, handle, bio, follower count, social links (X, LinkedIn, YouTube, Instagram, TikTok, website…)**, Stripe country, first post date, logo | | **Post** (`type: "post"`) | title, subtitle, URL, publish date, type (newsletter / podcast / thread), audience (`everyone`, `only_paid`, `founding`), **likes, comments, restacks**, word count, tags, section, authors, cover image, preview text, podcast URL & duration — and with *Include full article text*: **`contentHtml` + `contentText`** | ### Four ways to choose what to scrape 1. **Newsletter URLs** — `https://name.substack.com`, custom domains (`https://www.lennysnewsletter.com`) or just the subdomain (`astralcodexten`). 2. **Post URLs** — `https://name.substack.com/p/post-slug` (also `open.substack.com/pub/...` links). 3. **Author profiles** — `https://substack.com/@handle`. 4. **Discovery** - **Keyword search** (`searchQueries`): e.g. `marketing`, `AI startups`, `real estate investing` — returns the newsletters of matching writers. - **Category leaderboards** (`categories`): `technology`, `business`, `finance`, `culture`, `science`, `crypto`, … — *Bestsellers* (top paid, 25 per category) or *Rising / all* (up to ~525 per category). Mix them freely in one run. Newsletters found through several sources are returned once. ### Pricing **$3.00 per 1,000 results** ($0.003 per result). One result = one newsletter **or** one post. Full article text, author enrichment and retries are included — no start fee, no extra per-request charges. | Example | Results | Cost | |---|---|---| | Top 25 paid newsletters in *Technology*, details only (`scrapePosts: false`) | 25 | $0.08 | | 100 marketing newsletters + their last 10 posts | 1,100 | $3.30 | | Full archive of one newsletter (≈1,000 posts) with article text | ≈1,000 | ≈$3.00 | Use **Max results (cost cap)** to put a hard ceiling on any run. The actor also stops by itself when your spending limit is reached. ### Input example ```json { "startUrls": ["https://www.lennysnewsletter.com", "https://substack.com/@noahpinion"], "searchQueries": ["marketing"], "categories": ["technology"], "leaderboardType": "paid", "maxNewslettersPerSource": 25, "includeNewsletterDetails": true, "enrichAuthors": true, "scrapePosts": true, "maxPostsPerNewsletter": 10, "postedAfter": "30 days", "includeContent": false } ``` | Option | Default | Notes | |---|---|---| | `maxNewslettersPerSource` | 25 | Per keyword / category. | | `includeNewsletterDetails` | `true` | Turn off if you only want posts. | | `enrichAuthors` | `true` | Adds author social links and follower count. | | `scrapePosts` | `true` | Turn off for a pure newsletter/lead list. | | `maxPostsPerNewsletter` | 20 | Newest first. `-1` = whole archive. | | `postedAfter` | — | `2026-01-01` or `30 days`, `2 weeks`, `1 year`. | | `includeContent` | `false` | Full text of free posts; preview for paywalled ones. | | `maxResults` | 0 | 0 = unlimited. | ### Output example **Newsletter** ```json { "type": "newsletter", "id": 10845, "name": "Lenny's Newsletter", "url": "https://www.lennysnewsletter.com", "subdomain": "lenny", "customDomain": "www.lennysnewsletter.com", "description": "Deeply researched product, growth, and career advice for product leaders, founders, and ambitious builders.", "language": "en", "subscriberCount": 1200000, "subscriberCountText": "Millions of subscribers", "paidSubscribersText": "Tens of thousands of paid subscribers", "isPaid": true, "monthlyPrice": 20, "annualPrice": 200, "currency": "USD", "leaderboardCategory": "Business", "leaderboardRank": 1, "leaderboardType": "paid", "hasPodcast": true, "rssFeedUrl": "https://www.lennysnewsletter.com/feed", "authorName": "Lenny Rachitsky", "authorHandle": "lenny", "authorProfileUrl": "https://substack.com/@lenny", "authorFollowerCount": 1384322, "authorLinks": { "twitter": "https://x.com/lennysan", "youtube": "https://www.youtube.com/c/LennysPodcast", "website": "https://www.lennyrachitsky.com/" }, "stripeCountry": "US", "firstPostDate": "2019-06-14T15:00:48.833Z", "source": "url" } ``` **Post** (with `includeContent: true`) ```json { "type": "post", "id": 215243350, "title": "The Specter Of Neuralese", "url": "https://www.astralcodexten.com/p/the-specter-of-neuralese", "postType": "newsletter", "publishedAt": "2026-09-24T22:12:39.073Z", "audience": "everyone", "isPaywalled": false, "wordCount": 3037, "reactionCount": 359, "commentCount": 165, "restackCount": 28, "tags": [], "authors": [{ "id": 12009663, "name": "Scott Alexander", "handle": "astralcodexten" }], "newsletterName": "Astral Codex Ten", "newsletterUrl": "https://www.astralcodexten.com", "contentHtml": "

I. At Long Last, We Have Created Neuralese Recurrence…

…", "contentText": "I. At Long Last, We Have Created Neuralese Recurrence…\nIn the public debates after the release of AI 2027…", "isContentComplete": true } ``` The dataset has two ready-made views in the Apify Console: **Newsletters** and **Posts**. Filter by `type` when exporting via API, e.g. `.../items?format=csv&fields=name,url,subscriberCount,authorLinks`. ### Use with AI agents (MCP), LangChain, RAG & the API This actor is a ready-made **tool for AI agents**: an LLM can call it, wait for the run and read clean JSON back. **MCP server (Claude Desktop, Claude Code, Cursor, VS Code, any MCP client)** — add Apify's hosted MCP server with this actor enabled as a tool: ```json { "mcpServers": { "apify": { "url": "https://mcp.apify.com/?tools=murch1k/substack-scraper", "headers": { "Authorization": "Bearer " } } } } ``` **LangChain / LangGraph** (`pip install langchain-apify`, set `APIFY_API_TOKEN`): ```python from langchain_apify import ApifyActorsTool, ApifyWrapper from langchain_core.documents import Document ## 1) give the actor to an agent as a tool tool = ApifyActorsTool("murch1k/substack-scraper") ## 2) or load results as Documents for a RAG index loader = ApifyWrapper().call_actor( actor_id="murch1k/substack-scraper", run_input={"startUrls": ["https://www.lennysnewsletter.com", "https://substack.com/@noahpinion"], "searchQueries": ["marketing"], "categories": ["technology"], "leaderboardType": "paid", "maxNewslettersPerSource": 25, "includeNewsletterDetails": True, "enrichAuthors": True, "scrapePosts": True, "maxPostsPerNewsletter": 10, "postedAfter": "30 days", "includeContent": False}, dataset_mapping_function=lambda item: Document(page_content=item["contentText"], metadata={"url": item.get("url")}), ) docs = loader.load() ``` **REST API — one synchronous call, dataset items in the response:** ```bash curl -X POST "https://api.apify.com/v2/acts/murch1k~substack-scraper/run-sync-get-dataset-items?token=" \ -H "Content-Type: application/json" \ -d '{"startUrls": ["https://www.lennysnewsletter.com", "https://substack.com/@noahpinion"], "searchQueries": ["marketing"], "categories": ["technology"], "leaderboardType": "paid", "maxNewslettersPerSource": 25, "includeNewsletterDetails": true, "enrichAuthors": true, "scrapePosts": true, "maxPostsPerNewsletter": 10, "postedAfter": "30 days", "includeContent": false}' ``` Also works with the Apify Python/JS clients, **Make, Zapier, n8n**, Google Sheets, webhooks and schedules. Export as JSON, CSV, Excel, XML or RSS. **RAG tip:** With `includeContent: true` every post row carries the full article text (`contentText`) — embed it per post and keep `newsletterName`, `title`, `publishedAt` and `url` as metadata for cited answers. ### Speed The actor reads Substack's own JSON endpoints over plain HTTP — no headless browser — so it is fast and cheap: - \~700 records (120 newsletters + 590 posts) in **under a minute** on 256–512 MB memory. - A full 180-post archive in ~4 seconds. - When Substack rate-limits, the request is retried immediately through a new proxy IP — no waiting, no lost records. ### FAQ **Do I need a Substack account or cookies?** No. Only publicly available data is collected. **Can it get paywalled articles?** No. For paid posts you get all metadata plus the public preview text; `isContentComplete` tells you whether the text is the full article. **Does it work with custom domains?** Yes — any newsletter hosted on Substack, whatever its domain. **Are subscriber counts exact?** They are the numbers Substack shows publicly (e.g. "1,200,000" or "95K+"). Substack hides counts for some newsletters; those get `null` plus the textual tier in `subscriberCountText` when available. **Do I need a proxy?** The actor uses Apify datacenter proxy by default (included in every Apify plan). Substack rate-limits cloud IPs, and the proxy lets the actor switch to a fresh IP immediately instead of waiting, which makes runs several times faster. You can turn it off for small runs. **Email addresses?** Substack does not publish writer emails; the actor returns `supportEmail` when a newsletter sets one, plus author website and social links, which usually lead to contact details. ### Legal This actor only extracts publicly available data. Make sure your use complies with Substack's Terms of Use and applicable data-protection law (e.g. GDPR) — especially when processing personal data such as author names and social profiles. ### Support Found a bug or need an extra field? Open an issue on the **Issues** tab — feature requests are usually shipped within days. # Actor input Schema ## `startUrls` (type: `array`): Any mix of newsletter URLs (https://name.substack.com or a custom domain like https://www.lennysnewsletter.com), post URLs (.../p/post-slug), author profiles (https://substack.com/@handle) or a bare subdomain like astralcodexten. ## `searchQueries` (type: `array`): Searches Substack writers (e.g. marketing, AI startups) and scrapes the newsletter of each match. Up to Max newsletters per source per keyword. ## `categories` (type: `array`): Scrape the top newsletters of Substack categories. Slugs: technology, business, finance, culture, us-politics, world-politics, news, science, health, crypto, education, design, food, travel, sports, music, art, history, philosophy, fiction, literature, parenting, faith, climate, humor, comics, fashionandbeauty, film-and-tv, games, home-garden, international, health-politics. ## `leaderboardType` (type: `string`): Bestsellers = top paid newsletters (max 25 per category). Rising / all = the broader list of up to ~525 newsletters per category. ## `maxNewslettersPerSource` (type: `integer`): Limit of newsletters taken from each search keyword or category. ## `includeNewsletterDetails` (type: `boolean`): One record per newsletter: subscriber count, paid plan prices, leaderboard rank, podcast/RSS feed, author name, bio and handle. ## `enrichAuthors` (type: `boolean`): Adds the author's X/Twitter, LinkedIn, YouTube, website and other links plus follower count (one extra request per newsletter). ## `scrapePosts` (type: `boolean`): Scrape posts of every newsletter (title, URL, date, likes, comments, restacks, word count, paywall status, tags, authors). ## `maxPostsPerNewsletter` (type: `integer`): Newest posts first. 0 = no posts, -1 = the whole archive. ## `postedAfter` (type: `string`): Date like 2026-01-01 or a relative period like 30 days, 2 weeks, 1 year. Empty = no date filter. ## `includeContent` (type: `boolean`): Adds contentHtml and contentText to each post. Free posts get the full text; paywalled posts get the public preview. One extra request per free post. ## `maxResults` (type: `integer`): Stop after this many records in total (newsletters + posts). 0 = no limit. ## `maxConcurrency` (type: `integer`): Parallel requests. Substack's newsletter APIs are fast; requests to substack.com itself (search, leaderboards, profiles) are automatically paced. ## `proxyConfiguration` (type: `object`): Substack rate-limits cloud IPs. The default Apify datacenter proxy lets the actor switch IP instantly when that happens, which makes runs several times faster. Turn it off only for small runs. ## Actor input object example ```json { "startUrls": [ "https://www.lennysnewsletter.com" ], "leaderboardType": "paid", "maxNewslettersPerSource": 25, "includeNewsletterDetails": true, "enrichAuthors": true, "scrapePosts": true, "maxPostsPerNewsletter": 20, "includeContent": false, "maxResults": 0, "maxConcurrency": 10, "proxyConfiguration": { "useApifyProxy": true } } ``` # Actor output Schema ## `newsletters` (type: `string`): No description ## `posts` (type: `string`): No description ## `runStats` (type: `string`): No description # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "startUrls": [ "https://www.lennysnewsletter.com" ], "proxyConfiguration": { "useApifyProxy": true } }; // Run the Actor and wait for it to finish const run = await client.actor("murch1k/substack-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "startUrls": ["https://www.lennysnewsletter.com"], "proxyConfiguration": { "useApifyProxy": True }, } # Run the Actor and wait for it to finish run = client.actor("murch1k/substack-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}") for item in client.dataset(run.default_dataset_id).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "startUrls": [ "https://www.lennysnewsletter.com" ], "proxyConfiguration": { "useApifyProxy": true } }' | apify call murch1k/substack-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "type": "http", "url": "https://mcp.apify.com/?tools=fetch-actor-details,murch1k/substack-scraper" } } } ``` The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer ` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations). ## OpenAPI specification Download the OpenAPI definition: https://api.apify.com/v2/actors/iH0QCKUoCfvNOCZvJ/builds/XFHCkofAqasfWjMIT/openapi.json