Substack Scraper — Posts, Full Text & Newsletter Search

Scrape any Substack newsletter: get every archive post with title, date, authors, likes, comments, paywall status, word count and (optionally) the full text as clean Markdown, ready for LLMs and RAG pipelines. You can also search all of Substack for posts or newsletters by keyword, with subscriber counts. No login and no browser, so runs are fast and cheap.

What can you do with it?

• Feed AI agents and LLM / RAG pipelines with newsletter content as clean Markdown. You can also call it from AI agents through the Apify MCP server.
• Monitor newsletters: track new posts from competitors, analysts or industry writers on a schedule (for example with postedAfter: "7 days").
• Content research: find the most-liked posts of any newsletter (sort: "top"), or what Substack writers say about a topic.
• Newsletter discovery and influencer research: find newsletters in a niche with subscriber counts, categories and author info, for sponsorships, partnerships or market sizing.
• Analytics: likes, comments, restacks, word counts and free/paid mix over time.

What data can you get?

Posts (from a newsletter archive, a post URL or a keyword search):

Field	Description
title, subtitle, description	Headline, subtitle and teaser
url, slug	Absolute post URL
publishedAt, updatedAt	ISO-8601 dates (UTC)
authors, authorHandles	Byline names and Substack handles
reactionCount, commentCount, restackCount	Likes, comments and restacks
audience, isPaywalled	everyone (free), only_paid, founding …
wordcount	Post length in words
coverImage	Cover image URL
tags, sectionName, language	Post tags, newsletter section, language
postType, podcastUrl, podcastDurationSeconds	newsletter / podcast / thread, and audio info
publicationId, publicationName, publicationUrl	The newsletter the post belongs to
contentMarkdown	Full post body as clean Markdown (when includeContent is on)
contentIsPreview	true when the post is paywalled and only the free preview is included
searchQuery, sourceInput, scrapedAt	Where the record came from and when

Newsletters / publications (from a keyword search with searchType: "publications"):

Field	Description
name, url, subdomain, customDomain	Newsletter name and address
description, logoUrl, createdAt	Tagline, logo, launch date
subscriberCount, subscriberCountNumber, subscriberBand	Public subscriber count as Substack shows it (e.g. "25K+", 25000, "Tens of thousands")
paidSubscriptionsEnabled	Whether it offers paid subscriptions
authorName, authorHandle, authorProfileUrl, authorBio, authorFollowerCount	Public writer profile info
bestsellerTier, leaderboardCategory, leaderboardRank	Substack bestseller badge and category leaderboard rank, when present

No email addresses or other private contact data are collected.

How to use

1. Add newsletter URLs to Newsletter or post URLs (https://name.substack.com, a custom domain like https://www.astralcodexten.com, just name, https://substack.com/@handle or a single post URL …/p/slug), and/or add Search keywords.
2. Choose options: Include full text (Markdown), Only posts published after, Only free posts, Archive order (newest or top), and the result limits.
3. Click Start, then download the results as JSON, CSV or Excel, or read them through the API.

Input example

{
  "startUrls": ["https://www.astralcodexten.com", "noahpinion", "https://substack.com/@climatehuman"],
  "searchQueries": ["retrieval augmented generation"],
  "searchType": "posts",
  "includeContent": true,
  "postedAfter": "1 year",
  "maxItems": 200,
  "maxItemsPerInput": 50
}

Output example

A post found by keyword search, with includeContent on (Markdown shortened here):

{
  "type": "post",
  "id": 139244404,
  "title": "A Practitioners Guide to Retrieval Augmented Generation (RAG)",
  "subtitle": "How basic techniques can be used to build powerful applications with LLMs...",
  "url": "https://cameronrwolfe.substack.com/p/a-practitioners-guide-to-retrieval",
  "slug": "a-practitioners-guide-to-retrieval",
  "publishedAt": "2024-02-05T10:24:15Z",
  "postType": "newsletter",
  "audience": "everyone",
  "isPaywalled": false,
  "authors": ["Cameron R. Wolfe, Ph.D."],
  "authorHandles": ["cwolferesearch"],
  "reactionCount": 99,
  "commentCount": 5,
  "restackCount": 10,
  "wordcount": 6574,
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/a38a821b-0915-4907-9bc8-d0aae93b38db_1916x1072.png",
  "tags": ["LLM Fundamentals"],
  "language": "en",
  "publicationId": 1092659,
  "publicationName": "Deep (Learning) Focus",
  "publicationUrl": "https://cameronrwolfe.substack.com",
  "contentMarkdown": "(from [1, 5])\n\nThe recent surge of interest in generative AI has led to …",
  "contentIsPreview": false,
  "searchQuery": "retrieval augmented generation",
  "sourceInput": "retrieval augmented generation",
  "scrapedAt": "2026-10-09T13:50:13Z"
}

A newsletter found by keyword search (searchType: "publications"):

{
  "type": "publication",
  "publicationId": 459118,
  "name": "The Bitcoin Layer",
  "url": "https://thebitcoinlayer.substack.com",
  "subdomain": "thebitcoinlayer",
  "description": "Bitcoin and global macro research firm.",
  "createdAt": "2021-08-25T19:31:30Z",
  "paidSubscriptionsEnabled": true,
  "authorName": "Nik Bhatia",
  "authorHandle": "thebitcoinlayer",
  "authorProfileUrl": "https://substack.com/@thebitcoinlayer",
  "subscriberCount": "25K+",
  "subscriberCountNumber": 25000,
  "subscriberBand": "Tens of thousands",
  "authorFollowerCount": 34627,
  "bestsellerTier": 100,
  "searchQuery": "bitcoin",
  "scrapedAt": "2026-10-09T13:50:24Z"
}

In the Apify Console, the Posts and Newsletters views show each record type as its own table.

Pricing

$1.00 per 1,000 results, with no start fee. You pay only for the results you get, and full text costs nothing extra. A result is one post or one newsletter. Set maxItems to cap the cost of a run. The run also stops by itself when it reaches the maximum cost you set for the run.

Tips

• Whole archive: set maxItems high (e.g. 5,000). Large newsletters with 1,000+ posts take a few minutes, longer with full text on.
• Most popular posts: set sort to top.
• Only recent posts: postedAfter accepts 2025-01-31 or relative values like 30 days, 6 months, 1 year.
• Precise post search: Substack's post search is semantic, so it also returns loosely related posts. Turn on strictKeywordMatch to keep only posts that contain every keyword.

FAQ

Is it legal to scrape Substack? This Actor reads only public data that Substack serves to any visitor without logging in: public post metadata, free post content, and public newsletter and profile info. It does not get around paywalls or log in, and it does not collect emails. You are responsible for how you use the data, including copyright and privacy rules (e.g. GDPR) in your country. Don't republish other writers' content without permission.

Can I get the full text of paid posts? No. For paywalled posts you get all the metadata plus the free preview Substack shows publicly (contentIsPreview: true).

Do I need proxies? Usually not. The Actor uses Substack's public JSON endpoints and retries with backoff when rate-limited. If you run very large jobs and see repeated HTTP 429 warnings, enable Apify Proxy in the input.

Are there limits?

• Keyword search returns what Substack's own search returns. Newsletter search typically yields up to about 50–100 newsletters per keyword, and post search a few hundred posts per keyword.
• Subscriber counts are the rounded public figures Substack displays (e.g. "25K+"). Some writers hide them, and then the field is empty.
• Substack Notes (short social posts) and comments are not scraped.
• A single bad or non-Substack URL is logged and skipped. It doesn't stop the run.

Can I use it with AI agents? Yes. Call the Actor from any MCP-compatible agent through the Apify MCP server, or from LangChain, LlamaIndex or your own code through the Apify API. Turn on includeContent to get LLM-ready Markdown.