Substack Scraper: Posts, Comments & Newsletter Search

Scrape Substack posts from any newsletter: publish date, paywall status, reactions, comments and word count — plus the full text of free posts.

Works with substack.com subdomains and custom domains alike; just give the name.

How to use

1. Sign up for Apify — the free plan's monthly credit is enough to try this Actor.
2. Open Substack Scraper, fill in the input (the form is pre-filled with a working example) and click Start.
3. Watch results stream into the dataset table while the run is going.
4. Export as JSON, CSV, Excel, XML, HTML or RSS — or pull the results from your own code with the Apify API.

What can you do with Substack newsletters data?

• Competitive research — how often a newsletter publishes, how long the posts are, what gets reactions.
• Content research — pull the full text of free posts for analysis or summarising.
• Track a list of newsletters — onlyNewPosts on a schedule gives you only what's new.
• Find the paywall line — isFree and audience show exactly what is gated.

How to scrape Substack newsletters

Field	What it does
publications	"lennysnewsletter", "astralcodexten", or a full URL
maxPostsPerPublication	Newest first
includePostText	Full text of free posts
onlyFreePosts	Drop subscriber-only posts
publishedAfter, keywords	Date and title/subtitle filters
onlyNewPosts	Monitoring mode

Input reference

Every option at a glance. A minimal input that works as-is:

{
  "publications": [
    "lennysnewsletter"
  ],
  "maxPublicationsPerSearch": 5,
  "maxPostsPerPublication": 50,
  "maxCommentsPerPost": 50,
  "maxResults": 1000
}

All fields:

• publications (Substack publications) — Names or URLs — “lennysnewsletter”, “platformer.news”, “https://stratechery.com”. Custom domains work too. Optional when you use search queries.
• searchQueries (Search Substack for newsletters) — Find newsletters by topic across all of Substack, e.g. “AI startups”, “personal finance”. Their posts are scraped like the publications above (largest newsletters first).
• maxPublicationsPerSearch (Newsletters per search) — How many newsletters to take from each search query. Default 5.
• maxPostsPerPublication (Max posts per publication) — Newest first. Default 50.
• includePostText (Include the full text) — Fetches the article body for free posts. Paywalled posts return metadata only. Default false.
• includeComments (Include comments) — Also export the comments (and replies) of each post, one row per comment with author, date and likes. Each comment is one result. Default false.
• maxCommentsPerPost (Max comments per post) — Newest comments first. Default 50.
• onlyFreePosts (Only free posts) — Drop subscriber-only posts. Default false.
• publishedAfter (Published after) — YYYY-MM-DD. Paging stops once posts get older.
• keywords (Only posts containing) — Only posts matching these words. Uses Substack’s own archive search, so it matches post text too, not just titles.
• onlyNewPosts (Only new posts) — Monitoring mode. Returns only posts published since the last run. Default false.
• maxResults (Max results) — Total limit. Default 1000.
• proxy (Proxy) — Apify Proxy is used by default — Substack blocks direct datacenter requests.

What the Substack newsletters output looks like

{
  "publication": "www.lennysnewsletter.com",
  "publicationName": null,
  "postId": "214360366",
  "title": "60+ new creative growth ideas",
  "subtitle": "How to stand out when everyone is running the same playbook",
  "url": "https://www.lennysnewsletter.com/p/60-creative-growth-ideas",
  "slug": "60-creative-growth-ideas",
  "publishedAt": "2026-09-15T12:45:28.445Z",
  "type": "newsletter",
  "audience": "only_paid",
  "isFree": false,
  "isPaywalled": true,
  "wordCount": 5674,
  "reactionCount": 383,
  "commentCount": 6,
  "coverImage": "https://substackcdn.com/image/fetch/$s_!NVVg!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F20cbbe62-5c82-4d4f-a316-de9b885e56f2_1456x970.png",
  "authors": [],
  "description": "How to stand out when everyone is running the same playbook",
  "scrapedAt": "2026-09-21T07:25:01.710Z"
}

How fast is it and how much does it cost to scrape Substack newsletters?

45 posts from 3 publications took 29 seconds, including resolving each name to its real domain.

How to find Substack newsletters on a topic

Put topics in searchQueries (for example AI startups or personal finance). The Actor searches all of Substack, takes the largest matching newsletters (maxPublicationsPerSearch, 5 by default) and scrapes their posts like any publication you list yourself.

How to scrape Substack comments

Turn on includeComments. Each post is followed by its comments and replies, one row per comment (recordType: "comment") with author, date, likes, reply count and the parent comment, newest first, up to maxCommentsPerPost.

Keyword search inside a newsletter

keywords uses Substack's own archive search, so it finds posts that mention a word in the body, not only in the title.

Limits you should know about

• Paywalled posts return every metadata field, but no body text — that is what the paywall is.
• A publication that has moved off Substack is reported with a warning and not charged.

Pricing — what you pay per post

This Actor uses pay-per-event pricing: no monthly rental, you pay for the results you get. Prices per 1,000 results by Apify plan:

Event	Free plan	Bronze	Silver	Gold+
post — One Substack post with reactions, comments and word count	$1.20	$1.08	$1.02	$0.96

There is also a small start fee of $0.00005 per GB of memory ($0.00005 per run at the default 1 GB). Platform compute is included in the price — you are not billed separately for run time. Set a maximum total charge on the run if you want a hard cap, and use the input limits to control how much is fetched. The current prices are always on the Pricing tab.

Tips

• Start with a small maxResults (say 20) to check the output, then raise it — you only pay per result.
• For ongoing tracking, schedule the Actor and turn on onlyNewPosts so each run returns only what is new.
• Keep the default proxy setting — it is only used as a fallback when a direct request is blocked.
• Rows the Actor cannot read are logged and skipped, never charged — check the run log if a count looks low.

Automate & schedule

Run this Actor on autopilot and pull results into your own stack:

• Apify API — start runs, fetch datasets and manage schedules over REST.
• apify-client for JavaScript and apify-client for Python — official SDKs.
• Schedules — run it on a cron to keep your data fresh.
• Webhooks — trigger downstream actions the moment a run finishes.

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'MY_APIFY_TOKEN' });
const run = await client.actor('tindacloud/substack-newsletter-scraper').call({
    "publications": [
        "lennysnewsletter"
    ],
    "maxPublicationsPerSearch": 5,
    "maxPostsPerPublication": 50,
    "maxCommentsPerPost": 50,
    "maxResults": 1000
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(`Got ${items.length} records`);

Python

from apify_client import ApifyClient

client = ApifyClient("MY_APIFY_TOKEN")
run = client.actor("tindacloud/substack-newsletter-scraper").call(run_input={
    "publications": [
        "lennysnewsletter"
    ],
    "maxPublicationsPerSearch": 5,
    "maxPostsPerPublication": 50,
    "maxCommentsPerPost": 50,
    "maxResults": 1000
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item)

cURL — run and get the results in one request:

curl -X POST "https://api.apify.com/v2/acts/tindacloud~substack-newsletter-scraper/run-sync-get-dataset-items?token=MY_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"publications": ["lennysnewsletter"], "maxPublicationsPerSearch": 5, "maxPostsPerPublication": 50, "maxCommentsPerPost": 50, "maxResults": 1000}'

Integrate with any app

Connect the dataset to thousands of apps — no code required:

• Make — multi-step automation scenarios.
• Zapier — push new records straight into your CRM or sheet.
• n8n — self-hosted workflows.
• Slack — get notified when a run finishes or finds something new.
• Google Drive / Sheets — auto-export every run to a spreadsheet.
• Airbyte — pipe results into your data warehouse.
• GitHub — trigger runs from commits or releases.
• Webhooks — call your own endpoint when a run succeeds or fails.

Use with AI assistants (MCP)

The output is clean, LLM-ready JSON. Call this Actor from Claude, ChatGPT, Cursor or any agent framework through the Apify MCP server — ask your assistant to "list the posts of these three Substack newsletters from this year with likes, comments and paywall status" and let it run this Actor for you.

Agent-ready: autonomous payments (x402 & Skyfire)

This Actor is agent-ready — AI agents can discover it, run it and pay for it on their own. It uses pay-per-event pricing and limited permissions, so it qualifies for Apify's agentic payments:

• x402 — an open, HTTP-native payment protocol. Agents pay in USDC through the Apify MCP server — no Apify account needed.
• Skyfire — agent-to-service payments with pre-funded Skyfire tokens.

Building an AI agent, MCP tool or data pipeline? This Actor is ready to plug in and pay as it goes.

Ready-made examples

Open one, change the fields, press Start — nothing to configure:

• Scrape Substack posts

Frequently asked questions about scraping Substack newsletters

Do I need an API key or a login?

No. Paste your input, press Start. There is nothing to connect and no account on the target site to create.

How do I export the results?

Every run produces a dataset you can download as JSON, CSV, Excel, XML or HTML, or read through the Apify API from your own code. You can also connect it to Make, Zapier or n8n.

Can I run this on a schedule?

Yes. Apify schedules run it every hour, day or week without you being there. With the monitoring option on, you only get posts published since the last run.

Can I call it from Python or JavaScript?

Yes — the Apify API runs any Actor and returns the dataset, and the official Python and JavaScript clients wrap it in a few lines. AI agents can call it too, through Apify's MCP server.

What does it cost?

You pay per result, not per hour. Filters are applied before anything is charged, so narrowing the input directly lowers the bill. Pages that cannot be read are reported in the log and never charged.

Is this data public?

Everything collected here is what Substack shows to anyone without logging in. Check the target site's terms and the rules that apply to you before using the data commercially.

Related scrapers

• Website to Markdown for AI & RAG — turn any site into clean Markdown
• Google News Scraper — news articles with full text
• Telegram Channel Scraper — public channel posts with view counts

Disclaimer

This Actor is an independent tool and is not affiliated with, endorsed by, or sponsored by Substack Inc.. All trademarks mentioned are the property of their respective owners. "Substack" is named here only in a descriptive sense — to identify the publicly available website this Actor reads. It collects only publicly available information that anyone can see without logging in; you are responsible for making sure your use of the data complies with applicable laws, regulations and the terms of the source website.