Substack Scraper - Posts, Newsletters & Comments

Scrape Substack posts, newsletters, authors, publication details, subscriber data, engagement metrics, public article content, and nested comments. Start from publication URLs, custom domains, direct post URLs, or search keywords.

The Actor uses lightweight public web endpoints instead of a browser, so routine runs are fast and use only 256 MB of memory. Results are available as JSON, CSV, Excel, XML, HTML, and RSS through the Apify Dataset API.

Features

• Discover Substack publications by keyword, topic, author, or niche
• Scrape complete publication archives or individual /p/ URLs
• Extract headlines, descriptions, dates, authors, tags, images, language, and word counts
• Collect reactions, comments, replies, restacks, and free or paid access status
• Return public article content as plain text, Markdown, and HTML
• Preserve nested public comment threads with author and reaction details
• Include publication name, description, logos, payment status, and visible subscriber counts
• Filter by date, post type, access, reactions, comments, and word count
• Support standard substack.com publications and Substack custom domains
• Produce a STATS record with per-publication results and run duration

Use Cases

• Competitor monitoring: track publishing frequency, topics, authors, and engagement
• Newsletter discovery: find publications in a market before building a prospect list
• Content research: compare headlines, tags, formats, and audience response
• AI and RAG datasets: export clean Markdown or text with source metadata
• Media monitoring: schedule recurring snapshots of selected newsletters
• Audience research: analyze public conversations in comments and replies
• Lead generation: identify relevant publications, authors, and custom domains by niche

Input

Provide at least one publication URL or search keyword.

Field	Type	Default	Description
substackUrls	array	optional	Publication URLs, custom domains, or direct post URLs
searchQueries	array	optional	Keywords used to discover relevant publications
maxSearchResultsPerQuery	integer	10	Publications discovered per keyword
maxPostsPerSubstack	integer	50	Matching posts per publication; 0 means the full archive
includeContent	boolean	true	Return public HTML, Markdown, and plain text
includeComments	boolean	false	Return public comments and nested replies
maxCommentsPerPost	integer	20	Maximum comments and replies per post
includePublicationInfo	boolean	true	Add publication and visible subscriber metadata
postType	string	all	all, newsletter, podcast, or thread
startDate	date	optional	Include posts published on or after this date
endDate	date	optional	Include posts published on or before this date
onlyFree	boolean	false	Exclude posts marked for paid subscribers
minReactions	integer	0	Minimum reaction count
minComments	integer	0	Minimum comment count
minWordCount	integer	0	Minimum word count
postConcurrency	integer	10	Parallel post requests, from 1 to 25

Scrape publications and comments

{
  "substackUrls": [
    "https://www.lennysnewsletter.com",
    "https://platformer.news"
  ],
  "maxPostsPerSubstack": 20,
  "includeContent": true,
  "includeComments": true,
  "maxCommentsPerPost": 30,
  "startDate": "2026-01-01",
  "onlyFree": true,
  "postConcurrency": 10
}

Discover newsletters by keyword

{
  "searchQueries": ["artificial intelligence", "product management"],
  "maxSearchResultsPerQuery": 5,
  "maxPostsPerSubstack": 10,
  "includeContent": false,
  "minReactions": 10
}

Output

Each Dataset item represents one post. Existing fields such as headline, date, likes, comments, and article_text remain available, with richer fields added.

{
  "record_type": "post",
  "post_id": 208730073,
  "publication_id": 10845,
  "substack_url": "https://www.lennysnewsletter.com",
  "post_url": "https://www.lennysnewsletter.com/p/example-post",
  "slug": "example-post",
  "headline": "Example newsletter post",
  "subheading": "A public subtitle",
  "post_type": "newsletter",
  "published_at": "2026-08-18T15:40:06.921Z",
  "date": "Aug 18, 2026",
  "free_or_paid": "Free",
  "is_paywalled": false,
  "likes": 346,
  "comments": 17,
  "child_comments": 14,
  "restacks": 9,
  "word_count": 933,
  "language": "en",
  "tags": ["Career"],
  "cover_image_url": "https://...",
  "author_name": "Example Author",
  "author_handle": "example",
  "author_url": "https://substack.com/@example",
  "article_text": "Plain text content...",
  "article_markdown": "## Markdown content...",
  "article_html": "<div>HTML content...</div>",
  "content_type": "full",
  "comment_threads": [
    {
      "id": 123,
      "body": "Useful analysis.",
      "author_name": "Reader",
      "reaction_count": 4,
      "is_author": false,
      "replies": []
    }
  ],
  "comments_collected": 1,
  "publication": {
    "name": "Example Newsletter",
    "url": "https://example.substack.com",
    "hero_text": "Publication description",
    "subscriber_count": 12000,
    "subscriber_count_label": "Over 12,000 subscribers",
    "payments_enabled": true,
    "logo_url": "https://..."
  },
  "search_query": "product management",
  "scraped_at": "2026-08-23T10:00:00Z"
}

content_type is full, preview_only, metadata_only, or failed. The Actor never bypasses a paywall; paid posts contain only content Substack exposes publicly.

How to Use

1. Open the Actor and select Try for free.
2. Add publication or post URLs, search keywords, or both.
3. Set the number of posts and optional filters.
4. Enable comment extraction only when comment bodies are needed.
5. Start the run and open the Output tab.
6. Export the overview or content view in the format required by your workflow.

For a fast competitor snapshot, disable includeContent. For AI or RAG exports, keep it enabled and use article_markdown or article_text.

API Examples

Python

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("scraper_guru/substack-scraper").call(run_input={
    "substackUrls": ["https://www.lennysnewsletter.com"],
    "maxPostsPerSubstack": 25,
    "includeContent": True,
    "includeComments": True,
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["headline"], item["post_url"])

JavaScript

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_APIFY_TOKEN' });
const run = await client.actor('scraper_guru/substack-scraper').call({
  searchQueries: ['AI newsletters'],
  maxSearchResultsPerQuery: 5,
  maxPostsPerSubstack: 10,
  includeContent: false,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Pricing

This Actor uses pay-per-event pricing, so platform usage is included.

Event	Price
Actor start	$0.00035 per run
Saved post on the Free plan	$0.0005
1,000 saved posts on the Free plan	$0.50
1,000 saved posts with the highest plan discount	$0.35

Only successfully saved Dataset items are charged as post results. Apify shows the exact plan-adjusted price before each run.

Reliability and Limits

• The Actor collects public data and does not log in or bypass paid-content restrictions.
• Visible subscriber counts are returned when Substack exposes them on the publication page.
• Keyword search discovers publications through Substack profiles, then applies your post settings to each publication.
• Very large full-archive runs take longer and produce more results. Test with 5 to 20 posts first.
• Comment extraction adds one request per post and can increase run duration.
• Substack can change its public web endpoints. Report reproducible extraction problems in the Actor's Issues tab.

FAQ

Can it scrape a Substack custom domain?

Yes. Use the custom publication URL, such as https://platformer.news, or a standard https://name.substack.com URL.

Can it scrape one specific post?

Yes. Add a direct URL containing /p/. The Actor processes that post instead of the full publication archive.

Can it extract full paid posts?

No. It returns only the preview and metadata available to the public. content_type is set to preview_only for paid posts.

How does keyword discovery work?

The Actor searches public Substack profiles and publications matching each keyword, deduplicates the publications, and then scrapes their newest matching posts.

How do I get only recent, popular posts?

Set startDate, minReactions, and minComments. Filters are applied before full content and comments are requested, reducing unnecessary work.

Why is a subscriber count missing?

Some publications do not display a public count. The field remains empty instead of estimating a value.

Which content format should I use for AI pipelines?

Use article_markdown when headings and links matter, or article_text for embeddings and simple text analysis.

Does it support scheduled monitoring?

Yes. Create an Apify Task from a working input and schedule it. Connect the Dataset to a webhook, Make, n8n, Zapier, Google Sheets, or your own API.