Substack Newsletter Content Scraper

Scrape public Substack newsletter posts with headlines, authors, publication dates, engagement metrics, access type, and available article text. The output is structured for content research, competitor tracking, and AI-ready datasets.

Features

• Extracts headline, subheading, author, date, post URL, and newsletter URL
• Collects publicly visible likes, comments, and restacks
• Returns full text for accessible posts and preview text for paywalled posts
• Discovers posts through sitemaps with an archive-page fallback
• Processes multiple newsletters in bounded batches
• Controls post concurrency to reduce timeouts
• Optionally builds an HTML newsletter digest

Use Cases

• Newsletter research: compare publishing frequency, topics, and engagement
• Competitive intelligence: monitor public posts from newsletters in a niche
• Content analysis: identify headlines and formats associated with engagement
• Media monitoring: create scheduled snapshots of selected publications
• AI datasets: collect public article text with source and access metadata

Input

Field	Type	Default	Description
substackUrls	array	required	Substack publication URLs
scrapingMethod	string	sitemap	sitemap or archive discovery
maxPostsPerSubstack	integer	50	Maximum posts per publication; 0 means unlimited
batchSize	integer	5	Publications processed in each batch
postConcurrency	integer	3	Post pages processed in parallel
generateNewsletterDigest	boolean	false	Save an optional HTML digest preview

{
  "substackUrls": ["https://tedhope.substack.com"],
  "scrapingMethod": "sitemap",
  "maxPostsPerSubstack": 50,
  "batchSize": 5,
  "postConcurrency": 3,
  "generateNewsletterDigest": false
}

Output

Each dataset item represents one Substack post:

{
  "substack_url": "https://tedhope.substack.com",
  "post_url": "https://tedhope.substack.com/p/example-post",
  "headline": "Example post",
  "subheading": "A public post subtitle",
  "author_name": "Ted Hope",
  "author_url": "https://substack.com/@example",
  "date": "July 10, 2026",
  "free_or_paid": "Free",
  "likes": 156,
  "comments": 23,
  "restacks": 12,
  "article_text": "Available article text...",
  "content_type": "full"
}

content_type is full, preview_only, or failed. The STATS record reports publication count, post count, limits, method, and concurrency.

How to Use

1. Open the Actor and click Try for free.
2. Add one or more Substack publication URLs.
3. Keep sitemap as the preferred discovery method.
4. Start with maxPostsPerSubstack: 10 for a quick test.
5. Reduce postConcurrency if a large run encounters timeouts.
6. Start the run and export the dataset from the Output tab.

from apify_client import ApifyClient

client = ApifyClient("YOUR_API_TOKEN")
run = client.actor("scraper_guru/substack-scraper").call(run_input={
    "substackUrls": ["https://tedhope.substack.com"],
    "maxPostsPerSubstack": 50,
    "postConcurrency": 3,
})

Pricing

This is a pay-per-event Actor. The base price is $0.0005 per saved post, or $0.50 per 1,000 posts. Apify plan discounts can reduce the per-post price to $0.00035, and any additional platform usage is shown by Apify.

FAQ

Can it extract full paywalled posts?
No. It returns only the public preview unless the full article is publicly accessible.

Which discovery method should I use?
Use sitemap first. Choose archive when a publication does not expose usable sitemap entries.

How do I collect only recent posts?
Use a small maxPostsPerSubstack. Discovered post URLs are processed in source order, which is typically newest first but can vary by publication.

Why can engagement fields be zero?
The source may show no engagement or may not expose a metric on that page layout.

Is scraping Substack legal?
Collect only public content, respect publisher rights and Substack's terms, and comply with copyright and data-protection rules.

For page-layout changes or extraction problems, open the Issues tab on this Actor.