Substack Newsletter Scraper 📰

Extract complete data from Substack newsletters including posts, authors, engagement metrics, and full article text.

What does this Actor do? 🎯

This actor scrapes Substack newsletters and extracts 13 data points per post:

Content Data

• Post headline & subheading
• Full article text (or preview for paid posts)
• Post URL

Author Information

• Author name
• Author profile URL

Publishing Data

• Publication date
• Free vs Paid status

Engagement Metrics

• Number of likes ❤️
• Number of comments 💬
• Number of restacks 🔄

Perfect for 💡

• Content Researchers - Analyze trends across newsletters
• Competitive Analysis - Track what competitors publish
• Data Scientists - Build training datasets
• Writers - Research popular topics
• AI Developers - Collect training data

Input

{
  "substackUrls": [
    "https://tedhope.substack.com",
    "https://theankler.com"
  ],
  "scrapingMethod": "sitemap",
  "maxPostsPerSubstack": 100,
  "batchSize": 20
}

Field	Type	Required	Description
substackUrls	Array	Yes	List of Substack URLs
scrapingMethod	String	No	"sitemap" or "archive"
maxPostsPerSubstack	Number	No	Limit posts (0 = unlimited)
batchSize	Number	No	Substacks per batch

Output

{
  "substack_url": "https://example.substack.com",
  "post_url": "https://example.substack.com/p/post-title",
  "headline": "Amazing Post Title",
  "subheading": "Subtitle here",
  "author_name": "John Doe",
  "author_url": "https://substack.com/@johndoe",
  "date": "December 10, 2024",
  "free_or_paid": "Free",
  "likes": 156,
  "comments": 23,
  "restacks": 12,
  "article_text": "Full article content...",
  "content_type": "full"
}

Features ✨

• Two scraping methods - Sitemap (fast) or Archive (fallback)
• Complete metadata - 13 fields per post
• Engagement metrics - Likes, comments, restacks
• Smart extraction - Handles paywalled content
• Structured output - Clean JSON/CSV format

Pricing 💰

Memory is automatically adjusted based on your input — no manual configuration needed.

Scale	Approximate Cost	Time
10 Substacks	$0.01-0.05	2-5 min
100 Substacks	$0.50-1.00	30-60 min
1,000 Substacks	$5-10	5-10 hours

Start with Apify's free tier - includes $5 monthly credit!

Tips 💪

1. Test first - Try 2-3 Substacks initially
2. Use sitemap - Faster and more reliable
3. Set limits - Use maxPostsPerSubstack for large newsletters
4. Schedule runs - Set up weekly/monthly automation

FAQ ❓

Q: Can I scrape paywalled content? A: You'll get previews for paid posts, not full content.

Q: How long does it take? A: ~10-30 seconds per Substack with 100 posts.

Q: What if sitemap doesn't work? A: Use "archive" method as fallback.

Q: How much memory does it use? A: Memory is allocated dynamically based on how many URLs you provide — you don't need to configure anything manually. Typical runs use 512MB–1GB.

---

Changelog 📋

v1.0.4 — April 2, 2026 🔧

Bug Fixes & Performance Improvements

• Fixed: Incomplete scraping results — The scraper was mishandling Substack's nested sitemap indexes, causing runs to stop prematurely after just a few posts. It now correctly parses all sitemap levels and captures 100% of available posts.
• Fixed: Memory misconfiguration — Default memory was incorrectly set to 16GB, making runs unnecessarily expensive. Memory is now dynamically allocated based on your input (scales with number of URLs provided), with a maximum cap of 2GB. Typical single-URL runs now use ~1GB.
• Improved: Cost efficiency — For a standard single-newsletter scrape, costs are now reduced by ~94% compared to previous versions.

---

v1.0.0 — Initial Release 🚀

• Full post scraping via Sitemap and Archive methods
• 13 data fields per post including engagement metrics
• Batch processing support
• Paid/free post detection

---

Support 📧

• 📖 Apify Documentation
• 💬 Contact Support

About 👨‍💻

Built by MUSTAPHA LIAICHI - Automation & Web Scraping Specialist

---

Happy Scraping! 🚀