Substack Scraper 📰

Scrape posts from any Substack publication. Get titles, authors, dates, tags, reactions, comments, and more from any newsletter.

🚀 Why use this scraper?

• Universal Compatibility: Works with any Substack publication—both custom domains and substack.com subdomains.
• Complete Metadata: Get all post details including titles, authors, tags, reactions, comments, and restacks.
• Fast & Lightweight: Uses Substack's API directly, no browser overhead.
• Cost Effective: $10/month for unlimited usage.

📥 Input

Publication URL (required)

• Custom domain: e.g., https://www.lennysnewsletter.com
• Substack subdomain: e.g., https://astralcodexten.substack.com

Max Posts (optional, default: 100)

• Maximum number of posts to collect. Set to 0 for unlimited.

📤 Output

Each post includes comprehensive metadata:

{
  "post_id": 180650333,
  "post_title": "How to spot a top 1% startup early",
  "subtitle": "Three key lessons from people who picked multiple iconic companies",
  "slug": "how-to-spot-a-top-1-startup-early",
  "post_date": "2025-12-09T13:30:55.231Z",
  "canonical_url": "https://www.lennysnewsletter.com/p/how-to-spot-a-top-1-startup-early",
  "author_name": "Lenny Rachitsky",
  "author_handle": "lenny",
  "cover_image": "https://substack-post-media.s3.amazonaws.com/...",
  "post_description": "Three key lessons from employees who picked multiple iconic companies",
  "wordcount": 2510,
  "tags": ["Career", "Investing"],
  "audience": "only_paid",
  "reactions": {"❤": 241},
  "reaction_count": 241,
  "comment_count": 7,
  "restacks": 7,
  "publication_name": "Lenny's Newsletter",
  "publication_url": "https://www.lennysnewsletter.com"
}

🎯 Example Usage

Scrape a newsletter with custom domain:

{
  "startUrl": "https://www.lennysnewsletter.com",
  "maxPosts": 50
}

Scrape a substack.com subdomain:

{
  "startUrl": "https://astralcodexten.substack.com",
  "maxPosts": 100
}

Scrape all posts (unlimited):

{
  "startUrl": "https://www.notboring.co",
  "maxPosts": 0
}

💰 Pricing

$10/month flat fee - Unlimited runs, unlimited posts.

❓ FAQ

How do I scrape posts from a Substack newsletter?

Provide the publication URL (either a custom domain like lennysnewsletter.com or a subdomain like newsletter.substack.com) and set your desired post limit. The scraper handles pagination automatically.

What data does this Substack scraper extract?

The scraper extracts 18 fields per post: post ID, title, subtitle, slug, publication date, canonical URL, author name, author handle, cover image URL, description, word count, tags, audience type, emoji reactions, reaction count, comment count, restack count, and publication info.

Does this scraper work with paid Substack newsletters?

Yes. The scraper fetches metadata for all posts including paid-only content. Posts behind a paywall are marked with audience: "only_paid". The scraper extracts metadata, not full article content.

Can I scrape the full article body and HTML content from Substack?

The current version extracts post metadata only (title, description, etc.), not the full article HTML. Contact the developer if you need full content extraction.

What Substack URL formats are supported?

Both formats work: custom domains (e.g., www.lennysnewsletter.com, newsletter.pragmaticengineer.com) and Substack subdomains (e.g., astralcodexten.substack.com).

How many Substack posts can I scrape?

Unlimited. Set maxPosts to 0 for no limit. The scraper paginates through the entire archive automatically.

Does this work with non-Substack sites?

No. The scraper uses Substack's internal API. Non-Substack sites will be detected and the scraper exits gracefully with a message.

What if a Substack publication returns no posts?

This typically means: (1) the URL is not a valid Substack publication, (2) the publication has no public posts, or (3) the publication is very new/inactive.

Is this scraper API-based or browser-based?

API-based. It calls Substack's archive API directly without requiring a browser, making it fast and resource-efficient.

How do I get Substack post engagement metrics like likes and comments?

Each post includes reaction_count (total likes), reactions (breakdown by emoji), comment_count, and restacks. These are real-time values from the archive API.

Can I filter Substack posts by date or tag?

The scraper returns posts sorted by newest first. Post-processing filtering by date or tags can be done on the exported dataset.

What's the difference between audience "everyone" and "only_paid"?

everyone means the post is free for all readers. only_paid means the full post is only available to paying subscribers.

📞 Feedback & Support

I'm actively maintaining this actor. If you find a Substack that isn't working or need additional features, please reach out!

📧 stanvanrooy6@gmail.com

---

Built with ❤️ for the Apify community