Substack Scraper

🚀 Substack Scraper — Publication Discovery & Public Archives

Collect public posts and publication profiles from any Substack newsletter, including those on custom domains like www.lennysnewsletter.com. Search Substack for newsletters on a topic, then export their recent posts. You can also get one profile row per publication, with its subscriber count, paid plans, sections and recent activity.

✨ Features

• 📚 Section-aware archives: many publications keep posts in sections that their main archive never lists. The scraper reads every section, including ones hidden from the navigation, and merges them into one newest-first list. On Lenny's Newsletter, 20 of the 25 newest posts only appear in sections.
• 🔍 Publication discovery: search Substack by keyword, such as technology or climate. Each publication that matches appears only once and lists every query that found it.
• 🏷️ Publication profiles: name, author, language, custom domain and the public subscriber count as Substack shows it ("Over 1,200,000 subscribers"). Profiles also include monthly, yearly and founding-member plans, sections, first post date, and how many recent posts are free or paid.
• 📰 Post metadata: title, subtitle, URL, date, free or paid audience, type (newsletter, podcast, thread), section, authors, tags, word count, reactions, comments, restacks and the public preview.
• 📝 Full text of free posts (optional): plain text and HTML for posts anyone can read. Paid posts are never unlocked; they keep the public preview only.
• 📅 Date filter: return only posts published after a given date.
• 🌐 Custom domains and aliases: name.substack.com, a custom domain and a post link all resolve to the same publication.

🛠️ How It Works

1. Add publications or search terms: paste newsletter addresses (name.substack.com, a custom domain or any post link), or type keywords into Search publications.
2. Choose what to collect: Posts returns one row per post; Publication profiles returns one row per newsletter.
3. Set limits: Maximum results caps the whole run, and Maximum posts per publication takes the newest N posts from each newsletter.
4. Run the scraper: results appear in the dataset. The SUMMARY record in the key-value store shows which sections were read for each publication and why each one stopped (limit, date or end of archive).

💰 Pricing

Pay only for results:

Event	Price
Post	$0.0015 per post ($1.50 per 1,000)
Free post full text	+$0.001 per free post delivered with its full text (only when enabled)
Publication profile	$0.003 per publication
Run start	$0.00005 per run

The default input (2 publications, 50 posts) costs about $0.08.

📊 Sample Output Data

Posts:

[
    {
        "postId": 215710129,
        "title": "How Warp ships 2,000 PRs a month with AI factories | Zach Lloyd (CEO, Warp)",
        "url": "https://www.lennysnewsletter.com/p/how-warp-ships-2000-prs-a-month-with",
        "postDate": "2026-09-21T12:04:11.791Z",
        "audience": "everyone",
        "isFree": true,
        "type": "podcast",
        "section": "How I AI",
        "foundIn": "how-i-ai",
        "authors": ["Claire Vo"],
        "tags": [],
        "wordCount": 454,
        "reactionCount": 3,
        "commentCount": 0,
        "restackCount": 0,
        "podcastDurationSeconds": 2811,
        "publicationId": 10845,
        "publicationName": "Lenny's Newsletter",
        "publicationUrl": "https://www.lennysnewsletter.com",
        "contentStatus": "not_requested"
    },
    {
        "postId": 161510512,
        "title": "Co-Existence and the End of Co-Intelligence",
        "subtitle": "Also: how pitch a book to an AI!",
        "url": "https://www.oneusefulthing.org/p/co-existence-and-the-end-of-co-intelligence",
        "postDate": "2026-06-04T21:13:42.065Z",
        "audience": "everyone",
        "isFree": true,
        "type": "newsletter",
        "section": null,
        "foundIn": "archive",
        "authors": ["Ethan Mollick"],
        "wordCount": 1450,
        "reactionCount": 967,
        "commentCount": 125,
        "restackCount": 81,
        "previewText": "It has been two years since Co-Intelligence, my book about AI, was published…",
        "publicationId": 1180644,
        "publicationName": "One Useful Thing",
        "contentStatus": "not_requested"
    }
]

Publication profile:

{
    "publicationId": 10845,
    "name": "Lenny's Newsletter",
    "url": "https://www.lennysnewsletter.com",
    "substackUrl": "https://lenny.substack.com",
    "customDomain": "www.lennysnewsletter.com",
    "language": "en",
    "authorName": "Lenny Rachitsky",
    "subscriberCountDisplay": "Over 1,200,000 subscribers",
    "subscriberCountLowerBound": 1200000,
    "paidPlans": [
        { "interval": "month", "price": 20, "currency": "USD", "isFoundingMember": false },
        { "interval": "year", "price": 200, "currency": "USD", "isFoundingMember": false },
        { "interval": "year", "price": 400, "currency": "USD", "isFoundingMember": true }
    ],
    "hasPodcast": true,
    "firstPostDate": "2019-06-14T15:00:48.833Z",
    "sections": [
        { "name": "How I AI", "slug": "how-i-ai", "hiddenFromNavigation": false },
        { "name": "Podcast summaries", "slug": "podcast-summaries", "hiddenFromNavigation": true }
    ],
    "latestPostDate": "2026-09-22T12:45:14.998Z",
    "recentPostsSampled": 23,
    "recentFreePosts": 5,
    "recentPaidPosts": 18,
    "matchedQueries": [],
    "inputUrls": ["https://lenny.substack.com"]
}

❓ Good to Know

• Subscriber counts are Substack's rounded public figures, so subscriberCountLowerBound is a floor, not an exact audience. When a publication hides its count, the value is null, not zero.
• recentFreePosts and recentPaidPosts describe the latest page of the main archive, not the whole history.
• Some podcast episodes appear only in a publication's podcast feed and may be missing from its archives.
• Only public data is collected: no logins, no paid content, and no subscriber or commenter lists.

Start exploring Substack publications with the Substack Scraper today! 🚀