Substack Scraper — Newsletters, Creators, Posts & Leads

Extract data from Substack without login or code. This Substack scraper finds the top newsletters in any category — each with author, subscriber count and contact — and pulls any publication's posts with full content, reactions, comments, restacks and podcast data. Use it to scrape Substack for creator lead lists, competitive research and AI content pipelines.

Export every record to JSON, CSV, Excel, or API.

Built for growth and sponsorship teams building creator lead lists, PR and influencer marketers, media analysts, newsletter operators benchmarking competitors, and AI/ML engineers feeding long-form text into LLMs and vector databases.

---

🔎 Why scrape Substack?

Substack hosts hundreds of thousands of independent newsletters and podcasts, and its per-category leaderboards are effectively a ranked directory of the most influential writers on the internet — complete with author names, subscriber magnitudes, taglines and monetization signals. That is a goldmine for anyone who needs to find, rank and reach creators, or who wants a clean feed of long-form editorial content to analyze.

The problem is that Substack has no public bulk export and no official data API. Doing it by hand means clicking through leaderboards, opening each publication, copying the author and subscriber count, then paging through every post — thousands of clicks for a single niche. Subscriber counts are shown as rounded magnitudes, contact details are buried, and engagement metrics (reactions, comments, restacks) live on individual post pages.

This Substack scraper automates all of it. Point it at a category to build a creator lead list, or at a publication to collect its full post archive with content and engagement — structured, de-duplicated and ready to export.

Common reasons people scrape Substack:

• 🧲 Build creator lead lists for newsletter sponsorships, ad buys and cross-promotion.
• 📊 Run competitor and market research — see who leads a niche and how big they are.
• 🎙️ Power PR and influencer outreach with targeted lists of writers and podcasters.
• 🤖 Feed full post text into LLMs, RAG pipelines, embeddings and vector databases.
• 📰 Monitor publications — track new posts, reactions, comments and restacks over time.
• 💵 Analyze monetization — which creators run paid subscriptions and podcasts.
• 🗂️ Archive your own or clients' newsletters as structured, portable data.

✨ What this Substack scraper does

• ✅ Newsletter discovery by category — browse any of 20+ category leaderboards (Technology, Business, Finance, Crypto, Science, News, Culture, U.S. Politics, Health & Wellness, Education, Music, Travel, History, Philosophy, Faith & Spirituality, Literature, Sports, Food & Drink, Climate & Environment, Humor and more) and get the top publications ranked.
• ✅ Creator lead fields — for each publication: name, author and author handle, subscriber count (e.g. 1,100,000), tagline, contact email (when public), author bio, bestseller tier, monetization flag, podcast flag, language, logo, cover image, creation date, first-post date and canonical URL.
• ✅ Full post scraping — give a publication subdomain, URL or custom domain and collect its posts: title, subtitle, full plain-text content, byline, publish date, audience, reaction count, comment count, restack count, word count, cover image and post URL.
• ✅ Direct post fetching — paste individual …/p/… post URLs to grab specific posts.
• ✅ Podcast support — audio URL and episode duration for podcast posts.
• ✅ Automatic pagination — walks category leaderboards and post archives page by page and returns hundreds to thousands of de-duplicated records in one run.
• ✅ Free / paid filtering — restrict to free (public) posts, paid posts, or all, and see which posts are paywalled.
• ✅ Pure HTTP, no browser — fast and cheap; runs on cost-effective datacenter proxies with automatic fresh-IP retries.
• ✅ Export anywhere — JSON, CSV, Excel, HTML or the Apify API, plus Make, Zapier and Google Sheets integrations.

🎯 Use cases

• Newsletter sponsorship & ad sales — pull the top finance or tech newsletters with author and subscriber count to build a prospecting list.
• PR & influencer marketing — assemble targeted outreach lists of writers and podcasters in a niche.
• Competitive intelligence — benchmark a publication against the category leaders on size, cadence and engagement.
• Content & media research — study what topics top creators publish and how audiences react.
• AI & RAG pipelines — ingest full post text for summarization, classification, embeddings or a knowledge base.
• Content monitoring & alerting — schedule the actor to track a publication's new posts, reactions and restacks.
• Monetization analysis — map which creators run paid tiers and podcasts across a vertical.
• Personal / client archiving — export a complete newsletter archive as structured data.

📥 Input

Configure the actor in the visual editor or via JSON. Leave everything blank to get a sample of the top Technology newsletters. Supply categories to discover creators, publications to scrape posts, or individual post URLs — you can combine all three.

Field	Type	Description
discoverCategories	array	Category IDs to find the top newsletters in (select from the dropdown of 20+ categories). Each result is a creator lead with author, subscriber count and contact.
publications	array	Substack publications to scrape posts from — a subdomain (astralcodexten), a full URL (https://newsletter.pragmaticengineer.com), or a custom domain.
postUrls	array	Optional. Individual Substack post URLs (…/p/…) to fetch directly.
scrapePosts	boolean	When on (default), collect each publication's posts. Turn off to get only the publication profile as a lead.
includePostContent	boolean	When on (default), fetch each post's full plain text. Turn off for a faster metadata-only list.
includeHtml	boolean	Also include the original HTML of each post's content. Default off.
audience	select	Filter posts by audience: all, free, or paid.
maxPostsPerPublication	integer	Max posts to collect per publication. 0 = all. Default 20.
maxPublications	integer	Max publications to return when discovering by category. Default 50.
sinceDate	string	Optional. Only collect posts published on or after this date, e.g. 2026-01-01.
proxyConfiguration	object	Proxy settings. Datacenter proxies (the default) are more than enough.
maxConcurrency	integer	How many publications/posts to process in parallel (1–12). Default 5.

Example — discover finance creators (lead list)

{
  "discoverCategories": ["153"],
  "maxPublications": 50,
  "scrapePosts": false
}

Example — scrape a publication's posts

{
  "publications": ["astralcodexten", "https://www.honest-broker.com"],
  "maxPostsPerPublication": 30,
  "includePostContent": true,
  "audience": "free"
}

📤 Output

Every record carries a recordType of either publication (a creator lead) or post. Results are stored in a standard Apify dataset you can preview as a table or export to JSON, CSV, Excel or the API.

Creator lead (recordType: "publication") — real record from a fresh run:

{
  "recordType": "publication",
  "publicationId": 458709,
  "name": "The Pragmatic Engineer",
  "url": "https://newsletter.pragmaticengineer.com",
  "subdomain": "pragmaticengineer",
  "customDomain": "newsletter.pragmaticengineer.com",
  "tagline": "Big Tech and startups, from the inside. Highly relevant for software engineers, AI engineers and engineering leaders.",
  "author": "Gergely Orosz",
  "authorHandle": "pragmaticengineer",
  "authorBio": "Big Tech and startups from the inside. Especially relevant for software engineers / AI engineers.",
  "authorBestsellerTier": 10000,
  "subscribers": "1,100,000",
  "subscribersLabel": "Over 1,100,000 subscribers",
  "contactEmail": null,
  "isMonetizing": true,
  "hasPodcast": true,
  "language": "en",
  "copyright": "Gergely Orosz",
  "logoUrl": "https://substackcdn.com/image/...",
  "coverImage": "https://bucketeer-....s3.amazonaws.com/public/images/....png",
  "createdAt": "2021-08-25T13:08:12.798Z",
  "firstPostDate": "2021-08-30T11:24:02.082Z",
  "category": "Technology",
  "categoryRank": 2,
  "scrapedAt": "2026-11-02T00:50:20.199Z"
}

Post (recordType: "post") — real record from a fresh run:

{
  "recordType": "post",
  "publication": "Astral Codex Ten",
  "publicationUrl": "https://www.astralcodexten.com",
  "publicationAuthor": "Scott Alexander",
  "postId": 212349565,
  "title": "Your Book Review: The History And Adventures Of An Atom",
  "subtitle": "Finalist #14 in the Book Review Contest",
  "slug": "your-book-review-the-history-and",
  "url": "https://www.astralcodexten.com/p/your-book-review-the-history-and",
  "postDate": "2026-10-30T16:30:33.689Z",
  "type": "newsletter",
  "audience": "free",
  "isPaid": false,
  "author": "Scott Alexander",
  "reactionCount": 81,
  "commentCount": 56,
  "restackCount": 3,
  "wordcount": 5789,
  "description": "Finalist #14 in the Book Review Contest",
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/....jpeg",
  "podcastUrl": null,
  "podcastDurationSeconds": null,
  "content": "[This is one of the finalists in the 2026 book review contest …full plain text…]",
  "scrapedAt": "2026-11-02T00:54:11.955Z"
}

Field reference

Field	Description
recordType	publication (creator lead) or post.
publicationId / postId	Stable Substack numeric id for the publication or post.
name	Newsletter name (publication records).
author / authorHandle	Creator / author name and their Substack handle.
subscribers	Subscriber count as a clean number (e.g. 1,100,000), present whenever the publication exposes it.
subscribersLabel	The verbose subscriber label as Substack shows it (e.g. Over 1,100,000 subscribers).
tagline	Publication tagline / hero text.
contactEmail	Publication contact email, present only when the publication publishes one (rare).
authorBio	Short author bio (publication records).
authorBestsellerTier	Substack bestseller tier signal, when ranked.
isMonetizing	Whether the publication has paid subscriptions enabled.
hasPodcast	Whether the publication runs a podcast.
language / copyright	Publication language code and copyright line.
logoUrl / coverImage	Publication logo and cover image (cover present when set).
createdAt / firstPostDate	Publication creation date and date of its first post (ISO 8601).
subdomain / customDomain	*.substack.com subdomain and custom domain (custom domain when configured).
category / categoryRank	Discovery category and rank within it (discovery records).
title / subtitle	Post title and subtitle (post records).
type	Post type, e.g. newsletter, podcast, thread.
content	Full post text in plain text (when includePostContent is on).
contentHtml	Original post HTML (when includeHtml is on).
description	Post meta description / summary.
postDate	Post publish date (ISO 8601).
audience / isPaid	free or paid, and a paywall boolean.
reactionCount	Number of reactions / likes on the post.
commentCount	Number of comments on the post.
restackCount	Number of restacks (shares) on the post.
wordcount	Post word count.
coverImage	Post cover image URL, when set.
podcastUrl / podcastDurationSeconds	Podcast media URL and duration, for podcast posts only.
publication / publicationAuthor / publicationUrl	Source newsletter context on post records.
url	Publication or post URL.
scrapedAt	ISO 8601 timestamp of collection.

Some fields populate only when available. In a fresh verification run, publication leads returned name, author/authorHandle, url, isMonetizing, hasPodcast, language, createdAt and firstPostDate at 100%, tagline/category/categoryRank at ~99%, and subscribers at ~96% — only publications that choose to hide their subscriber count come back blank. contactEmail is sparse because Substack rarely publishes a public email. For posts, the core fields (title, url, postDate, reactionCount, commentCount, restackCount, wordcount, audience, isPaid, coverImage, content) returned at 100%; podcastUrl appears only on podcast posts, and publication/publicationAuthor are empty on posts fetched by direct URL. Paid posts return their public preview text.

🔁 Pagination & scale

The actor paginates automatically on both data sources. Category discovery walks the leaderboard page by page (?page=0,1,2…), following Substack's own "more" flag and stopping when the list ends or maxPublications is reached. Post collection pages through each publication's archive by offset (12 posts per request), continuing until the archive runs out or maxPostsPerPublication is reached (0 = all posts). Records are de-duplicated and the actor comfortably returns hundreds to thousands of publications and posts across a run — a recent verification pulled 482 newsletter leads across 8 categories in a single run. Scale is controlled entirely by maxPublications, maxPostsPerPublication and the number of categories/publications you pass; maxConcurrency tunes throughput.

⚙️ How to use

1. Open the actor on the Apify Console.
2. Pick one or more categories to discover creators, and/or paste publication URLs (and optionally individual post URLs).
3. Choose whether to scrape posts, include full content, and filter by audience; set maxPublications / maxPostsPerPublication.
4. Click Start. Watch records stream into the dataset in real time.
5. Export to JSON, CSV, Excel or HTML, or pull results via the Apify API.
6. Automate it: run on a schedule for monitoring, or wire the dataset into Make, Zapier, Google Sheets or a webhook.

💡 Tips

• Discover first, scrape later — find the newsletters in a niche with discoverCategories + scrapePosts: false, then feed the best subdomains back into publications to pull their posts.
• Turn off includePostContent for a fast, lightweight metadata list; turn it on for full text and AI pipelines.
• Filter by audience: "free" to focus on fully public posts that return complete content.
• Use sinceDate for incremental runs that only pick up posts published after your last scrape.
• Sort by subscribers downstream to surface the biggest creators in a category — it is now a clean number you can sort and threshold on.
• Keep maxConcurrency at 3–8 — datacenter proxies with fresh-IP retries handle this comfortably.

❓ FAQ

Is scraping Substack legal? This Substack scraper reads only publicly available information — public leaderboards, publication profiles and public posts. You are responsible for how you use the data; respect copyright, Substack's terms, and privacy law (any contact data needs a lawful basis for outreach, and you should honor opt-out requests).

How many results can I get and how fast? Because it is pure HTTP, the Substack scraper is fast and can return hundreds to thousands of records per run — a recent run returned 482 newsletter leads across 8 categories. Throughput depends on maxConcurrency, how many publications you scrape and whether full content is enabled.

Do I need proxies or a login? No login is required. Datacenter proxies (the default) are sufficient; the actor rotates IPs and retries automatically on rate limits.

Can I get subscriber counts? Yes — the subscribers field returns a clean number (e.g. 1,100,000) whenever the publication exposes its count, with the verbose subscribersLabel (e.g. Over 1,100,000 subscribers) alongside it. In testing ~96% of discovery leads carried a subscriber count; the only blanks are publications that hide the number.

Can I get full post content? Yes for public (free) posts. Paid posts return the public preview text only. Enable includePostContent (on by default) for plain text, and includeHtml for the original HTML.

Is there a Substack API? Substack has no official public data API. This actor is the practical Substack API alternative — it returns clean, structured JSON you can consume programmatically via the Apify API.

How fresh is the data? Every run scrapes live, so results reflect Substack at run time. Schedule the actor to keep a dataset continuously up to date.

Can I filter by category, audience or date? Yes — pick categories in discoverCategories, restrict posts with audience, and limit by publish date with sinceDate.

What formats can I export? JSON, CSV, Excel and HTML, plus the Apify API and integrations like Make, Zapier and Google Sheets.

A common gotcha? If you only want leads (no posts), set scrapePosts: false so you get lightweight publication profiles instead of full archives — much faster and cheaper.

🔗 Related scrapers

• RSS Feed Reader – Parse RSS, Atom & Podcast Feeds to JSON
• Article Content Extractor — Clean Text, Markdown & Metadata
• Academic Paper Resolver — DOI/arXiv to Citations for AI
• USAspending Federal Awards Scraper
• MLB Stats Scraper — Baseball Teams, Players, Scores & Standings
• EPA ECHO Facility Compliance & Enforcement Scraper

---

Ready to build your creator lead list or content feed? Click Start and scrape Substack now. 📰