Substack Scraper - Find Newsletters, Posts & Comments

🔎 What does Substack Intelligence Scraper do?

Substack Intelligence Scraper finds Substack newsletters and turns them into structured post, comment, author, and publication datasets. Give it a category like Technology or Crypto and it returns the ranked leaderboard, then collects each newsletter's archive. You can also hand it *.substack.com publications, custom domains, archive pages, or individual post URLs.

Use it as a fast Substack API alternative for competitive research, newsletter monitoring, content analysis, sponsorship research, and AI-ready corpora. The Actor uses Substack's public HTTP endpoints and server-rendered data, so it does not need a browser or customer-managed proxy settings.

You do not need to know the newsletters first. Most Substack scrapers, including the largest one on the Store, require you to supply every publication URL yourself, and the leaderboard is sold as a separate Actor. This one does discovery and the archive in a single run.

Key capabilities include:

• Discovery by Substack category leaderboard, with the rank each newsletter holds
• Discovery by search phrase, for topics that do not map onto one category
• Three leaderboard rankings: all, paid bestsellers, and free
• Complete public archive pagination with 0 for unlimited depth
• Public post HTML, normalized plain text, and Markdown
• Nested comment threads plus optional analytics-friendly comment rows
• Publication, audience, author, tag, podcast, reaction, and restack metadata
• Date, post type, paywall, include-keyword, and exclude-keyword filters
• Direct collection from a single /p/... post URL
• Persistent onlyNew monitoring for scheduled runs
• Per-source and per-post error rows, so one bad URL does not discard a batch

Ready-to-run examples

• Top Technology Substack newsletters - ranked leaderboard, no URLs needed
• Substack paid bestsellers by category - who actually sells subscriptions
• Top newsletters and their latest posts - discovery and archive in one run
• Monitor new Substack newsletter posts
• Export Substack comments and replies
• Build a Substack RAG dataset

👥 Who is this Substack scraper for?

• Newsletter teams monitoring competitors and topic coverage
• Researchers comparing publishing cadence and reader engagement
• Sponsorship and partnership teams evaluating public publications
• Data journalists studying newsletter ecosystems
• AI teams preparing searchable text or RAG datasets
• Creators auditing their own public archives

🧭 Find newsletters without knowing their URL

Set Substack categories (or Search phrases) and the Actor returns a ranked list of publications before it collects anything else. Each discovered row carries leaderboardRank, leaderboardCategory and leaderboardRanking, so the ranking itself is data you can sort and track, not just a way to find URLs.

• 33 categories, from Technology and Finance to Crypto, Parenting and Games.
• Three rankings. paid is a genuinely different ordering of the same category, not a filter of all, so it answers "who actually sells subscriptions here?".
• discoveryOnly returns the ranked list and stops, which is much faster and cheaper when the list is all you need.
• Anything discovered flows into the normal pipeline, so every filter below (dates, keywords, paywall, post type) applies to newsletters you never had to name.

Discovered publications are then resolved through their homepage, which is where the subscriber count comes from. The leaderboard payload itself does not carry one.

⚙️ How to scrape Substack newsletters

1. Add one or more newsletter homepages, custom domains, archive pages, or post URLs.
2. Keep the safe default of 50 posts per publication or use 0 for a complete archive.
3. Choose whether to fetch public content and select HTML, text, Markdown, or all formats.
4. Optionally enable comments and select nested, separate-row, or combined output.
5. Apply date, content-type, paywall, or keyword filters.
6. Start the Actor and open the Output tab.
7. Download JSON, CSV, Excel, XML, RSS, or connect through the Apify API.

Apify can run the same configuration on a schedule, send a webhook when it finishes, and connect the dataset to Make, Zapier, Google Sheets, cloud storage, or your own application.

📥 Input

{
  "categories": ["Technology"],
  "leaderboardRanking": "paid",
  "maxPublicationsPerSource": 25,
  "discoveryOnly": false,
  "maxPostsPerNewsletter": 20
}

• categories - Substack categories to pull the ranked leaderboard for. Optional.
• leaderboardRanking - all, paid, or free. Default all.
• searchQueries - find publications by phrase instead of category. Optional.
• maxPublicationsPerSource - publications to take per category or phrase. Default 25.
• discoveryOnly - return the ranked list and skip the archives. Default false.
• urls - explicit publication or post URLs. Optional once you use discovery.

Everything below still applies, and now applies to discovered newsletters too.

See the Input tab for every option. A typical run looks like this:

{
    "urls": ["https://example.substack.com", "https://newsletter.example.com/p/a-public-post"],
    "maxPostsPerNewsletter": 100,
    "includeContent": true,
    "contentFormat": "all",
    "includeComments": true,
    "commentOutput": "both",
    "includePublicationInfo": true,
    "publicationOutput": "both",
    "contentType": "all",
    "onlyFree": false,
    "includeKeywords": ["AI", "product strategy"],
    "onlyNew": false
}

maxPostsPerNewsletter limits matching posts, not archive pages scanned. Keyword matching is case-insensitive and checks titles, subtitles, descriptions, public preview text, and tags.

📤 What Substack data can be extracted?

Data	Important fields
Post identity	postId, title, subtitle, slug, url
Timing and access	publishedAt, updatedAt, postType, audience, isPaid
Public content	bodyHtml, bodyText, bodyMarkdown, truncatedBodyText
Engagement	reactionCount, reactions, commentCount, restacks
Discovery	tags, language, coverImage, description, wordCount
Authors	authors, authorName, authorHandle, public bios and photos
Publication	name, URL, subscriber signals, branding, benefits, paid-plan status
Comments	body, date, author, reactions, pinned status, reply depth and parent ID

You can download the dataset in various formats such as JSON, HTML, CSV, or Excel. The Output tab includes dedicated views for posts, comments, publications, and all records.

Example post row:

{
    "itemType": "post",
    "postId": 123456789,
    "title": "A practical guide to product strategy",
    "url": "https://example.substack.com/p/product-strategy",
    "publishedAt": "2026-08-01T12:00:00.000Z",
    "postType": "newsletter",
    "isPaid": false,
    "wordCount": 2450,
    "tags": ["Product", "Strategy"],
    "reactionCount": 120,
    "commentCount": 18,
    "bodyText": "Public article content...",
    "bodyMarkdown": "## Public article content...",
    "publicationName": "Example Newsletter",
    "scrapedAt": "2026-08-09T12:00:00.000Z"
}

💵 How much does it cost to scrape Substack?

The recommended Store pricing is pay per event with no start fee: $2.00 per 1,000 posts and $0.50 per 1,000 comments, with lower prices for higher Apify subscription tiers. Publication and recoverable error rows are free. Comments are charged once when collected, whether you choose nested, separate, or combined comment output.

For cost-efficient runs:

• Start with includeContent: false to survey metadata quickly.
• Keep comments disabled unless reader discussion is required.
• Use a date window for recurring research.
• Keep the default 50-post cap for the first run.

🔁 How to monitor new Substack posts

Enable onlyNew and run the same input on an Apify schedule. The Actor stores emitted post IDs in a configuration-specific named key-value store. Later runs suppress previously delivered posts while preserving separate state for different URL and filter combinations.

For RSS delivery, request the last task dataset in RSS format and map title, url, description, and publishedAt to feed fields.

🛠️ Substack scraper API

Run the Actor from Node.js with apify-client:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('webdata_labs/substack-intelligence-scraper').call({
    urls: ['https://on.substack.com'],
    maxPostsPerNewsletter: 20,
    contentFormat: 'text',
    onlyNew: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

The same input works through the REST API, Apify CLI, webhooks, schedules, and MCP integrations.

⚠️ Limits and responsible use

• A category and ranking serves at most 525 publications. Substack stops at page 20 and returns an error beyond it. Smaller categories run out well before that. The payload claims there is more even on the last page, so the number is the source's, not a setting.
• Leaderboard rows carry no subscriber count. Substack does not expose one on that endpoint. Subscriber counts come from each publication's own homepage, so they appear on publication and post rows rather than on discovery rows, and only when a publication publishes the figure.
• Ranks are a snapshot. The leaderboard is read at run time; there is no rank history. Schedule the Actor if you want a series.
• Search finds fewer newsletters than a category. Category leaderboards are the deeper surface; search is for topics that do not map onto one category.

The Actor collects only data available without authentication. It does not log in, bypass access controls, or return paid post bodies. Paid posts can still expose public metadata and preview text. Substack can change undocumented public endpoints or page payloads, so recoverable failures are emitted as structured error rows.

Public comments and author profiles can contain personal data. Use the dataset only for a legitimate purpose and comply with Substack's terms, GDPR, CCPA, copyright rules, and other applicable laws. If you are unsure whether your use is legitimate, consult qualified legal counsel.

🧩 Related Actors

• Bluesky Scraper - posts, profiles and follower graphs on the open network.
• Reddit Monitor & Scraper - brand mentions and buying signals in the threads people trust.
• RedNote & Douyin Scraper - what is trending on Xiaohongshu and Douyin before it reaches the West.
• UFC Scraper - fighters, fights, events and rankings from ufc.com.
• Threads Scraper - posts, profiles and search on Meta's text network.

❓ FAQ and support

Does it support custom Substack domains?

Yes. Add the full custom-domain URL and the Actor discovers its canonical publication metadata.

Can it scrape a single post?

Yes. A URL containing /p/post-slug is treated as a direct post and does not expand into the full archive.

Why is a paid post body empty?

The Actor intentionally omits paid post bodies. Public metadata, engagement, tags, word count, and preview text can still be returned.

How are duplicate inputs handled?

Duplicate source strings, publication archives, and post IDs are deduplicated within each run. onlyNew extends post deduplication across scheduled runs with the same monitoring configuration.

For bugs or feature requests, use the Actor's Issues tab and include the public source URL plus the run ID. Do not include private credentials or subscription cookies.