Substack Scraper — Posts, Comments, Notes & Search

Scrape Substack newsletters into clean data: full post text, authors, reactions and complete comment threads, plus keyword search across all of Substack, category leaderboards with subscriber counts, and writers' Substack Notes. No login, no API key, no cookies. Download the results as JSON, CSV, Excel or HTML, or pull them through the API.

What does Substack Scraper do?

Substack Scraper collects public data from Substack in five ways, which you can combine in one run:

• Publications — give it newsletters (lenny, on.substack.com, www.astralcodexten.com) and get their posts from the full archive, not just the latest 12: title, subtitle, full body as HTML and plain text, authors, date, word count, reactions, comment count, cover image and a paywall flag.
• Comments — every comment and nested reply on those posts, with author, date, reactions and thread position (parentId, depth).
• Search all of Substack — enter keywords and get matching posts (with full text) or matching publications (newsletter name, URL, author, subscriber count).
• Leaderboards — Substack's Bestseller and Rising rankings per category (Technology, Business, Culture, Finance, U.S. Politics and 30 more) with rank, author and subscriber counts.
• Substack Notes — the short-form Notes of any writer: text, date, reactions, restacks, replies and attachments (linked posts, links, images, quoted notes).

For scheduled runs, Only new posts mode remembers what earlier runs delivered and outputs just the posts published since then.

Use cases

• Newsletter and media monitoring — track competitors' or partners' newsletters daily and get only the new posts in Slack, Google Sheets or your CRM.
• Market and trend research — search all of Substack for a topic ("AI agents", "climate tech") and read what independent writers publish about it.
• Sponsorship and partnership prospecting — pull the top 100 publications of a category with subscriber counts and author handles to build an outreach list.
• Audience and engagement analysis — compare reactions, comment counts and comment threads across posts to see which topics resonate.
• AI and RAG pipelines — feed full post text and Notes into embeddings, summaries or an LLM knowledge base.
• Creator research — follow what influential writers post on Substack Notes, with reactions and restacks as engagement signals.

How much does it cost?

You pay only for the items you receive (pay-per-event). Duplicates and items over your maxItems limit are never charged.

Event	What you pay for	Free plan	Paid plans (from)
post	One post with full text and metadata	$1.00 per 1,000	$0.50 per 1,000
comment	One comment or reply	$0.30 per 1,000	$0.15 per 1,000
note	One Substack Note	$0.50 per 1,000	$0.25 per 1,000
publication	One publication from search or a leaderboard	$1.00 per 1,000	$0.50 per 1,000

"Paid plans (from)" is the Gold-and-above price; Bronze and Silver plans sit in between. Apify also charges a tiny actor-start fee ($0.00005 per run).

Worked examples (Free plan prices):

1. Archive of 10 newsletters, 50 posts each — 500 posts × $1.00 / 1,000 = $0.50. With comments on (say 3,000 comments) add 3,000 × $0.30 / 1,000 = $0.90.
2. Top 100 Technology bestsellers + 200 Notes from 4 writers — 100 publications ($0.10) + 200 notes ($0.10) = $0.20.

The Apify Free plan includes $5 monthly credit, enough for about 5,000 posts or 10,000 notes every month.

How to use Substack Scraper

In Apify Console:

1. Open Substack Scraper and click Try for free.
2. Fill in at least one target: Publications, Search keywords, Leaderboard categories or Notes authors.
3. Optional: turn on Include comments, set the limits, or enable Only new posts for scheduled runs.
4. Click Start and wait for the run to finish.
5. Open the Output tab and export the dataset as JSON, CSV, Excel or HTML.

Through the API:

1. Get your API token in Apify Console → Settings → Integrations.
2. Call the actor with your input JSON (see Integrations and API).
3. Read the dataset items from the run.

Input

Fill in at least one of publications, searchQueries, leaderboardCategories or noteProfiles.

Field	Type	Default	Description
publications	array	–	Newsletters: subdomains (on, lenny), hosts (on.substack.com, www.lennysnewsletter.com) or URLs. Aliases of the same publication are scraped once.
includePosts	boolean	true	Output one item per post with full text.
includeComments	boolean	false	Output every comment and reply of the collected posts (publication and search posts).
maxPostsPerPublication	integer	50	Posts per publication (1–1000).
query	string	""	Keep only posts of the listed publications whose title or text contains this word (scans up to 500 posts per publication).
sort	string	new	Archive order: new (newest first) or top.
onlyNewPosts	boolean	false	Output only posts not delivered by an earlier run.
stateName	string	default	Separate "already delivered" memory per schedule or task.
searchQueries	array	–	Keywords searched across all of Substack, each separately.
searchType	string	posts	posts (full posts) or publications (newsletters).
maxSearchResults	integer	50	Results per keyword (1–1000).
leaderboardCategories	array	–	Category slugs or names (technology, business, culture, finance, us-politics, podcast, …) or URLs like https://substack.com/leaderboard/technology/paid.
leaderboardType	string	paid	paid (Bestsellers) or trending (Rising). A /rising or /paid URL overrides it.
maxPublicationsPerLeaderboard	integer	100	Ranked publications per category.
noteProfiles	array	–	Writers: @handle, handle, https://substack.com/@handle or https://substack.com/profile/<id>-<name>.
maxNotesPerProfile	integer	50	Most recent Notes per writer (1–5000).
maxItems	integer	1000	Hard cap on all output items in the run.
proxyConfiguration	object	Apify Proxy	Proxy settings.
debug	boolean	false	More detailed run log.

Example — a daily monitoring run that also pulls a leaderboard and a writer's Notes:

{
    "publications": ["on.substack.com", "www.astralcodexten.com"],
    "maxPostsPerPublication": 10,
    "includeComments": true,
    "onlyNewPosts": true,
    "searchQueries": ["climate tech"],
    "searchType": "posts",
    "maxSearchResults": 20,
    "leaderboardCategories": ["technology"],
    "leaderboardType": "paid",
    "noteProfiles": ["@hamish"],
    "maxNotesPerProfile": 30,
    "maxItems": 2000
}

Output

All items go into one dataset; the type field tells them apart: post, comment, note or publication. Examples below come from real runs (long text shortened).

Post

{
    "type": "post",
    "publication": "on",
    "publicationName": "On Substack",
    "postId": 222802023,
    "title": "New on Substack: chat channels, lists, freewalls, discounts, podcast updates, and more",
    "subtitle": "More ways to connect with your audience, get paid for your work, and be in control of your feed",
    "slug": "new-on-substack-chat-channels-lists",
    "url": "https://on.substack.com/p/new-on-substack-chat-channels-lists",
    "publishedAt": "2026-10-30T17:03:17.633Z",
    "authors": [],
    "audience": "everyone",
    "bodyHtml": "<div class=\"captioned-image-container\"><figure>...",
    "bodyText": "Illustrations by Jac Saik, Designer at Substack\nOur latest product updates give everyone more ownership over what they read...",
    "wordCount": 1115,
    "reactions": 736,
    "commentCount": 0,
    "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/4be31bf5-e657-46bf-b998-79e7cb8f0306_2398x1342.png",
    "bodyTruncated": false,
    "fetchedAt": "2026-11-02T13:55:48Z"
}

• bodyTruncated is true when a paid post is only available as its free preview; the scraper never bypasses paywalls.
• authors is empty when the publication hides bylines.
• Posts found by keyword search also carry searchQuery.

Comment

{
    "type": "comment",
    "publication": "astralcodexten",
    "publicationName": "Astral Codex Ten",
    "postId": 223080029,
    "commentId": 376501150,
    "parentId": null,
    "depth": 0,
    "author": "Whenyou",
    "authorHandle": "personfromdenmark123",
    "isPostAuthor": false,
    "body": "An older case of an intellectually disabled man ...",
    "createdAt": "2026-11-02T09:53:02.947Z",
    "reactions": 1,
    "fetchedAt": "2026-11-02T12:16:44Z"
}

Top-level comments have parentId: null and depth: 0; replies point to their parent comment. Deleted comments are skipped, their visible replies are kept.

Note

{
    "type": "note",
    "noteId": 374632951,
    "url": "https://substack.com/@hamish/note/c-374632951",
    "author": "Hamish McKenzie",
    "authorHandle": "hamish",
    "authorId": 3567,
    "body": "Bundle of goodies ",
    "publishedAt": "2026-10-30T22:43:42.413Z",
    "editedAt": null,
    "reactions": 44,
    "restacks": 4,
    "replies": 4,
    "attachments": [
        {
            "type": "post",
            "url": "https://on.substack.com/p/new-on-substack-chat-channels-lists",
            "title": "New on Substack: chat channels, lists, freewalls, discounts, podcast updates, and more",
            "imageUrl": "https://substack-post-media.s3.amazonaws.com/public/images/4be31bf5-e657-46bf-b998-79e7cb8f0306_2398x1342.png"
        }
    ],
    "profile": "@hamish",
    "fetchedAt": "2026-11-02T13:56:28Z"
}

attachments[].type is post, link, image, note (a quoted note) or publication_recommendation. body can be empty when a note only shares a post.

Publication (leaderboard or search)

{
    "type": "publication",
    "publicationId": 6349492,
    "name": "SemiAnalysis",
    "subdomain": "semianalysis",
    "url": "https://newsletter.semianalysis.com",
    "customDomain": "newsletter.semianalysis.com",
    "description": "Bridging the gap between the world's most important industry, semiconductors, and business.",
    "authorName": "Dylan Patel",
    "authorHandle": "semianalysis",
    "authorId": 21783302,
    "logoUrl": "https://substackcdn.com/image/fetch/$s_!y_cf!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1af867ac-65cc-46ad-aa7b-13323ae19142_444x415.png",
    "language": "en",
    "subscriberCount": 326000,
    "subscriberCountText": "Over 326,000 subscribers",
    "paidSubscribersText": "Tens of thousands of paid subscribers",
    "source": "leaderboard",
    "searchQuery": null,
    "category": "Technology",
    "categorySlug": "technology",
    "leaderboardType": "paid",
    "rank": 1,
    "fetchedAt": "2026-11-02T13:56:26Z"
}

• subscriberCount is the number Substack displays publicly ("Over 326,000", "8.6K+"), so it is rounded; subscriberCountText keeps the original wording. It is null when the writer hides the count.
• paidSubscribersText is Substack's paid-subscriber range (leaderboards only).
• rank, category and leaderboardType are filled for leaderboard rows; searchQuery for search rows. Fields Substack does not show for a row are null.

Integrations and API

Run the actor and get the items back in one call:

curl -X POST "https://api.apify.com/v2/acts/innolabs~substack-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"publications": ["on.substack.com"], "maxPostsPerPublication": 10}'

Python with apify-client:

from apify_client import ApifyClient

client = ApifyClient("YOUR_TOKEN")
run = client.actor("innolabs/substack-scraper").call(
    run_input={"searchQueries": ["AI agents"], "searchType": "posts", "maxSearchResults": 20}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["type"], item.get("title") or item.get("name"), item.get("url"))

• Make, Zapier, n8n — use the Apify app/node to start the actor and send new posts to Google Sheets, Airtable, Notion, Slack or email.
• Scheduling — create a schedule in Apify Console (e.g. every morning) and turn on Only new posts, so each run delivers just what was published since the last one.
• Webhooks — trigger your pipeline when a run finishes.
• AI agents (MCP) — connect Claude, Cursor or other MCP clients through the Apify MCP server at https://mcp.apify.com and let your agent search Substack or read newsletters on demand.

FAQ

Is it legal to scrape Substack?

The scraper only collects publicly visible data: free posts, public previews of paid posts, public comments, Notes, search results and leaderboards that anyone can see without logging in. It does not log in or bypass paywalls. Personal data in comments and Notes may be protected by laws such as GDPR, so make sure you have a legitimate reason to store it and check that your use complies with Substack's terms.

Do I need proxies?

Usually not: Substack's public pages are light on blocking and the scraper retries throttled requests automatically. Apify Proxy is on by default; keep it on for large or frequent runs.

How many items can I get?

Up to 1,000 posts per publication, 1,000 results per search keyword, the full leaderboard of each category (Substack lists up to 100 publications) and up to 5,000 Notes per writer. maxItems caps the whole run so you never spend more than you planned.

Can I get only new posts?

Yes. Turn on Only new posts and run the actor on a schedule. The first run delivers posts normally and remembers them; later runs output only posts published after that (for search keywords: results not delivered before). The memory is kept in your Apify account under the name in State name, so use different state names for different schedules, or a new name to start over.

What if a publication, writer or category fails?

Each target is processed on its own. If one newsletter, post, writer or category cannot be loaded (wrong name, deleted, temporarily unavailable), the run continues with the rest and the problem is listed in the run log and in the run's SUMMARY record. Failed items are not charged.

Can it scrape paid posts?

It returns what Substack shows to non-subscribers. For paid posts that is the free preview, and bodyTruncated: true tells you the text is incomplete.

Limits

• Paid posts: only the free preview is available (bodyTruncated: true).
• Subscriber counts are Substack's rounded public figures; exact numbers are not public. Paid-subscriber figures are ranges ("Tens of thousands") and appear only on leaderboards.
• Leaderboards exist for Substack's top-level categories; subcategories have no public ranking. Each leaderboard lists up to 100 publications.
• Publication search returns the newsletters Substack itself suggests for the keyword (typically up to ~100 per keyword).
• Notes: the writer's own Notes are collected (newest first); restacks of other people's Notes are skipped. Replies to a Note are counted (replies) but not downloaded.
• Only new posts applies to posts (from publications and keyword search); Notes, comments of older posts, leaderboards and publication search are always delivered in full.
• Comments are collected for the posts scraped in the same run.