Substack Scraper (public newsletter posts)

Substack hosts tens of thousands of newsletters, and the archive of any of them is a list of posts you can only read page by page. Watching a publication over time, comparing several, or pulling every post that mentions a topic needs those posts as rows, with the likes and comment counts attached.

Give it publication names (or links) and every post comes back as one item: title, subtitle, date, the author byline, audience (free or paid), likes, comment count, word count, tags, cover image and a short preview. Switch on includeBody and free posts also come with their full text. No login, no API key, no Substack account. It reads the public archive that every Substack publication serves to any visitor. The publication's name and the author names printed on each post stay on the row as its byline; commenters, subscribers, readers, author handles, bios and avatars are never collected.

Quick start (30 seconds)

1. Put one or more publications into publications, e.g. astralcodexten or https://www.lennysnewsletter.com, https://substack.com/@lenny.
2. Click Start. That's the whole minimum setup.
3. When the run finishes, open the dataset's Overview tab for a clean table, or Export it as CSV/Excel/JSON.

Optional knobs: a keyword to search for inside the publications, newest-first or most-popular-first, a date window, free or paid posts only, and the full text of free posts. Any input works on a daily Schedule.

What you can build with it

• Watch a publication for new posts. Put it in publications and set postedAfter to yesterday. The run stops as soon as it reaches older posts, so a daily Schedule only reads what is new.
• Find out what a topic does on Substack. Put a term in searchQuery across a list of publications and compare reactionCount, commentCount and restackCount to see which posts and which publications the topic travels in.
• Build a reading corpus. Turn on includeBody and every free post arrives as plain text next to its date, authors and tags, ready to embed, summarise or search.
• Feed an AI agent clean data. Every field is structured and predictable, so an assistant or pipeline can read a publication's last 100 posts directly.

What you get

Each post is one dataset item:

{
  "id": 208730073,
  "title": "Announcing Lenny’s Jobs: The best place in the world to find, vet, and land your dream job",
  "subtitle": "Where product managers, engineers, designers, and growth/marketing professionals discover high-quality open roles at tech companies",
  "slug": "announcing-lennys-jobs-the-best-place",
  "url": "https://www.lennysnewsletter.com/p/announcing-lennys-jobs-the-best-place",
  "canonicalUrl": "https://www.lennysnewsletter.com/p/announcing-lennys-jobs-the-best-place",
  "publicationName": "Lenny's Newsletter",
  "publicationUrl": "https://www.lennysnewsletter.com",
  "authors": ["Lenny Rachitsky"],
  "postDate": "2026-08-18T15:40:06.921Z",
  "audience": "everyone",
  "type": "newsletter",
  "section": null,
  "reactionCount": 389,
  "commentCount": 18,
  "restackCount": 10,
  "wordCount": 933,
  "coverImage": "https://substackcdn.com/image/fetch/$s_!PFZn!,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F0f696879-6374-42d6-a171-47933f6eb126_1456x970.png",
  "tags": ["Career"],
  "excerpt": "👋 Hey there, I’m Lenny. Each week, I share deeply researched product, growth, and career advice. For more: Lenny’s Jobs | Lenny’s Podcast | How I AI | Lennybot | Become an AI-Native Builder and some other favorite AI/PM courses"
}

With includeBody on, free posts carry one more field, bodyText, with the full text.

Field notes, so you know exactly what you are buying:

• id is Substack's numeric post id, unique across publications; slug is the post's name inside its publication. url is the human link on the publication's own address, which is its custom domain when it has one.
• authors are the names printed in the post's byline, guest authors included. Names only: no handles, profile links, avatars or bios.
• audience is everyone for free posts and only_paid for posts behind the paywall. A paid post comes back with all its metadata and the short public preview in excerpt, never its text. bodyText is null for paid posts whatever includeBody says.
• wordCount is the length of the whole post as Substack reports it, also for paid posts.
• excerpt is the short preview Substack shows for every post, about 200 characters. For many publications it is a standard intro line that repeats across posts.
• reactionCount, commentCount and restackCount are Substack's figures at scrape time. The comments themselves are not collected.
• postDate is the publication time in UTC.
• coverImage is the link as Substack serves it at run time; download it if you need to keep it.
• section is filled only for publications that sort their posts into sections.

How much does it cost to scrape Substack?

$3.00 per 1,000 posts delivered, pay-as-you-go, no subscription, and you pay per result rather than per hour. A run that finds nothing still costs the $0.005 actor start. In plain dollars:

• 100 posts ≈ $0.30: a daily multi-publication watch.
• 500 posts ≈ $1.50: a solid archive of a publication.
• A 10,000-post crawl ≈ $30: a large set of publications, deduplicated.

The price is all-inclusive: your runs' platform usage is covered by it, with no separate compute or proxy charges. Measured on the platform: a 25-post run finishes in about 5 seconds and a 300-post run in about 15 seconds. With includeBody on, plan for about two seconds per free post, since each text is a separate request. The actor reads Substack directly and only falls back to the Apify proxy if a request is blocked, so there is nothing to configure.

Free-plan runs are limited to a sample of 5 items, enough to evaluate the output format against your real publications.

Not technical? Let your AI assistant set it up

Copy this into ChatGPT, Claude or any AI assistant, fill in the one line, and follow the conversation:

Help me set up the "Substack Scraper" actor on Apify
(https://apify.com/lowlanddata/substack-scraper). Guide me one step at a time.

What I want to watch: [E.G. "new posts in astralcodexten and lenny that mention 'AI'"]

Guide me to:
1. Propose my input values: publications (names or links, comma-separated),
   an optional searchQuery for a keyword inside the publications, postedAfter for a
   date window, audience (all, free or paid), includeBody if I want the full text of
   free posts, and maxItems.
2. Check that each publication exists by opening its address in my browser
   (name.substack.com, or its own domain). If it shows the newsletter's home page,
   the actor can read it.
3. Create a free Apify account (apify.com), open the actor page, paste the values
   into the Input form, and start a run.
4. Set up a daily Schedule in the Apify Console with the same input, plus an email
   or Slack integration so new posts reach me automatically.
5. Show me how to export results as CSV/Excel, or read them from the API if I code.
6. If the results are what I wanted, remind me at the end to leave a quick rating on the actor page, and to report anything broken or missing on its Issues tab.

Input

Field	Description
publications	Publication names, addresses, custom domains or substack.com/@author links, comma- or space-separated, e.g. astralcodexten, https://www.lennysnewsletter.com. Required.
searchQuery	Only posts that match this term, searched inside each publication and listed by relevance. Optional.
sort	new (default) walks newest-first, top lists the most popular first. Ignored while a search term is set.
audience	all (default), free or paid. Paid posts come back as metadata and preview only.
postedAfter	Only posts published on or after this date (YYYY-MM-DD); a newest-first walk stops at older history.
postedBefore	Only posts published on or before this date (YYYY-MM-DD).
includeBody	Add bodyText, the full text of each free post. Off by default; needs one extra request per post.
maxItems	Stop after this many posts in total (default 500).
proxyConfiguration	Optional. Direct by default; the Apify proxy joins on its own if blocked. Set one to force a proxy.

Publications are read in the order you list them, and maxItems counts across all of them. An @author link is followed to the publication that author runs, and a link to a single post reads the publication it belongs to. A name that is not a Substack publication, or no longer exists, is not charged: the rest of the run is delivered, with an honest note in the run status. A name that cannot be recognised at all, such as localhost, is refused up front with the offending names in the message.

Use it from your code

Run the actor and get posts straight back with one HTTP call (fine for scoped runs up to ~5 minutes):

curl "https://api.apify.com/v2/acts/lowlanddata~substack-scraper/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -X POST -H "Content-Type: application/json" \
  -d '{"publications": "astralcodexten", "maxItems": 100}'

Node.js:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });
const run = await client.actor('lowlanddata/substack-scraper').call({
  publications: 'astralcodexten, https://www.lennysnewsletter.com',
  searchQuery: 'AI',
  maxItems: 200,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();

Python:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("lowlanddata/substack-scraper").call(
    run_input={"publications": "astralcodexten", "maxItems": 100})
items = client.dataset(run["defaultDatasetId"]).list_items().items

Schedules, webhooks and the Make/Zapier/n8n integrations all work out of the box: this is a standard Apify actor.

Use it with AI agents (MCP)

Claude, Cursor and other MCP-capable agents can run this scraper as a tool through Apify's hosted MCP server: the agent fills in the publications itself, starts the run and reads the results: no glue code.

Claude Code:

$claude mcp add apify --transport http "https://mcp.apify.com?actors=lowlanddata/substack-scraper"

Cursor or Claude Desktop (add a custom connector / MCP server with this URL):

https://mcp.apify.com?actors=lowlanddata/substack-scraper

Sign in with your Apify account when prompted. Runs are billed to it. Setup details per client: Apify MCP docs.

Prompts that work once connected:

• "Get the last 50 posts from the Substack publication astralcodexten and summarize what it covered this month."
• "Search the Substack publications noahpinion and lenny for posts about pricing and list them with their like counts."
• "Compare the average likes per post of these three Substack publications over their last 30 posts each."

What about paid posts?

Substack lets authors put posts behind a paywall. The actor never opens that text. A paid post still comes back as a row (title, subtitle, date, authors, likes, comment count, word count, tags, cover image) plus the short public preview Substack shows to everyone, so a publication's row count matches what its archive lists. Set audience to free for free posts only, or paid to see only what sits behind the paywall. With includeBody on, only free posts get their text.

Is it legal to scrape Substack?

A Substack publication is a broadcast: its archive and its free posts are open to anyone without logging in, and that is the only surface this scraper reads. It does not log in, subscribe, read comments or notes, or touch subscriber and reader data, and requests are paced at about one a second. The publication's name and the author names printed on each post stay on the row as its byline; everything beyond the post itself (commenters, subscribers, author handles, bios, avatars, the author's other publications) is never read, so none of it can reach your dataset.

One honest limit: titles, previews and post texts are the authors' own words, delivered as-is. If an author names a person or publishes contact details in a post, that text is not rewritten. The guarantee covers the data fields, not what authors choose to publish.

This is not legal advice. Substack's terms of use contain a clause on automated access, so check them against your own use. Whether your use of a publication's posts is allowed under those terms, copyright and your jurisdiction's rules is yours to assess.

Is there a Substack API alternative?

Substack publishes no official API for reading a publication's posts. This actor is the practical alternative: the same archive as structured JSON through one HTTP call (run-sync-get-dataset-items), on a schedule, or as an MCP tool for AI agents, with no account to create.

Does Substack block scrapers?

Substack serves each publication's archive openly and does not disallow its archive and post endpoints in robots.txt. This actor stays inside that welcome: it waits about a second between requests, goes direct, and keeps the load negligible. It does not read the paths robots.txt asks crawlers to leave alone (comments, notes, embeds, subscribe pages) and does not try to get around the paywall. No CAPTCHA fights, no bot-wall cat-and-mouse, which is also why runs are reliable enough for daily schedules.

How do I monitor a Substack publication?

Put the publication in publications, add a searchQuery if you only care about a topic, set postedAfter to yesterday's date, and add a daily Schedule in the Apify Console with an email/Slack integration on the runs. The dataset then contains only the new posts, each with its likes and comments at that moment. The AI-assistant prompt above walks a non-technical user through exactly this setup.

FAQ

Which publications can it read? Any public Substack publication, on a substack.com address or on its own domain. Name it (astralcodexten), link it, or give an author's substack.com/@handle link.

Can I get the text of paid posts? No. Paid posts come back with metadata and the public preview only. The text is collected for free posts, and only when includeBody is on.

Why does a name come back as "not a Substack publication"? The name does not exist on Substack, the publication was deleted, or the address belongs to a site that is not on Substack. Open name.substack.com in a browser: if it shows no newsletter there, the actor cannot read it either. It costs nothing and the rest of the run is delivered.

Can I search a publication by keyword? Yes, set searchQuery and only matching posts are returned, most relevant first. A search always lists by relevance, so the sort option is ignored and a date window filters the results instead of stopping the walk early.

Can I get only the newest posts? Yes, set postedAfter to a date. On a plain newest-first run each publication's walk stops as soon as it reaches older posts, so a daily watch collects exactly the new posts and nothing else.

How fresh is the data? Live at run time: every run reads the archive directly. For continuous freshness, schedule the actor.

Can I export to Excel or CSV? Yes, every dataset exports as CSV, Excel, JSON or XML from the Apify Console or API.

Does it work with Make, Zapier or n8n? Yes, it is a standard Apify actor; all platform integrations, webhooks and schedules apply.

Do I need a Substack account or API key? Neither. The public archive needs no login, and that is the only surface this actor reads.

Does it collect commenters, subscribers or author profiles? No. You get the publication's name and the author names printed on the post as its byline, and the post itself. Comments, commenter names, subscriber and reader data, author handles, bios and avatars are never read.

How do I scrape several publications in one run? List them comma- or space-separated in publications: "astralcodexten, lenny, noahpinion". Links and @author links are accepted too. maxItems counts across all of them.

What are reactionCount and restackCount? Likes and restacks (shares inside Substack) at the moment of the run. Run the actor daily on the same publications and diff the snapshots to see which posts travel.

Can I download the cover images? The rows carry links to them as Substack served them at run time. Download what you need to keep, since the links are not a permanent archive.

What happens on a large archive? The actor pages through the archive 50 posts at a time until maxItems, your date window or the first post of the publication. Measured on the platform: a publication with a 1,500-post archive came back whole in about two and a half minutes, using under 100 MB of memory.

How much does Substack scraping cost here? $3.00 per 1,000 posts, metered on delivery. A 100-post daily watch is roughly $0.30; a failed or empty run costs the $0.005 actor start and nothing per result.

Can I try it before spending anything? Yes, a free-plan run returns up to 5 real posts from your publications, enough to check the fields against your pipeline.

Can an AI agent fetch Substack posts through this? Yes, connect Apify's hosted MCP server and agents like Claude or Cursor can pick publications, run the actor and read the posts themselves. Setup is in the MCP section above.

Related scrapers

More clean-output data sources from the same maker:

• Bluesky Scraper: public posts from the open social network.
• Telegram Channel Scraper: posts from public Telegram channels.
• Kleinanzeigen.de Scraper: Germany's biggest classifieds site.

Troubleshooting

The actor fails fast with the reason in the run's status message:

• "not a Substack publication, nothing charged: ...": check the names, and open name.substack.com in a browser to confirm the newsletter exists. The rest of the run is delivered.
• "Not a Substack publication name, link or @handle: ...": a name in publications is not valid. Use a name like astralcodexten, a link like https://www.lennysnewsletter.com, or an author link like https://substack.com/@lenny.
• "could not be read (blocked or unreachable): ...": those publications failed after retries; the rest of the run is delivered normally. Try again later, or set a proxy.
• "N free posts had no body to read": the post was listed but its text was gone by the time it was fetched; the row keeps its metadata and bodyText is null.
• Fewer items than requested on a free plan: the 5-item free sample cap; run on a paid Apify plan for full results.

Support

Found an issue or missing a field you need? The fastest route is the actor's Issues tab - it is monitored and reports get fixed. For anything private, email contact@lowlanddata.com. This actor is actively maintained.

Working well for you? A rating on this page takes ten seconds and helps other users find a reliable option. It is also the clearest signal of what we should build next.