Substack Newsletter Scraper: Posts, Archive and Publication Data

Give this Actor a list of Substack newsletters and it gives you back their public posts as clean, structured data. For every post you get the title, subtitle, URL, publication date, author bylines, post type, cover image, the full article body as both HTML and plain text, and the public engagement counts that Substack shows on the page: likes, comments and restacks. It reads the same public JSON that a Substack page loads in your browser, so there is no login, no account and no browser automation involved. Custom domains such as www.lennysnewsletter.com work, plain something.substack.com addresses work, and so does a bare handle like lenny. Results land in a standard Apify dataset that you can download as JSON, CSV or Excel, or pull straight from the API.

Who it is for

Media monitoring and research across many newsletters. Point the Actor at the fifty newsletters that matter in your field, schedule it daily, and you have a searchable archive of everything they published, with dates and authors attached. Analysts and communications teams use this to see when a topic starts moving and who moved it first.

Writers and analysts tracking a beat. Pull the whole back catalogue of two or three publications you follow, then search the plain text body offline instead of scrolling an archive page. The likes, commentCount and restacks fields let you sort by what actually landed with readers, which is a fast way to find the questions a beat keeps returning to.

Marketers looking for sponsorship or partnership targets. Export a shortlist of newsletters, sort by engagement, and see publishing frequency, typical post length and which authors write what. That is enough to size an audience and pick the three newsletters worth a real conversation, without opening fifty tabs.

What you get

One dataset row per post. Here is a real row from a test run, with the body fields shortened so the example stays readable:

{
  "rowType": "post",
  "publication": "newsletter.pragmaticengineer.com",
  "publicationUrl": "https://newsletter.pragmaticengineer.com",
  "publicationName": "The Pragmatic Engineer",
  "postId": 213733051,
  "slug": "the-pragmatic-engineer-five-years",
  "url": "https://newsletter.pragmaticengineer.com/p/the-pragmatic-engineer-five-years",
  "title": "The Pragmatic Engineer: Five years",
  "subtitle": "As the newsletter reaches its fifth birthday, we reflect on how the publication has changed, and what to expect.",
  "description": "As the newsletter reaches its fifth birthday, we reflect on how the publication has changed, and what to expect.",
  "type": "newsletter",
  "audience": "everyone",
  "paywalled": false,
  "authors": [
    { "name": "Gergely Orosz", "handle": "pragmaticengineer", "url": "https://substack.com/@pragmaticengineer" }
  ],
  "publishedAt": "2026-09-01T17:14:34.151Z",
  "updatedAt": "2026-09-01T17:27:08.159Z",
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/e6e5687b-9265-4270-ab3f-84ad0059e6b1_420x300.png",
  "bodyHtml": "<p><span>Wow, has it </span><em><span>already</span></em><span> been five years?...",
  "bodyText": "Wow, has it already been five years?! The newsletter hits a big milestone this week...",
  "bodyIsPreview": false,
  "bodyOmittedReason": null,
  "wordCount": 3207,
  "likes": 254,
  "commentCount": 21,
  "restacks": 5,
  "podcastUrl": null,
  "scrapedAt": "2026-09-07T22:56:05.587Z"
}

Field notes worth knowing before you build on this:

• audience is the publisher's own setting: everyone, only_free, only_paid or founding.
• paywalled is true for anything above everyone. bodyIsPreview tells you whether the body you received is the publicly readable preview rather than the whole article, and it is empty whenever no body was returned at all.
• bodyOmittedReason is normally empty. It is filled in on the rare occasion the body was left out on purpose, which is explained under Data policy below.
• podcastUrl is filled in for posts of type podcast and empty otherwise.
• authors is a list, because Substack posts often carry more than one byline.
• error on a post row means part of that post could not be fetched, so the row carries the listing fields and no body.
• A publication that cannot be read produces a single row with rowType: "error" and a plain explanation. Those rows are free.

Turning on Include public comments adds rows with rowType: "comment" carrying postId, commentId, parentCommentId, authorName, authorHandle, body, likes and publishedAt. Replies point at their parent through parentCommentId, so you can rebuild a thread.

How to use it

Paste your newsletters into Substack publications and press Start. Everything else has a sensible default.

Input	What it does
publications	The newsletters to scrape. Full URL, name.substack.com, or just the handle.
maxPostsPerPublication	How many posts per newsletter. 0 takes the whole archive. Default 50.
sort	new for the latest issues, top for the newsletter's most popular posts of all time.
postedAfter	Optional date. Older posts are skipped and never billed.
includeBody	On by default. Turn it off for a faster, cheaper run with metadata only.
includeComments	Off by default. Adds the public comment threads as extra rows.
postTypes	Keep only newsletter, podcast, thread or video posts.

Scheduling for monitoring. For a daily digest, set sort to new, set maxPostsPerPublication to something small like 10, and add a schedule in the Apify console. Set postedAfter to yesterday's date if you want a strict window and the smallest possible bill. Each run appends to its own dataset, so you can diff runs or push them into your own store.

Run it from the API. Replace the token and the Actor id with your own:

curl -X POST "https://api.apify.com/v2/acts/YOUR_USERNAME~substack-scraper/runs?token=YOUR_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{
    "publications": ["https://www.lennysnewsletter.com", "astralcodexten.substack.com"],
    "maxPostsPerPublication": 25,
    "sort": "new",
    "includeBody": true
  }'

Then fetch the results:

$curl "https://api.apify.com/v2/datasets/YOUR_DATASET_ID/items?format=csv&token=YOUR_TOKEN"

Use it from an AI assistant. Apify Actors are available over MCP, so an assistant that speaks the Model Context Protocol can call this Actor as a tool and read the dataset back. That makes questions like "what did these five newsletters publish about pricing this month" answerable without writing any glue code.

Pricing

You pay per post delivered. No monthly subscription, no platform usage charge on top, no minimum.

• Each post row is one billable event.
• Each comment row is one billable event, and comments are off by default.
• Error rows are free. If a publication cannot be read, you are not charged for it.
• Posts filtered out by postedAfter or postTypes are never fetched and never billed.

The practical effect is that a small test run costs a small amount, and you can predict a big run exactly: it is the number of publications times maxPostsPerPublication, capped by how many posts those newsletters have actually published.

Data policy and limits

This Actor reads only what a logged out visitor can see.

• Public data only. It calls Substack's public JSON endpoints and reads public pages. Every request is a fresh logged out request: no login, no cookies, no session, no authorization header and no attempt to work around any access control.
• Paywalled posts return the free preview only. When a publisher marks a post for paid subscribers, Substack serves the public the opening section. That is what you get, and the row is flagged with paywalled: true and bodyIsPreview: true so you never mistake a preview for a full article. If a subscriber-only post ever came back looking complete, the body is withheld entirely and bodyOmittedReason says so, rather than passing on text a subscriber paid for. The rest of the row still arrives, so you keep the title, date, author and engagement counts.
• No subscriber or member data. Subscriber lists, email addresses and anything else behind a publisher's account are out of scope and are never collected.
• Substack publications only. Before anything is scraped or billed, the address has to prove it is a real Substack: the page has to carry Substack's own markers and the archive has to return Substack post records that belong to that publication. A site that has left Substack, or never was one, gets a free error row instead of scraped content. Private and internal addresses are refused outright.
• Comments are the public ones. Only comments visible to a logged out reader are returned, and deleted comments are skipped.
• What the endpoints will give. Substack serves the archive in pages of 23 posts, so very large archives take proportionally more requests. Addresses that do not exist produce a free error row explaining what happened. The same newsletter given twice, once as a custom domain and once as a substack.com address, is recognised as one publication and scraped once.
• Sensible ceilings. Up to 1,000 publications per run, up to 10,000 posts per publication, up to 5,000 comments per post. Nothing is ever delivered or billed twice.
• Be polite and lawful. Requests are rate limited by default. Newsletter text is the property of its authors: use it for research, monitoring and analysis, and respect copyright when you republish anything.

Support

Something wrong, missing or confusing? Open an issue on the Actor's Issues tab with the input you used and the publication that gave you trouble, and it will be looked at. Feature requests are welcome, especially if you can say what decision the extra field would help you make.