Substack Scraper: Newsletter Posts & Archives

Substack Scraper reads the public feed of any Substack publication and returns one structured record per post, title, subtitle, author, publish date, the full body as clean text, word count, cover image, podcast audio and whether the post is a paywalled preview.

Every Substack publication serves RSS at /feed, and Substack puts the entire post body in it. This Actor reads that feed directly: no proxy setup, no browser, no credentials to manage. Cloudflare sits in front of Substack but does not challenge feed requests, a plain HTTP GET with an ordinary User-Agent returns 200 every time.

Agent skill: SKILL.md

https://api.apify.com/v2/key-value-stores/t7YoTxpZEJOWvw4Ug/records/substack-newsletter-scraper.md

What you get

Output field	Meaning
publication	The publication's slug, or its custom domain if it has moved off substack.com
feedUrl, feedTitle	Feed that was actually read after redirects, and the publication's display name
postId	Substack's guid for the post, stable, and equal to the canonical post URL
title, subtitle	Post headline and the one-line subtitle Substack shows beneath it
author	Byline from dc:creator
publishedAt	Publish time, always ISO-8601
link	Canonical post URL
contentText	Full post body as clean plain text, HTML stripped, entities decoded
contentHtml	Substack's original markup, when you ask to keep it
isPaywalled	true when the feed carries only a subscriber preview of the post
wordCount	Words in contentText, a quick way to spot previews and short notes
imageUrl	Cover image from the post's image enclosure, or the first image in the body
audioUrl	MP3 enclosure, present on podcast and voiceover posts
scrapedAt	Run timestamp

A RUN_SUMMARY record in the key-value store holds per-run counts, a per-publication breakdown including paywalled counts, the filters used, and any publication that failed.

Common use cases

• Track thought leaders. Follow a shortlist of writers and get every new post as structured data.
• Competitive newsletter monitoring. Watch what rival publications ship, how often, and how long.
• Research corpora. Build a text dataset of long-form writing for analysis or retrieval.
• Newsroom and PR. Spot stories breaking in newsletters before they reach the trade press.
• Content recommendation. Index posts by author, subtitle and body for a discovery feed.

Quick start

Two publications, everything their feeds hold:

{
 "publications": ["astralcodexten", "newsletter.pragmaticengineer.com"]
}

Mixed input, slugs, subdomains, post links and custom domains all resolve to the same feed:

{
 "publications": [
 "noahpinion",
 "bariweiss.substack.com",
 "https://www.thefp.com",
 "https://newsletter.pragmaticengineer.com/p/formal-methods-with-hillel-wayne"
 ],
 "maxPostsPerPublication": 10
}

A cheap scheduled sweep that only picks up what is new:

{
 "publications": ["oneusefulthing", "construction-physics"],
 "sinceDate": "2026-08-01",
 "includeContent": false
}

Input

Field	Type	Default	Notes
publications	array	-	Required. Slugs, {slug}.substack.com URLs, post links or custom domains. Mixed input is fine.
maxPostsPerPublication	integer	0	Cap saved posts per publication, applied after the date filter. 0 = everything the feed carries.
sinceDate	string	""	Keep only posts published after this date. 2026-01-01 or a full ISO-8601 timestamp.
includeContent	boolean	true	Include the post body in contentText. Off gives a far smaller dataset.
stripHtmlTags	boolean	true	On, bodies are plain text only. Off additionally keeps Substack's markup in contentHtml.

Because the feed holds 20 posts, maxPostsPerPublication above 20 has no effect.

Output example

{
 "publication": "newsletter.pragmaticengineer.com",
 "feedUrl": "https://newsletter.pragmaticengineer.com/feed",
 "feedTitle": "The Pragmatic Engineer",
 "postId": "https://newsletter.pragmaticengineer.com/p/formal-methods-with-hillel-wayne",
 "title": "Formal methods with Hillel Wayne",
 "subtitle": "Hillel Wayne explains why formal methods like TLA+ matter, how they help build reliable software, and whether AI will finally bring formal verification into the mainstream",
 "author": "Gergely Orosz",
 "publishedAt": "2026-07-29T16:22:31.000Z",
 "link": "https://newsletter.pragmaticengineer.com/p/formal-methods-with-hillel-wayne",
 "contentText": "Stream the latest episode\n\nListen and watch now on YouTube, Apple and Spotify …",
 "contentHtml": null,
 "isPaywalled": false,
 "wordCount": 1643,
 "imageUrl": "https://substackcdn.com/image/fetch/…/substack-post-media.s3.amazonaws.com/public/images/…jpeg",
 "audioUrl": "https://api.substack.com/feed/podcast/208490793/39e73f549d0e5b122bbc9dfe29a9b11a.mp3",
 "scrapedAt": "2026-08-06T11:47:48.866Z"
}

Finding a publication slug

Open the publication and look at the address bar:

URL you see	What to pass
astralcodexten.substack.com	astralcodexten
astralcodexten.substack.com/p/some-post	astralcodexten
newsletter.pragmaticengineer.com	newsletter.pragmaticengineer.com
www.thefp.com/p/some-post	www.thefp.com

You can paste any of them verbatim, the Actor reduces a URL to its origin and appends /feed itself. Many publications have moved from {slug}.substack.com to their own domain; the old subdomain still 301s to the new one, and the redirect is followed. If two of your inputs land on the same feed after redirects, the second is skipped rather than scraped twice.

API example

curl -X POST "https://api.apify.com/v2/acts/arman-bd~substack-newsletter-scraper/run-sync-get-dataset-items?token=YOUR_TOKEN" \
 -H "Content-Type: application/json" \
 -d '{
 "publications": ["astralcodexten", "noahpinion"],
 "maxPostsPerPublication": 5,
 "includeContent": false
 }'

JavaScript example

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: 'YOUR_TOKEN' });
const run = await client.actor('arman-bd/substack-newsletter-scraper').call({
 publications: ['astralcodexten', 'newsletter.pragmaticengineer.com'],
 sinceDate: '2026-08-01',
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const post of items) console.log(`${post.publishedAt}, ${post.title} (${post.wordCount} words)`);

Limits and behaviour

• This is not the full archive. Substack's /feed returns exactly the 20 most recent posts, and there is no pagination parameter on it. Every publication tested, large and small, returned 20. To go deeper you need the publication's sitemap or archive pages, which this Actor deliberately does not touch. Run it on a schedule and you build the archive forward from today.
• Paywalled posts come back as previews. Substack truncates subscriber-only posts in the feed and closes them with a "Read more" link. The Actor detects that and sets isPaywalled: true; contentText then holds only the free preview. Nothing here bypasses a paywall.
• No Cloudflare problem. Substack is behind Cloudflare, but /feed is not challenged. Requests with an ordinary User-Agent, a bot-shaped User-Agent, and no User-Agent at all all returned HTTP 200 during testing.
• Custom domains are handled. {slug}.substack.com 301s to the publication's own domain; redirects are followed and the final URL is reported in feedUrl.
• One failure never kills the run. An unknown slug returns 404, is logged into RUN_SUMMARY.failures, and the run continues. The Actor only errors out if every publication fails.
• Transient errors are retried. 429 and 5xx get three attempts with backoff, honouring Retry-After.
• One request per second between publications, with a 45-second timeout each.

FAQ

Do I need a Substack account? No. You supply no credentials.

Can I get posts older than the last 20? Not from this Actor. The feed simply does not contain them.

Does it include paid posts? It includes the free preview Substack publishes for them, flagged with isPaywalled: true. It cannot and does not retrieve subscriber-only text.

Why is subtitle null? Substack writes a literal . as the description when a post has no subtitle; that is normalised to null rather than passed through.

Why does publication show a domain rather than my slug? Because that publication has migrated to its own domain, and the record reflects where the content actually lives. feedUrl shows the same thing.

How many publications can I pass at once? There is no hard cap. Feeds are read sequentially at one per second, so hundreds per run is normal.

Can I integrate it with something else? Yes, Apify API, client libraries, webhooks, scheduled runs, dataset exports (JSON/CSV/Excel) or MCP. Output is structured JSON.