Substack Scraper: newsletter posts, full text as markdown, RAG chunks and rankings

Scrape Substack newsletters without an account or cookies. Give a publication (on substack.com or on its own domain), a post link, an author's @handle, a search term or a category, and get every post with title, subtitle, date, authors, likes, comments, restacks, word count, cover image, audience and a clear paywall flag. Turn on full text to get each public post as clean markdown, or ask for RAG chunks ready for embeddings.

Built for daily monitoring: schedule the same newsletters every morning and get only the posts published since the last run. A quiet day costs nothing.

At a glance

• Price per 1,000 posts, Free plan: US$ 0.80 (down to US$ 0.68 on Gold and above)
• Full post text as markdown (optional): US$ 1.067 per 1,000, on top; paid-post previews are free
• Publication rows (subscribers, prices, rank): US$ 1.067 per 1,000
• Fee per run start: None
• Publications on substack.com and on their own domains: Yes
• Post search and publication search across all of Substack: Yes
• Category rankings (bestsellers, rising, all): Yes
• Only new posts since the last run (onlyNew): Yes, free when nothing is new
• RAG chunks with section headings and overlap: Yes, same price as the text
• Paid content: Never; paid posts come with the public preview only, flagged isPaywalled
• Error rows (bad URL, unknown publication, no results): Free, with an errorCode

One real row

From a test run on 2026-10-21 (a Noahpinion post link, chunkSizeWords: 300), shortened; this post has 28 chunks:

{
  "type": "post",
  "id": 221088139,
  "title": "Can a Million AI Agents Create Superintelligence?",
  "subtitle": "Ramez Naam gives a skeptic’s take on agent swarms.",
  "url": "https://www.noahpinion.blog/p/can-a-million-ai-agents-create-superintelligence",
  "publishedAt": "2026-10-19T04:33:30Z",
  "postType": "newsletter",
  "audience": "everyone",
  "isPaywalled": false,
  "authors": [{ "name": "Ramez Naam", "handle": "rameznaam", "profileUrl": "https://substack.com/@rameznaam" }],
  "reactionCount": 241,
  "commentCount": 17,
  "restacks": 24,
  "wordCount": 7178,
  "coverImage": "https://substackcdn.com/image/fetch/...6335ee9e-3d13-4a47-9213-3c1d01cd8eff_1075x717.jpeg",
  "publicationName": "Noahpinion",
  "publicationUrl": "https://www.noahpinion.blog",
  "markdown": "![](https://substackcdn.com/image/fetch/...)\n\n*Art by GPT-6*\n\n*Ramez Naam is, in my opinion, one of the best futurists alive. But although [he has a Substack](https://www.rameznaam.com/) (which you should definitely follow), he doesn’t write nearly enough...*",
  "textWordCount": 7061,
  "textIsPreview": false,
  "chunks": [
    {
      "index": 0,
      "text": "*Art by GPT-6*\n\n*Ramez Naam is, in my opinion, one of the best futurists alive. But although [he has a Substack](https://www.rameznaam.com/) (which you should definitely follow), he doesn’t write nearly enough...*",
      "wordCount": 328,
      "section": null,
      "postId": 221088139,
      "postTitle": "Can a Million AI Agents Create Superintelligence?",
      "postUrl": "https://www.noahpinion.blog/p/can-a-million-ai-agents-create-superintelligence"
    }
  ]
}

And the publication row of the same newsletter (includePublicationDetails):

{
  "type": "publication",
  "id": 35345,
  "name": "Noahpinion",
  "url": "https://www.noahpinion.blog",
  "subdomain": "noahpinion",
  "authorName": "Noah Smith",
  "subscriberCount": 460000,
  "subscriberCountText": "Over 460,000 subscribers",
  "paidSubscribersText": "Tens of thousands of paid subscribers",
  "hasPaidSubscriptions": true,
  "monthlyPrice": 10.0,
  "yearlyPrice": 100.0,
  "priceCurrency": "USD",
  "hasPodcast": true
}

What you can do with it

• AI and RAG: feed the newsletters you trust into a vector database. Full text in markdown, chunks with section headings and overlap, every chunk linked back to its post.
• Newsletter monitoring: follow competitors, analysts or your industry's top writers every morning and send only the new posts to Slack, e-mail, Notion or Google Sheets (onlyNew).
• Market research and sponsorships: find newsletters on a topic with subscriber counts and prices, or pull a category's bestseller and rising rankings to pick where to advertise or whom to pitch.
• Content and trend analysis: which posts get the most likes, comments and restacks, how often a publication writes, how much of it is paywalled.
• Media and PR: search all of Substack for posts that mention your brand, product or topic.

Input

Field	What it does
startUrls	Publications (https://www.noahpinion.blog, https://name.substack.com, oneusefulthing), post links (…/p/post-name, substack.com/@handle/p-123, open.substack.com/pub/name/p/post), or @handle
searchPosts	Terms searched across every Substack post (AI agents, interest rates)
searchPublications	Terms searched across Substack publications (climate, venture capital)
categories	Substack categories (Technology, Business, Finance, Culture, U.S. Politics, Science...)
categoryRanking	bestsellers (default), rising or all
maxPostsPerPublication	Posts per publication, default 20; 0 = the whole archive (up to 10,000)
maxResultsPerSearch	Results per search or category, default 20
postsPerFoundPublication	Latest posts of every publication found by a search or category, default 0
sortBy	newest (default) or top (the publication's most popular posts)
includePublicationDetails	One publication row per publication in startUrls
includeFullText	Full text of each public post as markdown
chunkSizeWords, chunkOverlapWords	RAG chunks (turns on the full text); 0 = off
onlyNew	Deliver only posts not delivered in earlier runs
proxyConfiguration	Default Apify Proxy

Example: new posts of three AI newsletters every morning, with markdown and chunks for a RAG pipeline:

{
  "startUrls": ["https://www.oneusefulthing.org", "https://www.noahpinion.blog", "https://www.astralcodexten.com"],
  "maxPostsPerPublication": 10,
  "chunkSizeWords": 300,
  "onlyNew": true
}

Or the 50 bestsellers in Technology with their three latest posts:

{
  "categories": ["Technology"],
  "categoryRanking": "bestsellers",
  "maxResultsPerSearch": 50,
  "postsPerFoundPublication": 3
}

How daily monitoring works

With onlyNew the Actor remembers what it delivered for each publication (by Substack's own publication id, so noahpinion, @noahpinion and https://www.noahpinion.blog share one memory), each search and each category, in a named key-value store in your own account, substack-scraper-state.

• The first run delivers up to the limit and marks everything it read as seen.
• Later runs read each publication newest first and stop when they reach posts already delivered. A day with nothing new delivers nothing and costs nothing.
• Searches and categories deliver only results not delivered before.
• Delete the key-value store to start over.

Paywalls

The Actor reads only what Substack shows to anyone without an account. Paid posts (audience: "only_paid" or "founding") come with isPaywalled: true and every field of the post; with full text on, you get the public preview that Substack shows above the paywall, marked textIsPreview: true, and the text event is not charged. Nothing behind the paywall is read, and no login is used.

Output

Every post row: id, title, subtitle, url, slug, publishedAt, updatedAt, postType (newsletter, podcast, thread, video), audience (everyone, only_free, only_paid, founding), isPaywalled, authors (name, handle, profile link), authorNames, reactionCount, reactions, commentCount, restacks, wordCount, coverImage, previewText, language, sectionName, tags, podcastDurationSeconds, publicationId, publicationName, publicationUrl, position, source (the input it came from), scrapedAt.

With full text: markdown, textWordCount, textIsPreview, and chunks (index, text, wordCount, section, postId, postTitle, postUrl) when chunking is on. If a text cannot be read, the post still comes with textError and only the post is charged.

Every publication row: id, name, url, subdomain, customDomain, description, authorName, authorHandle, authorProfileUrl, logo, language, publicationType, createdAt, firstPostAt, subscriberCount (when the publication shows it), subscriberCountText, paidSubscribersText, bestsellerTier, hasPaidSubscriptions, monthlyPrice, yearlyPrice, foundingMemberPrice, priceCurrency, hasPodcast, podcastTitle, category, categoryRanking, position (rank in the search or category), source, scrapedAt.

Dataset views: Posts, Full text, Publications and RAG chunks (one row per chunk).

People data is limited to the author's public name and handle, which are the byline of the post. No e-mails, photos, bios or social accounts.

Error codes (never charged)

errorCode	Meaning
INVALID_INPUT	A value the Actor does not understand (unknown category, a substack.com page that is not a publication, post or profile, a bad option)
NOT_FOUND	No Substack publication at this address or name (including sites that left Substack), or the post was removed
NO_RESULTS	A search or category with no results, or a publication with no public posts
PUBLICATION_INFO_UNAVAILABLE	The publication details could not be read; its posts still come
BLOCKED	Substack refused every attempt; run again
NOT_REACHED	The run timeout arrived before this input
TIMEOUT_TOO_SHORT	The run timeout is too short to read even one page; raise it
UNEXPECTED	A page came in a shape we did not expect; the rest of the run goes on

Pricing

Event	Per 1,000
Post delivered	US$ 0.80
Full post text (public posts only, with includeFullText or chunks)	US$ 1.067
Publication delivered (searches, categories, includePublicationDetails)	US$ 1.067

A daily onlyNew run on 20 newsletters that finds 15 new posts costs US$ 0.012; with full text, about US$ 0.03. The first run of a 500-post archive with full text costs about US$ 0.93. Error rows, duplicates, posts already delivered and paid-post previews are free, and there is no start fee.

Limitations

• Paid content is never read: paid posts come with the public preview only.
• Subscriber counts appear only when the publication shows them on Substack; otherwise subscriberCount is empty and subscriberCountText or paidSubscribersText may still carry Substack's rounded label.
• Substack's post search returns the posts it finds most relevant, not every post that mentions the term.
• Comment texts are not collected, only counts.
• A site that moved off Substack (to its own platform) is reported as NOT_FOUND, not filled with its old archive.

FAQ

Do I need a Substack account? No. No login and no cookies; the default proxy works.

Does it work with custom domains? Yes. https://www.noahpinion.blog, noahpinion.blog, NOAHPINION.BLOG, https://noahpinion.substack.com, @noahpinion and noahpinion all reach the same publication.

Can I get the newsletter as text for ChatGPT, Claude or my own RAG app? Yes. Turn on includeFullText for markdown, or set chunkSizeWords (300 is a good start) for chunks with section headings, ready for embeddings.

Can I use it from n8n, Make, Zapier or an AI agent? Yes, through the Apify app or the Apify MCP server. Error rows are free, so an agent can explore cheaply.

What if a run is cut by its timeout? The Actor stops shortly before the limit and ends successfully with what it delivered, telling you in the status message how to get the rest.

Changelog

• 0.1 (2026-10-07): first version. Publications by URL, custom domain, @handle or name; post links; post search; publication search; category rankings; full text as markdown; RAG chunks; paywall flag with free previews; onlyNew for daily schedules; free error rows.