Medium and Substack Articles Scraper

Get Medium and Substack articles in one dataset, with the full text of free posts and a paywall flag on paid ones, for $0.002 per article.

Give it Medium authors, publications or tags, Substack publications or authors, or Substack search terms. Every row has the same fields on both platforms. Substack rows also carry likes, comments and restacks. Duplicates, posts outside your date range and paid posts you choose to skip aren't charged.

• Medium authors (@netflixtechblog), publications (better-programming, or one on its own domain like netflixtechblog.com) and tags (python).
• Substack publications by name, name.substack.com or custom domain (www.lennysnewsletter.com). Older posts are read through the publication's sitemap, so you can go past the archive's first page.
• Substack authors (@lenny), read from their own publication and filtered by byline.
• Substack site-wide search, page by page.
• Date filters, duplicate removal across sources, and an option to leave paywalled posts out.

Press Start with an empty form and you get up to 10 posts each from two example sources, netflix-techblog on Medium and astralcodexten on Substack. That run costs at most $0.04.

What you get

One row per article.

Field	Example	Notes
platform	substack	medium or substack
id	222262321	Post ID on the platform
url	https://www.astralcodexten.com/p/links-for-november-2026	Tracking parameters removed
title, subtitle	Links For November 2026	subtitle is Substack only
author, authorHandle	Scott Alexander, astralcodexten	Several bylines are comma-separated
publication	Astral Codex Ten	When known
publishedAt, updatedAt	2026-11-05T02:01:33.734Z	ISO 8601, UTC
tags	[]	Medium categories or Substack post tags
isPaywalled	false	true = paid or member-only, null = unknown (Medium tag feeds)
contentStatus	full	full, preview or none
contentText	[I haven’t independently verified each link...	Whole text, free posts only
contentHtml	null	Whole HTML, only with "Include HTML" on
previewText	null	Snippet or free preview when the whole text isn't saved
wordCount	3370
likes, comments, restacks	128, 173, 12	Substack only, null on Medium
coverImage	https://substack-post-media.s3.amazonaws.com/...
language, postType	en, newsletter	language is null on Medium
sourceType, sourceInput	substack-publication, astralcodexten	Which input found the row
sourceId	substack:publication:astralcodexten.substack.com	Internal source key
scrapedAt	2026-11-05T14:21:34.509Z

The dataset has two views. Articles is the overview table, and Full text puts the text next to the title and link. A run report per source (saved, full text, previews, duplicates, errors) goes to the OUTPUT record in the key-value store.

Example output

This row comes from a test run on Apify on 2026-11-05 with the empty input. Only contentText is shortened here.

{
  "platform": "medium",
  "id": "6b0811a1c015",
  "url": "https://netflixtechblog.com/making-failure-diagnosis-reliable-with-pensive-ai-6b0811a1c015",
  "title": "Making Failure Diagnosis Reliable with Pensive AI",
  "subtitle": null,
  "author": "Netflix Technology Blog",
  "authorHandle": null,
  "publication": "Netflix TechBlog",
  "publishedAt": "2026-10-26T17:04:57.000Z",
  "updatedAt": "2026-10-26T17:04:58.324Z",
  "tags": ["pensive", "data-engineering", "maestros", "orchestration", "data-science"],
  "language": null,
  "postType": "story",
  "isPaywalled": false,
  "contentStatus": "full",
  "contentText": "By Praneeth Yenugutala, Victor Dusa, Anastasia Samoilova, Megan Ren, Justin Wei, Marcelo Mayworm\n\nAt Netflix, data and machine learning practitioners run millio...",
  "contentHtml": null,
  "previewText": null,
  "wordCount": 3531,
  "likes": null,
  "comments": null,
  "restacks": null,
  "coverImage": "https://cdn-images-1.medium.com/max/1024/1*2TSVpCdPYj4fMMlFPuheCg.png",
  "sourceType": "medium-publication",
  "sourceInput": "netflix-techblog",
  "sourceId": "medium:publication:netflix-techblog",
  "scrapedAt": "2026-11-05T15:15:42.560Z"
}

Pricing

It's pay per event. You pay for saved articles and nothing else the Actor does.

Event	Price	When it's charged
article	$0.002	Once per row saved to the dataset

The price is the same for a free post with full text and a paid post with its preview. Here's what typical runs cost.

Run	Articles	Cost
Empty form (two example sources, 10 each)	up to 20	up to $0.04
5 Substack newsletters, last 50 posts each	250	$0.50
1,000 articles	1,000	$2.00
20 Medium tags, 10 stories each	200	$0.40

You aren't charged for duplicates, posts outside the date range, skipped paywalled posts or sources that fail. If the platform restarts a run, the rows saved before the restart aren't saved or charged again. Set a maximum cost per run in the run options and the Actor stops before it goes over. A maximum below the price of one article ends the run with 0 rows and no charge.

Input

Field	Default	What it does
Medium authors	none	Up to 50: @user, https://medium.com/@user or user.medium.com
Medium publications	none	Up to 50: a name, a medium.com/<publication> URL, or the publication's own domain (read over https)
Medium tags	none	Up to 50: python, machine learning, https://medium.com/tag/startup
Substack publications	none	Up to 50: a name, name.substack.com, a custom domain or any URL on it
Substack authors	none	Up to 50: @handle or https://substack.com/@handle
Substack search	none	Up to 20 terms (200 characters each), one per line. Commas stay part of the term
Max articles per source	10	1 to 1,000 per author, publication, tag or search term
Published from / to	none	YYYY-MM-DD (whole UTC days) or relative, like 7 days, 2 weeks, 6 months
Full text of free posts	on	Substack needs one extra request per post for the text. Off saves the metadata and Substack's short preview
Include HTML	off	Also save contentHtml for free posts
Skip paywalled posts	off	Leave paid and member-only posts out. Stories from Medium tag feeds are kept, see the FAQ
Remove duplicates	on	Save a post once per run, even when two sources return it
Proxy	none	Optional. See the FAQ

If every source field is empty or left out (the input {}, say), the run uses the two example sources and logs a warning. A field you leave out counts as empty, so the examples never get added to an input that names its own sources.

In the source fields you can put several entries on one line, separated by commas. Search terms are one per line. Entries that can't be read (a Medium tag URL pasted into the authors field, say) are skipped with a message that says where they belong, and the rest of the run goes on. A plain word in the authors field, like python, is read as the author @python, so put tags in the tags field.

This input gets the last 30 days from two Substack newsletters and one Medium publication, without paid posts.

{
  "mediumPublications": ["netflix-techblog"],
  "substackPublications": ["www.lennysnewsletter.com", "astralcodexten"],
  "maxArticlesPerSource": 50,
  "dateFrom": "30 days",
  "skipPaywalled": true
}

Run it through the API

The synchronous endpoint starts a run and returns the dataset rows in one HTTP call. Replace APIFY_TOKEN with your token from Apify Console.

curl -s -X POST \
  "https://api.apify.com/v2/acts/zovo-one~medium-substack-articles/run-sync-get-dataset-items?token=APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"substackPublications": ["astralcodexten"], "maxArticlesPerSource": 5}'

A synchronous call waits at most 300 seconds for the run. If the run takes longer, the call ends without the rows, but the run keeps going. A few sources with the default 10 posts fit in that. For long back catalogues, start a normal run and read the dataset when it ends.

Run it from Python

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("zovo-one/medium-substack-articles").call(run_input={
    "substackPublications": ["www.lennysnewsletter.com"],
    "dateFrom": "30 days",
    "skipPaywalled": True,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["publishedAt"], item["title"], item["likes"])

Run it from Node.js

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('zovo-one/medium-substack-articles').call({
    substackAuthors: ['@lenny'],
    maxArticlesPerSource: 25,
    skipPaywalled: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((a) => `${a.publishedAt} ${a.title} (${a.likes} likes)`));

Use it from AI agents through MCP

The Apify MCP server at mcp.apify.com can add this Actor as a tool for AI agents. Put the Actor in the tools parameter. Each MCP client sets up a remote server in its own way, and the configurator at https://mcp.apify.com/ gives the setup for yours. Clients that read an mcpServers config with a url key, such as Cursor, take the snippet below. VS Code uses a servers key with "type": "http" in .vscode/mcp.json, and Claude Desktop adds remote servers as connectors in its settings.

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=zovo-one/medium-substack-articles"
    }
  }
}

With the URL alone, your browser opens on the first connection so you can sign in to Apify. To use a token instead, add a header.

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=zovo-one/medium-substack-articles",
      "headers": { "Authorization": "Bearer APIFY_TOKEN" }
    }
  }
}

The Actor runs with limited permissions and charges per event, so the MCP server can call it. Then ask your agent something like "Get the last 30 days of posts from lennysnewsletter.com and summarize the free ones." The agent gets the same rows as the dataset, including isPaywalled, so it can tell a full post from a preview.

Limits

Medium's public feeds hold the 10 newest stories of each author, publication or tag. That's all Medium serves without a login, so Medium sources return at most 10 articles each and there's no Medium keyword search. Medium's article pages, search and API sit behind a Cloudflare bot check, which is also why clap counts aren't available (likes is null on Medium).

Substack's archive API gives anonymous clients only its first page (about 20 to 25 newest posts). For older posts the Actor reads the publication's sitemap.xml and then each post. That works for the whole back catalogue, but it's one request per post, so big pulls take longer. Substack also rate-limits fast clients. The Actor spaces its requests per site and slows down by itself after a 429.

Substack search results come by relevance, not date. A date filter skips results outside the range but doesn't stop the search early. Each term reads at most 40 result pages.

A Substack author is read from their own primary publication. Guest posts they wrote for other publications aren't included. In a group publication the Actor checks each post's byline. After 300 posts in a row by other people it stops reading that publication's sitemap and logs a warning, so older posts by the author may be missed.

A "Published to" date far in the past is slow on a big Substack publication. The sitemap doesn't carry publication dates, so every newer post is fetched once and skipped.

A run sends at most 5,000 requests, counting the ones that don't produce a row. When it gets there the run stops, saves what it has and says so in OUTPUT. For bigger jobs, split the input over several runs.

The Actor only contacts public web addresses. It refuses IP addresses, and names that point to a private or local network. A Medium publication on its own domain has to serve Medium's own feed, and other sites are reported as an error.

FAQ

Paid and member-only posts

The Actor never logs in and never gets around a paywall. It saves what each site shows to anyone.

On Substack, a post whose audience is anything other than everyone is paywalled. The row gets isPaywalled: true and contentStatus: "preview". With "Full text of free posts" on, previewText holds the free part Substack shows above the paywall (cut at 2,000 characters). With it off, previewText holds Substack's short teaser. contentText and contentHtml stay empty.

On Medium, author and publication feeds include the whole body of free stories and only a snippet of member-only ones. So a story without a body there gets isPaywalled: true and the snippet. Tag feeds give a snippet for every story, so those rows have isPaywalled: null and contentStatus: "preview". If you need the text of a tagged story, add its author or publication as a source.

Turn on "Skip paywalled posts" to leave paid posts out. They aren't saved or charged. Medium tag-feed rows have isPaywalled: null, so the skip keeps them and they're charged like other rows. On Substack, paid posts on the archive's first page (the newest 20 or so) aren't fetched at all. Older posts come from the sitemap, which lists only addresses, so the Actor fetches each one to learn whether it's paid and then drops it.

Why a Medium source stops at 10

Medium's public feed for an author, publication or tag holds 10 stories. A higher "Max articles per source" still applies to Substack sources, and the run log tells you Medium is capped.

When a site blocks the run

The Actor uses a direct connection by default. If a site refuses it on the Apify platform (403, 429 or a bot check page), the run switches once to Apify Proxy and tries that source again. You can also pick a proxy in the input. When every source is blocked and nothing was saved, the run fails with a message that says so, and you pay nothing per article.

When one source fails

A source that fails while running (a site that's down, say) is reported in OUTPUT and the run moves on to the next one. If 3 sources in a row fail on the same service (medium.com, Substack, or one custom domain), the remaining sources on that service are skipped and the other services still run.

Republishing the text

The Actor reads public feeds and public JSON endpoints, without a login, and it doesn't save paywalled text. Articles are still their authors' work. Check the copyright and terms that apply before you republish text, and use the data in line with Medium's and Substack's terms and the laws that apply to you.

Support Zovo

Zovo builds and maintains this Actor. If it saves you time, you can support the work on GitHub Sponsors. Sponsorship is optional, and the tool works the same without it.

Zovo is made by theluckystrike on GitHub.

More tools by Zovo

• Website Screenshot API. Saves full-page screenshots and PDFs of public web pages.
• YouTube Transcript Scraper. Gets YouTube transcripts and captions from videos, playlists and channels.
• Google News Scraper. Gets Google News results with the real publisher URLs.
• Web Fetch for AI Agents. Turns web pages into clean Markdown, text or HTML for AI agents.