# Substack & Medium Scraper (`zovo-one/medium-substack-articles`) Actor

Medium and Substack articles by author, publication, tag or search. Full text of free posts, a preview plus a paywall flag for paid ones, Substack likes and comments, date filters, duplicate removal. $0.002 per article.

- **URL**: https://apify.com/zovo-one/medium-substack-articles.md
- **Developed by:** [Michael Lip](https://apify.com/zovo-one) (community)
- **Categories:** News, Social media
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$2.00 / 1,000 articles

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Medium and Substack Articles Scraper

Get Medium and Substack articles in one dataset, with the full text of free posts and a paywall flag on paid ones, for $0.002 per article.

Give it Medium authors, publications or tags, Substack publications or authors, or Substack search terms. Every row has the same fields on both platforms. Substack rows also carry likes, comments and restacks. Duplicates, posts outside your date range and paid posts you choose to skip aren't charged.

- Medium authors (`@netflixtechblog`), publications (`better-programming`, or one on its own domain like `netflixtechblog.com`) and tags (`python`).
- Substack publications by name, `name.substack.com` or custom domain (`www.lennysnewsletter.com`). Older posts are read through the publication's sitemap, so you can go past the archive's first page.
- Substack authors (`@lenny`), read from their own publication and filtered by byline.
- Substack site-wide search, page by page.
- Date filters, duplicate removal across sources, and an option to leave paywalled posts out.

Press Start with an empty form and you get up to 10 posts each from two example sources, `netflix-techblog` on Medium and `astralcodexten` on Substack. That run costs at most $0.04.

### What you get

One row per article.

| Field | Example | Notes |
|---|---|---|
| `platform` | `substack` | `medium` or `substack` |
| `id` | `222262321` | Post ID on the platform |
| `url` | `https://www.astralcodexten.com/p/links-for-november-2026` | Tracking parameters removed |
| `title`, `subtitle` | `Links For November 2026` | `subtitle` is Substack only |
| `author`, `authorHandle` | `Scott Alexander`, `astralcodexten` | Several bylines are comma-separated |
| `publication` | `Astral Codex Ten` | When known |
| `publishedAt`, `updatedAt` | `2026-11-05T02:01:33.734Z` | ISO 8601, UTC |
| `tags` | `[]` | Medium categories or Substack post tags |
| `isPaywalled` | `false` | `true` = paid or member-only, `null` = unknown (Medium tag feeds) |
| `contentStatus` | `full` | `full`, `preview` or `none` |
| `contentText` | `[I haven’t independently verified each link...` | Whole text, free posts only |
| `contentHtml` | `null` | Whole HTML, only with "Include HTML" on |
| `previewText` | `null` | Snippet or free preview when the whole text isn't saved |
| `wordCount` | `3370` | |
| `likes`, `comments`, `restacks` | `128`, `173`, `12` | Substack only, `null` on Medium |
| `coverImage` | `https://substack-post-media.s3.amazonaws.com/...` | |
| `language`, `postType` | `en`, `newsletter` | `language` is `null` on Medium |
| `sourceType`, `sourceInput` | `substack-publication`, `astralcodexten` | Which input found the row |
| `sourceId` | `substack:publication:astralcodexten.substack.com` | Internal source key |
| `scrapedAt` | `2026-11-05T14:21:34.509Z` | |

The dataset has two views. Articles is the overview table, and Full text puts the text next to the title and link. A run report per source (saved, full text, previews, duplicates, errors) goes to the `OUTPUT` record in the key-value store.

### Example output

This row comes from a test run on Apify on 2026-11-05 with the empty input. Only `contentText` is shortened here.

```json
{
  "platform": "medium",
  "id": "6b0811a1c015",
  "url": "https://netflixtechblog.com/making-failure-diagnosis-reliable-with-pensive-ai-6b0811a1c015",
  "title": "Making Failure Diagnosis Reliable with Pensive AI",
  "subtitle": null,
  "author": "Netflix Technology Blog",
  "authorHandle": null,
  "publication": "Netflix TechBlog",
  "publishedAt": "2026-10-26T17:04:57.000Z",
  "updatedAt": "2026-10-26T17:04:58.324Z",
  "tags": ["pensive", "data-engineering", "maestros", "orchestration", "data-science"],
  "language": null,
  "postType": "story",
  "isPaywalled": false,
  "contentStatus": "full",
  "contentText": "By Praneeth Yenugutala, Victor Dusa, Anastasia Samoilova, Megan Ren, Justin Wei, Marcelo Mayworm\n\nAt Netflix, data and machine learning practitioners run millio...",
  "contentHtml": null,
  "previewText": null,
  "wordCount": 3531,
  "likes": null,
  "comments": null,
  "restacks": null,
  "coverImage": "https://cdn-images-1.medium.com/max/1024/1*2TSVpCdPYj4fMMlFPuheCg.png",
  "sourceType": "medium-publication",
  "sourceInput": "netflix-techblog",
  "sourceId": "medium:publication:netflix-techblog",
  "scrapedAt": "2026-11-05T15:15:42.560Z"
}
```

### Pricing

It's pay per event. You pay for saved articles and nothing else the Actor does.

| Event | Price | When it's charged |
|---|---|---|
| `article` | $0.002 | Once per row saved to the dataset |

The price is the same for a free post with full text and a paid post with its preview. Here's what typical runs cost.

| Run | Articles | Cost |
|---|---|---|
| Empty form (two example sources, 10 each) | up to 20 | up to $0.04 |
| 5 Substack newsletters, last 50 posts each | 250 | $0.50 |
| 1,000 articles | 1,000 | $2.00 |
| 20 Medium tags, 10 stories each | 200 | $0.40 |

You aren't charged for duplicates, posts outside the date range, skipped paywalled posts or sources that fail. If the platform restarts a run, the rows saved before the restart aren't saved or charged again. Set a maximum cost per run in the run options and the Actor stops before it goes over. A maximum below the price of one article ends the run with 0 rows and no charge.

### Input

| Field | Default | What it does |
|---|---|---|
| Medium authors | none | Up to 50: `@user`, `https://medium.com/@user` or `user.medium.com` |
| Medium publications | none | Up to 50: a name, a `medium.com/<publication>` URL, or the publication's own domain (read over https) |
| Medium tags | none | Up to 50: `python`, `machine learning`, `https://medium.com/tag/startup` |
| Substack publications | none | Up to 50: a name, `name.substack.com`, a custom domain or any URL on it |
| Substack authors | none | Up to 50: `@handle` or `https://substack.com/@handle` |
| Substack search | none | Up to 20 terms (200 characters each), one per line. Commas stay part of the term |
| Max articles per source | 10 | 1 to 1,000 per author, publication, tag or search term |
| Published from / to | none | `YYYY-MM-DD` (whole UTC days) or relative, like `7 days`, `2 weeks`, `6 months` |
| Full text of free posts | on | Substack needs one extra request per post for the text. Off saves the metadata and Substack's short preview |
| Include HTML | off | Also save `contentHtml` for free posts |
| Skip paywalled posts | off | Leave paid and member-only posts out. Stories from Medium tag feeds are kept, see the FAQ |
| Remove duplicates | on | Save a post once per run, even when two sources return it |
| Proxy | none | Optional. See the FAQ |

If every source field is empty or left out (the input `{}`, say), the run uses the two example sources and logs a warning. A field you leave out counts as empty, so the examples never get added to an input that names its own sources.

In the source fields you can put several entries on one line, separated by commas. Search terms are one per line. Entries that can't be read (a Medium tag URL pasted into the authors field, say) are skipped with a message that says where they belong, and the rest of the run goes on. A plain word in the authors field, like `python`, is read as the author `@python`, so put tags in the tags field.

This input gets the last 30 days from two Substack newsletters and one Medium publication, without paid posts.

```json
{
  "mediumPublications": ["netflix-techblog"],
  "substackPublications": ["www.lennysnewsletter.com", "astralcodexten"],
  "maxArticlesPerSource": 50,
  "dateFrom": "30 days",
  "skipPaywalled": true
}
```

### Run it through the API

The synchronous endpoint starts a run and returns the dataset rows in one HTTP call. Replace `APIFY_TOKEN` with your token from Apify Console.

```bash
curl -s -X POST \
  "https://api.apify.com/v2/acts/zovo-one~medium-substack-articles/run-sync-get-dataset-items?token=APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"substackPublications": ["astralcodexten"], "maxArticlesPerSource": 5}'
```

A synchronous call waits at most 300 seconds for the run. If the run takes longer, the call ends without the rows, but the run keeps going. A few sources with the default 10 posts fit in that. For long back catalogues, start a normal run and read the dataset when it ends.

### Run it from Python

```python
from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("zovo-one/medium-substack-articles").call(run_input={
    "substackPublications": ["www.lennysnewsletter.com"],
    "dateFrom": "30 days",
    "skipPaywalled": True,
})
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["publishedAt"], item["title"], item["likes"])
```

### Run it from Node.js

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('zovo-one/medium-substack-articles').call({
    substackAuthors: ['@lenny'],
    maxArticlesPerSource: 25,
    skipPaywalled: true,
});
const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items.map((a) => `${a.publishedAt} ${a.title} (${a.likes} likes)`));
```

### Use it from AI agents through MCP

The Apify MCP server at `mcp.apify.com` can add this Actor as a tool for AI agents. Put the Actor in the `tools` parameter. Each MCP client sets up a remote server in its own way, and the configurator at https://mcp.apify.com/ gives the setup for yours. Clients that read an `mcpServers` config with a `url` key, such as Cursor, take the snippet below. VS Code uses a `servers` key with `"type": "http"` in `.vscode/mcp.json`, and Claude Desktop adds remote servers as connectors in its settings.

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=zovo-one/medium-substack-articles"
    }
  }
}
```

With the URL alone, your browser opens on the first connection so you can sign in to Apify. To use a token instead, add a header.

```json
{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=zovo-one/medium-substack-articles",
      "headers": { "Authorization": "Bearer APIFY_TOKEN" }
    }
  }
}
```

The Actor runs with limited permissions and charges per event, so the MCP server can call it. Then ask your agent something like "Get the last 30 days of posts from lennysnewsletter.com and summarize the free ones." The agent gets the same rows as the dataset, including `isPaywalled`, so it can tell a full post from a preview.

### Limits

Medium's public feeds hold the 10 newest stories of each author, publication or tag. That's all Medium serves without a login, so Medium sources return at most 10 articles each and there's no Medium keyword search. Medium's article pages, search and API sit behind a Cloudflare bot check, which is also why clap counts aren't available (`likes` is `null` on Medium).

Substack's archive API gives anonymous clients only its first page (about 20 to 25 newest posts). For older posts the Actor reads the publication's `sitemap.xml` and then each post. That works for the whole back catalogue, but it's one request per post, so big pulls take longer. Substack also rate-limits fast clients. The Actor spaces its requests per site and slows down by itself after a 429.

Substack search results come by relevance, not date. A date filter skips results outside the range but doesn't stop the search early. Each term reads at most 40 result pages.

A Substack author is read from their own primary publication. Guest posts they wrote for other publications aren't included. In a group publication the Actor checks each post's byline. After 300 posts in a row by other people it stops reading that publication's sitemap and logs a warning, so older posts by the author may be missed.

A "Published to" date far in the past is slow on a big Substack publication. The sitemap doesn't carry publication dates, so every newer post is fetched once and skipped.

A run sends at most 5,000 requests, counting the ones that don't produce a row. When it gets there the run stops, saves what it has and says so in `OUTPUT`. For bigger jobs, split the input over several runs.

The Actor only contacts public web addresses. It refuses IP addresses, and names that point to a private or local network. A Medium publication on its own domain has to serve Medium's own feed, and other sites are reported as an error.

### FAQ

#### Paid and member-only posts

The Actor never logs in and never gets around a paywall. It saves what each site shows to anyone.

On Substack, a post whose audience is anything other than `everyone` is paywalled. The row gets `isPaywalled: true` and `contentStatus: "preview"`. With "Full text of free posts" on, `previewText` holds the free part Substack shows above the paywall (cut at 2,000 characters). With it off, `previewText` holds Substack's short teaser. `contentText` and `contentHtml` stay empty.

On Medium, author and publication feeds include the whole body of free stories and only a snippet of member-only ones. So a story without a body there gets `isPaywalled: true` and the snippet. Tag feeds give a snippet for every story, so those rows have `isPaywalled: null` and `contentStatus: "preview"`. If you need the text of a tagged story, add its author or publication as a source.

Turn on "Skip paywalled posts" to leave paid posts out. They aren't saved or charged. Medium tag-feed rows have `isPaywalled: null`, so the skip keeps them and they're charged like other rows. On Substack, paid posts on the archive's first page (the newest 20 or so) aren't fetched at all. Older posts come from the sitemap, which lists only addresses, so the Actor fetches each one to learn whether it's paid and then drops it.

#### Why a Medium source stops at 10

Medium's public feed for an author, publication or tag holds 10 stories. A higher "Max articles per source" still applies to Substack sources, and the run log tells you Medium is capped.

#### When a site blocks the run

The Actor uses a direct connection by default. If a site refuses it on the Apify platform (403, 429 or a bot check page), the run switches once to Apify Proxy and tries that source again. You can also pick a proxy in the input. When every source is blocked and nothing was saved, the run fails with a message that says so, and you pay nothing per article.

#### When one source fails

A source that fails while running (a site that's down, say) is reported in `OUTPUT` and the run moves on to the next one. If 3 sources in a row fail on the same service (medium.com, Substack, or one custom domain), the remaining sources on that service are skipped and the other services still run.

#### Republishing the text

The Actor reads public feeds and public JSON endpoints, without a login, and it doesn't save paywalled text. Articles are still their authors' work. Check the copyright and terms that apply before you republish text, and use the data in line with Medium's and Substack's terms and the laws that apply to you.

[zovo-support-start]: #

### Support Zovo

Zovo builds and maintains this Actor. If it saves you time, you can support the work on [GitHub Sponsors](https://github.com/sponsors/theluckystrike). Sponsorship is optional, and the tool works the same without it.

Zovo is made by [theluckystrike](https://github.com/theluckystrike) on GitHub.

### More tools by Zovo

- [Website Screenshot API](https://apify.com/zovo-one/website-screenshot-pdf-api). Saves full-page screenshots and PDFs of public web pages.
- [YouTube Transcript Scraper](https://apify.com/zovo-one/tubetext-youtube-transcript-scraper). Gets YouTube transcripts and captions from videos, playlists and channels.
- [Google News Scraper](https://apify.com/zovo-one/google-news-scraper). Gets Google News results with the real publisher URLs.
- [Web Fetch for AI Agents](https://apify.com/zovo-one/web-fetch-fallback). Turns web pages into clean Markdown, text or HTML for AI agents.

[zovo-support-end]: #

# Changelog

This Actor's version history is a separate document: https://apify.com/zovo-one/medium-substack-articles/changelog.md

# Actor input Schema

## `mediumAuthors` (type: `array`):

Medium usernames or profile URLs, up to 50: @netflixtechblog, https://medium.com/@user or user.medium.com. Each returns the author's 10 newest stories, with full text for free ones. Leave every source field empty to try the Actor on two example sources (Medium netflix-techblog and Substack astralcodexten).

## `mediumPublications` (type: `array`):

Publication names or URLs, up to 50: netflix-techblog, https://medium.com/better-programming, or a publication on its own domain such as netflixtechblog.com (always read over https). 10 newest stories per publication. If every source field is empty, the run reads netflix-techblog and astralcodexten as examples.

## `mediumTags` (type: `array`):

Tags or tag URLs, up to 50: python, machine-learning, https://medium.com/tag/startup. Medium's tag feed has the 10 newest stories and only a short snippet of each (no full text, paywall state unknown). Add the author or publication as a source to get full text.

## `substackPublications` (type: `array`):

Up to 50: a name (astralcodexten), name.substack.com, a custom domain (www.lennysnewsletter.com) or any URL on it. Reads the newest posts first, then older ones through the publication's sitemap, so you can go back past the archive's first page.

## `substackAuthors` (type: `array`):

Profiles, up to 50: @lenny or https://substack.com/@lenny. Reads the author's own (primary) publication and keeps posts that list the author in the byline.

## `substackSearch` (type: `array`):

Search terms, up to 20 (200 characters each), one per line. Commas stay part of the term. Uses Substack's site-wide search; results come by relevance, not date.

## `maxArticlesPerSource` (type: `integer`):

Per author, publication, tag or search term (1-1,000). Each saved article costs $0.002, so 10 articles from each of 2 sources cost $0.04. Medium sources stop at 10 because Medium's feeds hold 10 stories.

## `dateFrom` (type: `string`):

Only posts published on or after this date: YYYY-MM-DD (start of the day, UTC) or relative, such as 7 days, 2 weeks, 6 months. Empty = no limit.

## `dateTo` (type: `string`):

Only posts published on or before this date: YYYY-MM-DD (end of the day, UTC) or relative, such as 1 day. Empty = up to now.

## `includeFullText` (type: `boolean`):

Substack: fetch each post to get its text (one extra request per post, not billed). Off = title, metadata and the short preview Substack lists. Posts older than the archive's first page still take one request each, because the sitemap lists only their address. Medium author and publication feeds always include the text of free stories.

## `includeHtml` (type: `boolean`):

Also save the article HTML (contentHtml) next to the plain text. Only for free posts with full text.

## `skipPaywalled` (type: `boolean`):

Leave out posts that need a paid subscription (Substack paid posts, Medium member-only stories). Skipped posts are not saved and not charged. Medium tag feeds don't say which stories are member-only (isPaywalled = null), so those stories are kept. Off = save them with the free preview in previewText and isPaywalled = true. The Actor never logs in and never saves paywalled text.

## `dedupe` (type: `boolean`):

Save each post once per run, even when several sources return it (for example an author and their publication). Skipped duplicates are not charged.

## `proxyConfiguration` (type: `object`):

Optional. The default is a direct connection. On the Apify platform, if a site blocks the direct connection (403, 429 or a bot check), the run switches once to Apify Proxy by itself.

## Actor input object example

```json
{
  "maxArticlesPerSource": 10,
  "includeFullText": true,
  "includeHtml": false,
  "skipPaywalled": false,
  "dedupe": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}
```

# Actor output Schema

## `articles` (type: `string`):

No description

## `fullText` (type: `string`):

No description

## `allFields` (type: `string`):

No description

## `runReport` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "maxArticlesPerSource": 10,
    "includeFullText": true,
    "includeHtml": false,
    "skipPaywalled": false,
    "dedupe": true,
    "proxyConfiguration": {
        "useApifyProxy": false
    }
};

// Run the Actor and wait for it to finish
const run = await client.actor("zovo-one/medium-substack-articles").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "maxArticlesPerSource": 10,
    "includeFullText": True,
    "includeHtml": False,
    "skipPaywalled": False,
    "dedupe": True,
    "proxyConfiguration": { "useApifyProxy": False },
}

# Run the Actor and wait for it to finish
run = client.actor("zovo-one/medium-substack-articles").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "maxArticlesPerSource": 10,
  "includeFullText": true,
  "includeHtml": false,
  "skipPaywalled": false,
  "dedupe": true,
  "proxyConfiguration": {
    "useApifyProxy": false
  }
}' |
apify call zovo-one/medium-substack-articles --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,zovo-one/medium-substack-articles"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/yL8FzuL4haqJJJpn3/builds/lopOYflqU8mNRavsC/openapi.json