# Substack Archive Scraper | Public Posts & Clean Text (`apivault_labs/substack-archive-scraper`) Actor

Export public Substack newsletter archives and post URLs with dates, authors, engagement and optional public text. Custom domains, filters, deduplication and continuation. Paid posts return metadata and short preview only.

- **URL**: https://apify.com/apivault_labs/substack-archive-scraper.md
- **Developed by:** [Apivault Labs](https://apify.com/apivault_labs) (community)
- **Categories:** News, Social media
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.75 / 1,000 substack posts

This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Archive Scraper — Public Posts & Clean Text

![Substack Archive Scraper — generated newsletter archive illustration](https://raw.githubusercontent.com/apivault-labs/apivault-labs/779738e9a066d6231553a109889f02e2d628df0e/assets/substack-archive-scraper/readme-cover-v1.png)

Turn public newsletter archives into structured post data for content research, publication audits and editorial monitoring. Supply Substack publication URLs, custom domains or specific post URLs. Export titles, source dates, author names, available engagement and optional public HTML/clean text to CSV, Excel or JSON.

This is an independent tool. It does not log in, access subscriber accounts or unlock paid articles. A restricted post returns metadata and a short source preview only. Keyword discovery, comments, subscriber counts, audio downloads and immutable archive snapshots are outside this version.

### First export: five articles

```json
{"urls":["https://www.experimental-history.com"],"maxItems":5}
```

Defaults request up to five metadata rows, at most 20 archive pages. The limit is global across inputs. Real output depends on matching posts, source availability, your spending cap and run resources. Some publishers or regions can return403/429; an empty Dataset alone does not prove an empty archive.

### What it does

| Goal | Supported result |
|---|---|
| Audit a newsletter archive | Post IDs, canonical URLs, titles, source publication dates and audience flags. |
| Export a reading corpus | Public/free bodyHtml and bodyText if returned by the anonymous source. |
| Track a publication window | Inclusive start/end date filters plus available author/engagement fields. |
| Compare several newsletters | Up to20inputs, one global result limit and cross-input identity deduplication. |
| Resume a capped collection | CONTINUATION.resumeInput keeps source offsets and previously accepted IDs. |

### Input settings

| Setting | Default / limit | Meaning |
|---|---|---|
| urls | One sample publication;1–20 | Public HTTPS homepage, /archive, or /p/post-slug. |
| maxItems | 5; maximum10000 | Maximum useful post rows across all inputs. |
| includeContent | false | Return public body HTML/text when present; never paid-body content. |
| onlyFree | false | Skip restricted audience/unlock-required posts. |
| startDate / endDate | omitted | Inclusive source post_date filters, YYYY-MM-DD. |
| maxPages | 20; maximum5000 | Global cap on archive/post input pages; up to20posts per archive page. Requested public content can add one detail request per accepted free post. |
| cursor | omitted | Copy from continuation only. Keep URLs/content/date/free settings unchanged. |

### Five ready-to-copy scenarios

#### Public text corpus

```json
{"urls":["https://www.experimental-history.com"],"maxItems":5,"includeContent":true,"onlyFree":true}
```

#### Several publications with one limit

```json
{"urls":["https://www.experimental-history.com","https://www.lennysnewsletter.com"],"maxItems":10}
```

#### Publication date window

```json
{"urls":["https://www.experimental-history.com"],"startDate":"2026-09-01","endDate":"2026-10-21","maxItems":10,"maxPages":10}
```

#### Specific article URL

```json
{"urls":["https://www.experimental-history.com/p/there-is-no-epidemic-of-loneliness"],"maxItems":1,"includeContent":true}
```

#### Metadata from a custom domain

```json
{"urls":["https://www.lennysnewsletter.com/archive"],"maxItems":5,"includeContent":false}
```

Five corresponding saved Tasks are prepared for verification. Public one-click Task availability must be checked separately; private Tasks are not published templates.

### Output fields

| Field group | What it means |
|---|---|
| postId, publicationId, url | Source identity; publicationId/postId deduplicate overlapping inputs. |
| title, subtitle, authors, tags | Source editorial metadata; optional values can be absent. |
| publishedAt, updatedAt | Source timestamps; updatedAt is unknown when not supplied. |
| isPaid, audience, contentStatus | Access classification; paid_preview_only never contains bodyHtml/bodyText. |
| previewText | Short available preview, capped at1000characters. |
| bodyHtml, bodyText | Only requested public/free body, without an assertion that it is the complete copyrighted article. |
| reactionCount, commentCount, restackCount | Available source counters, not extracted comments. Missing values remain unknown. |
| sourceUrl, scrapedAt | Actual anonymous endpoint and observation time. |

No sample customer data or fabricated article output is inserted into the Dataset. wordCount is the source's article count when provided; it can exceed the available public text length. contentStatus distinguishes not_requested, public_body_returned, public_body_unavailable and paid_preview_only.

### Larger archives and continuation

Use asynchronous runs for larger exports. Increase maxItems and maxPages within the schema; allow enough time and memory. This is a finite, bounded exporter, not a promise of unlimited archive depth or a500Kexport. Public content can require an additional identity-checked detail request per accepted free post. The first source version reads pages sequentially and writes accepted rows as they are processed.

When capped by items/pages/budget, read CONTINUATION.resumeInput in the run's Key-Value Store and submit that object as the next Actor input. Previously accepted IDs in the cursor prevent repeated billing across an unchanged continuation chain up to its100000IDwindow. Source updates can move offsets: this is not an immutable snapshot and cannot guarantee no omissions during concurrent index changes. Also deduplicate at your destination. Independent runs begin a new collection.

Check SUMMARY.isComplete and ERRORS. COMPLETE means the requested source list was exhausted without recorded source errors, not that every historical Substack post exists in the source. ITEM_LIMIT_REACHED, PAGE_LIMIT_REACHED and CHARGE_LIMIT_REACHED retain continuation. A source failure preserves its current offset; mixed results can be useful but incomplete. A complete source outage fails the run with no fabricated paid rows.

### Pricing and run resources

Candidate result tariff: **$1.00 per1,000 post rows on Free; $0.75 per1,000 on all paid tiers**. Metadata and requested public body use the same post event; there is no extra body event. Actor Start is $0.00005 per GB of memory, minimum one; platform resources are additional and paid by the user. Read the Pricing tab for effective account rates.

Useful post rows are the only lead/result event. Source errors, duplicates, filtered-out posts and diagnostics are not billed as posts. Actor platform resources can still apply on empty or failed runs. An available paid-post metadata row is still a useful result and can be billed; it is not a full paid article. Downloading an existing Dataset in another format does not start a new export.

Start with512MB and900seconds for a small export. A source response is limited to8MB, individual requests to25seconds and redirects are checked for public HTTPS destinations. The source may block this service or return incomplete text. No external API key, proxy subscription or publisher login is required by this version; reliability and constant speed are not guaranteed.

### API, MCP and integrations

Actor slug: `apivault_labs/substack-archive-scraper`. Start an [asynchronous Actor run](https://docs.apify.com/api/v2/act-runs-post), wait for a terminal status, read the default Dataset and its diagnostics, and [download Dataset items](https://docs.apify.com/api/v2/dataset-items-get). Keep your Apify token in secure client credentials.

Official [Python client](https://github.com/apify/apify-client-python), [JavaScript client](https://github.com/apify/apify-client-js) and [Apify MCP](https://docs.apify.com/platform/integrations/mcp) can call this exact slug with its input. They are generic clients, not dedicated packages for this product. No matching dedicated npm/PyPI package is currently verified. n8n/Make can use HTTP integration; execution and n8n verification are not claimed.

### FAQ

**Does it bypass a paywall?** No. Restricted/unlock-required posts are metadata plus a short preview only; body fields are omitted even if an anonymous endpoint happens to supply them.

**Can I search all Substack by keyword?** Not in this version. Supply known publications or post URLs. A keyword option must not silently substitute an unrelated source.

**Does it support custom domains?** It queries the public Substack archive/post shape at the supplied domain and follows safe redirects. A custom domain that no longer uses Substack is reported as an error, not scraped as an unrelated website.

**Why are there fewer rows than maxItems?** Date/free filters, overlap, source exhaustion, request/page limits and spending cap can all reduce the result. Inspect the summary and continuation.

**Are missing counters zero?** No. Unknown source fields are null, not guessed zeros. Comments themselves and subscriber counts are not collected.

**Is this a monitoring snapshot?** No. Continuation is offset-based with identity deduplication. Store your own observations by source identity to compare runs; editing/deletion history is not reconstructed.

# Changelog

This Actor's version history is a separate document: https://apify.com/apivault_labs/substack-archive-scraper/changelog.md

# Actor input Schema

## `urls` (type: `array`):

Public HTTPS newsletter homepage, /archive or /p/post-slug. Custom Substack domains are supported; not arbitrary websites.

## `maxItems` (type: `integer`):

Global output limit across all inputs. Budget, source pages and matching data can stop earlier.

## `includeContent` (type: `boolean`):

Public/free body HTML and clean text when source returns it. Paid posts always metadata and short preview only.

## `onlyFree` (type: `boolean`):

Skip posts whose audience is restricted or unlock is required.

## `startDate` (type: `string`):

Inclusive YYYY-MM-DD based on source post_date.

## `endDate` (type: `string`):

Inclusive YYYY-MM-DD. Filters scan pages; no server-side date index guarantee.

## `maxPages` (type: `integer`):

Global archive/input page cap,20posts per archive page. Requested public content can add one detail request per accepted free post. Filtered and duplicate rows consume archive pages.

## `cursor` (type: `object`):

Copy only from CONTINUATION.resumeInput, keeping URLs/filters unchanged. Not an immutable snapshot.

## Actor input object example

```json
{
  "urls": [
    "https://www.experimental-history.com"
  ],
  "maxItems": 5,
  "includeContent": false,
  "onlyFree": false,
  "maxPages": 20
}
```

# Actor output Schema

## `posts` (type: `string`):

No description

## `summary` (type: `string`):

No description

## `errors` (type: `string`):

No description

## `continuation` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("apivault_labs/substack-archive-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("apivault_labs/substack-archive-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call apivault_labs/substack-archive-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,apivault_labs/substack-archive-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/nJFN711cBZgSpDITt/builds/zu6kAO9G6xndDeZcm/openapi.json