# Substack Scraper — Free Posts & Publications (`acute_view/substack-free-posts-scraper`) Actor

Extract free Substack posts from publication URLs, custom domains or publication search. Get clean text, HTML, bylines, dates and engagement. No paywall bypass.

- **URL**: https://apify.com/acute_view/substack-free-posts-scraper.md
- **Developed by:** [abhinav subedi](https://apify.com/acute_view) (community)
- **Categories:** News, Social media
- **Features:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 0.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

$1.80 / 1,000 free post scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper — Free Posts & Publications

Extract publicly available **free** Substack posts from newsletter subdomains and custom domains. Each row contains publication identity, post metadata, author bylines, engagement and optional full HTML/plain text. Export from Apify as JSON, CSV, Excel or through the dataset API.

### Quick start

```json
{
  "publicationUrls": ["astralcodexten.substack.com", "https://www.lennysnewsletter.com", "noahpinion.substack.com"],
  "maxPostsPerPublication": 10,
  "publishedAfter": "2026-01-01",
  "includeBody": true
}
```

Provide at least one publication URL or subdomain. Custom domains must actually host a Substack publication; not every newsletter does.

### Inputs

| Field | Default | Meaning |
| --- | --- | --- |
| `publicationUrls` | empty | Array of bare Substack subdomains, HTTPS publication URLs or custom domains |
| `maxPostsPerPublication` | 10 | Maximum accepted free posts per publication (1–1000) |
| `publishedAfter`, `publishedBefore` | none | Inclusive ISO dates/timestamps; date alone means midnight UTC |
| `includeBody` | true | Include `bodyText` and `bodyHtml`; false omits both fields |
| `requestDelayMillis` | 1500 | Pause before each request, at least 1000ms |
| `maxArchivePages` | 100 | Safety cap per publication, up to 1000 pages of 12 |

### Sample output

```json
{
  "publicationName": "Astral Codex Ten",
  "publicationUrl": "https://www.astralcodexten.com",
  "publicationSubdomain": "astralcodexten",
  "title": "The AI Forecasting Agenda",
  "subtitle": "...",
  "url": "https://www.astralcodexten.com/p/the-ai-forecasting-agenda",
  "slug": "the-ai-forecasting-agenda",
  "publishedAt": "2026-11-19T20:29:45.205Z",
  "audience": "everyone",
  "bylines": [{"name":"Scott Alexander","handle":"astralcodexten","profileUrl":"https://substack.com/@astralcodexten"}],
  "reactions": {"❤":141},
  "likeCount": 141,
  "commentCount": 139,
  "wordCount": 3638,
  "coverImage": "https://substack-post-media.s3.amazonaws.com/public/images/4125aad4-afa1-465e-ac49-4f4926beff76_1536x1024.png",
  "bodyText": "Full freely accessible article text…",
  "bodyHtml": "<p>Full freely accessible article HTML…</p>"
}
```

Counts change over time. Missing publication metadata falls back to the input host; the subdomain is null for unidentified custom domains. `likeCount` uses Substack's reaction count, with the heart count as fallback. Byline objects contain only name, handle and profile URL: no author email, biography, subscriber profiles or comments are collected. Body content remains publisher-authored public content and may itself mention people.

### Pricing

Target price: **$0.0018 per result ($1.80 per 1,000 posts)**, one `result` charge event per saved post, with or without bodies. No charge events for skipped paid posts, errors or empty results. This is below the popular competitor's $0.002–$0.0023 full-content price checked at launch. Apify's live pricing panel is authoritative; monetization requires the owner's payout/billing setup and may remain unavailable until completed. Platform usage charges may apply as shown by Apify.

### Limits, reliability and responsible use

- No credentials, subscriber cookies, paywall bypass, or paid previews. Only `audience: everyone` detail responses with a nonempty body and no truncation/paywall flags are accepted. This check runs even when bodies are omitted. Archive excerpts (`truncated_body_text`) are never used as full content.
- One request at a time. Requests pause between calls; HTTP 429 and server errors retry up to three times with exponential backoff and respect `Retry-After`. Access-denied responses are not bypassed.
- Archives are requested in pages of 12 with `sort=new`, an empty `search` parameter and an offset. If the public archive rejects an offset with HTTP 400, the scraper continues through public `previous_post_slug` links. Both paths obey the same free-content checks, rate limit and scan bound. Substack's undocumented endpoints can change; failures yield partial results with explicit errors. A bounded scan can return fewer posts than requested. Repeated pages and links are detected.
- Dataset URLs are deduplicated within a run. No stored history across runs. Metadata-only or empty-body posts are intentionally skipped.
- Inspect the `SUMMARY` key-value record for result/skipped counts and per-publication errors. A run with errors and no results fails; a legitimately empty archive succeeds with zero results.
- Public access does not grant republication rights. Respect publisher terms, copyright, privacy and applicable law. Use for permitted research/analysis; do not redistribute full articles without permission. Render untrusted HTML safely.

### Development

Node.js 22+. `npm ci`, `npm test`, then `apify run --purge --input-file example-input.json`. Deploy only this directory using `apify push`. The folder-local `publish.mjs` configures only this Actor, never other Actors in the account.

# Actor input Schema

## `publicationUrls` (type: `array`):

Substack URLs, custom domains, or bare subdomains. At least one is required.

## `maxPostsPerPublication` (type: `integer`):

Paid and preview-only posts do not count toward this limit.

## `publishedAfter` (type: `string`):

Optional inclusive UTC date/time, e.g. 2026-01-01.

## `publishedBefore` (type: `string`):

Optional inclusive UTC date/time. A date alone means midnight UTC.

## `includeBody` (type: `boolean`):

When false, bodies are omitted from output; detail is still checked to reject previews.

## `requestDelayMillis` (type: `integer`):

Minimum pause between requests. Concurrency is always one.

## `maxArchivePages` (type: `integer`):

Safety bound when many posts are paid or outside the date range.

## Actor input object example

```json
{
  "publicationUrls": [
    "astralcodexten.substack.com",
    "https://www.lennysnewsletter.com"
  ],
  "maxPostsPerPublication": 10,
  "includeBody": true,
  "requestDelayMillis": 1500,
  "maxArchivePages": 100
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "publicationUrls": [
        "astralcodexten.substack.com",
        "https://www.lennysnewsletter.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("acute_view/substack-free-posts-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "publicationUrls": [
        "astralcodexten.substack.com",
        "https://www.lennysnewsletter.com",
    ] }

# Run the Actor and wait for it to finish
run = client.actor("acute_view/substack-free-posts-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "publicationUrls": [
    "astralcodexten.substack.com",
    "https://www.lennysnewsletter.com"
  ]
}' |
apify call acute_view/substack-free-posts-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,acute_view/substack-free-posts-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/lNXVMsebNP9fbYTpB/builds/RBMm7WlmlqaIlLNxZ/openapi.json
