# Substack Scraper - Newsletters, Authors & Public Posts (`igolaizola/substack-scraper`) Actor

Discover substack.com newsletters by keyword or publication URL. Collect author profiles, public subscriber labels, subscription plans, sections, and optional public posts. Export JSON, CSV, or Excel, or connect via API and MCP for newsletter research, creator discovery, and partnership prospecting.

- **URL**: https://apify.com/igolaizola/substack-scraper.md
- **Developed by:** [Iñigo Garcia Olaizola](https://apify.com/igolaizola) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.70 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Substack Scraper — Newsletters, Authors & Public Posts

Substack Scraper finds newsletters by keyword or publication URL and collects author profiles, public subscriber labels, subscription plans, sections, and optional public posts. Export newsletter research as JSON, CSV, or Excel for creator discovery, partnership prospecting, and content monitoring.

### 🤖 What does Substack Scraper do?

- **Find publications** by keyword or specific public Substack URL.
- **Return rich publication profiles** including descriptions, author names and biographies, images, language, creation dates, sections, and available subscription information.
- **Optionally include posts** with titles, dates, authors, descriptions, public content or previews, categories, and media links.
- **Export results** as JSON, CSV, or Excel, or integrate through the Apify API and MCP.

**Great for:** researching newsletter niches, discovering potential partners, comparing publication profiles, and monitoring recent public writing.

### 🧠 Choose your collection mode

#### Discover newsletters by keyword

Enter one or more words or phrases in `keywords`, such as `["software", "artificial intelligence"]`. Each keyword is searched in order, and `_keyword` identifies which search produced each result.

#### Look up known publications

Enter publication `urls`, such as `https://pragmaticengineer.substack.com`. You can combine URLs with keywords: supplied URLs are processed first, then keyword searches, sharing one `maxItems` limit. If publication details are unavailable, the URL is still included in your results.

In either mode, enable `fetchPosts` to include available recent public posts. This is not a complete historical archive.

### 🚀 How to use

1. Enter `keywords`, such as `["software", "technology"]`, supply publication `urls`, or use both.
2. Choose `maxItems`, the maximum publications to collect, or set it to `0` for unlimited results.
3. Enable `fetchPosts` if you also need available recent public posts.
4. Run the actor and export the Dataset.

### 📝 Input parameters

| Parameter | Type | Required | Description |
| --- | --- | --- | --- |
| `maxItems` | Integer | Yes | Maximum publications across all keywords or supplied URLs; default `100`. Set to `0` for unlimited results. |
| `keywords` | Array of strings | No\* | Search each keyword or phrase in order. Blank entries are skipped. Each result includes its `_keyword`. |
| `urls` | Array of strings | No | Publication links such as `https://pragmaticengineer.substack.com`. Use the publication's Substack address, not a custom domain. Works alongside `keywords`; URLs are processed first within the shared `maxItems` limit. URLs without available details are included with `_found: false`. |
| `fetchPosts` | Boolean | No | Include available recent public posts under `_posts`. Default `false`. Adds a charge per returned publication, not per post, including publications with no available posts. Does not cover individual sections or the full archive. |

\* Provide at least one non-empty keyword or publication URL. Without either, no results are collected. See the actor's pricing information for current charges before enabling `fetchPosts`.

**1️⃣ Find software and technology publications**

```json
{"keywords":["software","technology"],"maxItems":100}
```

**2️⃣ Fetch a known publication and its public posts**

```json
{"urls":["https://pragmaticengineer.substack.com"],"maxItems":1,"fetchPosts":true}
```

**3️⃣ Combine known publications with keyword discovery**

```json
{"urls":["https://pragmaticengineer.substack.com"],"keywords":["software"],"maxItems":25}
```

### 📊 Output and results

Each row contains one publication result, with available author details, audience information, subscription options, and optional posts:

- `_keyword`: the keyword or phrase that produced the result; absent when looking up supplied URLs.
- `id`, `name`, `subdomain`, `base_url`, `custom_domain`, `hero_text`: publication identity and description.
- `author_name`, `author_handle`, `author_bio`, `author_photo_url`, `contributors`: public author information.
- `language`, `copyright`, `created_at`, `first_post_date`: publication metadata.
- `logo_url`, `cover_photo_url`, `sections`, `plans`, subscription benefits, podcast and payment flags: available public profile details.
- `freeSubscriberCount`, `rankingDetailFreeSubscriberCount`, and related labels: public audience indicators where provided. These may be rounded labels, not exact counts.
- `_url`: the publication's Substack address.
- `_found`: `true` when publication details are available. Otherwise, the result includes the supplied publication's address and `_found: false`. This does not mean the publication does not exist.
- `_posts`: included only with `fetchPosts`. Posts can include titles, descriptions, links, authors, publication dates, public content or previews, categories, and media links. The number of entries is the returned post count.

A shortened example from a URL lookup with `fetchPosts` enabled:

```json
{
  "id": 458709,
  "name": "The Pragmatic Engineer",
  "subdomain": "pragmaticengineer",
  "author_name": "Gergely Orosz",
  "language": "en",
  "_url": "https://pragmaticengineer.substack.com",
  "_found": true,
  "_posts": [
    {
      "title": "What’s the ROI of software engineers’ spending on AI?",
      "url": "https://newsletter.pragmaticengineer.com/p/whats-the-roi-of-software-engineers",
      "publishedAt": "2026-10-20T17:34:18Z"
    }
  ]
}
```

Some profile details may be unavailable. The **Overview** view shows selected fields; choose **All fields** when exporting to include complete profiles and posts. Private subscriber information is not collected.

URLs without available publication details still count toward `maxItems` and can include posts when `fetchPosts` is enabled. Unsupported or invalid URLs are skipped.

### 🧭 Tips and limitations

- Keep `fetchPosts` disabled for faster publication research.
- Start with a small `maxItems`, such as `10`, and inspect the results before increasing your collection size.
- Use `urls` for known publications and `keywords` for discovery. Each keyword is searched separately, in input order, until the overall `maxItems` limit is reached.
- Post collection covers available recent posts, not a publication's complete history or individual sections.
- A publication stays in your results even when no posts are available. In that case, `_posts` is an empty list.
- Search rankings and fields can change. This actor returns the results currently made public by Substack.

### ⚖️ Legal and ethical considerations

- Use only publicly available publication information and posts.
- Respect Substack and publishers' terms and applicable privacy law.
- Keep collection volumes and run frequency reasonable, and preserve source links and author attribution when using the results.
- No sign-in, access to private subscriptions, subscriber-only podcasts, or paywall bypass is offered.
- Post content is limited to publicly available text and previews. Media links are included without downloading the files.
- This Actor is independent and is not affiliated with, endorsed by, or sponsored by Substack, Inc.

### ❓ FAQ

**Why did I receive fewer publications than `maxItems`?**
The limit is a maximum, not a guaranteed count. The actor stops when the available search results are exhausted or the run budget is reached. Try additional relevant keywords to explore a wider niche.

**What does `_found: false` mean?**
Publication details could not be found for the supplied URL. The URL is still saved, and the publication may still exist.

**Does `fetchPosts` return every historical or paid post?**
No. It returns available recent posts with publicly accessible content or previews, not the full archive or subscriber-only content.

**Can I provide a custom domain?**
Use the publication's `substack.com` address. Results can still include its custom website address when available.

### 🛟 Support

Need another field or workflow? Open an issue from the actor's **Issues** tab, or [contact the developer](https://igolaizola.com/#contact).

# Actor input Schema

## `maxItems` (type: `integer`):

Maximum publications across all keywords or supplied URLs. Set to 0 for unlimited results, subject to your run budget and available results. Negative values are automatically changed to 0. This limits publications, not posts.

## `keywords` (type: `array`):

Search each keyword or phrase in order. Results include their \_keyword. At least one non-empty keyword is required unless publication URLs are provided; blank entries are skipped.

## `urls` (type: `array`):

Publication URLs such as https://pragmaticengineer.substack.com. Can be combined with keywords. URLs are processed first, sharing the same maxItems limit with keyword searches. URLs without available publication details are still included with \_found=false. Posts are included only when fetchPosts is enabled.

## `fetchPosts` (type: `boolean`):

Include available recent public posts and previews under \_posts. Adds a charge per returned publication, not per post, including publications with no available posts. Does not cover individual sections or the full archive.

## Actor input object example

```json
{
  "maxItems": 100,
  "keywords": [
    "software"
  ],
  "fetchPosts": false
}
```

# Actor output Schema

## `results` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "keywords": [
        "software"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("igolaizola/substack-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = { "keywords": ["software"] }

# Run the Actor and wait for it to finish
run = client.actor("igolaizola/substack-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "keywords": [
    "software"
  ]
}' |
apify call igolaizola/substack-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,igolaizola/substack-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/PXijxpTf9q4NGHABs/builds/NHm3pkYonsU4NgZmh/openapi.json