# Substack Scraper - Discover Newsletters, Posts & Authors (`undivided_alpenglow/substack-intelligence`) Actor

Search all of Substack by keyword to discover newsletters, then scrape full publication data, post archives, and author metadata. The only Substack actor with cross-platform discovery — find every AI newsletter, crypto blog, or niche publication in seconds.

- **URL**: https://apify.com/undivided\_alpenglow/substack-intelligence.md
- **Developed by:** [Ebik Maharjan](https://apify.com/undivided_alpenglow) (community)
- **Categories:** Social media, Automation, Lead generation
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $3.00 / 1,000 publication scrapeds

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event

## What's an Apify Actor?

Actors are web data automations that power AI and operations. They run on the Apify platform to scrape websites, process data, connect APIs, and automate workflows.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.
Actors are written with capital "A".

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.
The best way to integrate Actors is as follows.

- **AI agents and MCP clients** — the [Apify MCP server](https://docs.apify.com/integrations/mcp.md) at `https://mcp.apify.com` (remote, streamable HTTP, OAuth on first use).
- **Agentic workflows and local Actor development** — [Agent Skills](https://apify.com/.well-known/agent-skills/index.json) with the [Apify CLI](https://docs.apify.com/cli/docs.md): `npm install -g apify-cli`, then `apify login`.
- **JavaScript/TypeScript projects** — the official [JS/TS client](https://docs.apify.com/api/client/js/docs.md): `npm install apify-client`.
- **Python projects** — the official [Python client](https://docs.apify.com/api/client/python/docs.md): `pip install apify-client`.
- **Any other language** — the [REST API](https://docs.apify.com/api/v2.md).

For usage examples, see the [API](#api) section below.

For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt).

# README

## Substack Scraper — Discover Newsletters, Posts & Author Data

**The only Substack actor that lets you find publications you don't already know exist.**

Every other Substack scraper on the Apify Store requires you to bring a publication URL. This one lets you search all of Substack by keyword ("AI newsletters with 10K+ subscribers", "crypto blogs launched in 2025") and then pipe the results straight into a full-metadata scrape.

### What you can do with it

- **Lead generation** — build a list of every Substack in your niche, with author names and subscriber counts
- **Competitive intelligence** — track what publications your competitors recommend, what their post cadence is, which posts get the most engagement
- **Sponsorship research** — filter for publications above a subscriber threshold, with a paid tier, in a specific category
- **AI / RAG datasets** — bulk export post HTML with clean metadata, ready for LlamaIndex / LangChain

### Three modes, one actor

#### 1. `discovery` — search all of Substack

Provide any list of keywords. The actor searches Substack's own search index and returns matching publications with:

- Name, subdomain, custom domain
- Author name, handle, bio
- Subscriber count (when public) + explicit `subscriber_count_hidden` flag
- Has-paid-tier, has-podcast, language, first post date

Optionally filter results by `minSubscribers` and `language`.

Substack's search index does not expose category or post count, so those fields
are null in this mode — use `category` mode if you need the category.

**Cost:** $0.003 per publication returned.

#### 2. `publication` — deep-scrape by URL

Paste one or more publication URLs (works with `*.substack.com` and custom domains). Returns:

- Full publication metadata (as above)
- All posts with title, subtitle, publish date, wordcount, reactions per emoji, comment count, tags
- Optional: full post HTML body (paid post previews included where available)

**Cost:** $0.001/post metadata-only, $0.003/post with body.

#### 3. `category` — browse Substack's leaderboards

Pick a category (technology, business, crypto, …) and return the top publications on Substack's own leaderboard, in ranked order.

**Cost:** $0.003 per publication.

### Pricing

Pay per result. No monthly rental.

| Event | Cost |
|---|---|
| Actor start (per run) | $0.005 |
| Publication returned | $0.003 |
| Post metadata only | $0.001 |
| Post with full body | $0.003 |

Example: discovering 500 AI newsletters ≈ $1.51. Scraping the archive (50 posts each, metadata only) of 20 of them ≈ $1.01.

### Example inputs

**Find every AI publication with 5K+ subscribers:**

```json
{
  "mode": "discovery",
  "searchQueries": ["artificial intelligence", "machine learning", "LLM"],
  "maxItemsPerQuery": 500,
  "minSubscribers": 5000
}
```

**Scrape a specific publication's last 200 posts with full bodies:**

```json
{
  "mode": "publication",
  "publicationUrls": ["https://stratechery.com"],
  "maxPostsPerPublication": 200,
  "includeFullBody": true
}
```

**Get the top 100 tech Substacks by leaderboard rank:**

```json
{
  "mode": "category",
  "category": "technology",
  "maxItemsPerQuery": 100
}
```

### Output schema

Every row is either a `publication` or a `post` (indicated by `record_type`). Both live in the same dataset so a single run can produce a unified CSV.

Publications include an explicit `subscriber_count_hidden: bool` so you can distinguish "0 subscribers" from "count hidden by publication" — no other actor does this and it matters for filtering.

### Known limitations

Honest about what any Substack scraper can and can't do:

- **Paywalled post bodies:** Substack returns only the preview text to unauthenticated requests. Only previews are scraped; there is no way to get full paid content without a subscriber cookie for each publication.
- **Subscriber counts are free-tier only:** Substack publicly exposes a publication's *free* subscriber count (never paid counts). When the exact number is hidden, the actor still returns an order-of-magnitude band in `subscriber_count_display` (e.g. `"93K+"`) and sets `subscriber_count_hidden: true`. The `minSubscribers` filter uses whichever is available.
- **Notes (Substack's Twitter-like feed):** Not yet supported — planned for v1.1.
- **Substack changes their API:** This actor uses undocumented endpoints. If they change, we ship a fix within 24 hours. Success rate is monitored publicly on the actor page.

### Roadmap

- **v1.1** — Notes feed scraping, comment tree extraction
- **v1.2** — Cross-publication recommendation graph
- **v1.3** — Growth tracking (subscriber deltas across scheduled runs)
- **v1.4** — Author search (find someone's Substacks by email or name)

### Support

Open an issue on this actor's page. Median first response < 24h.

# Actor input Schema

## `mode` (type: `string`):

What do you want to do?

## `searchQueries` (type: `array`):

Keywords or topics to search for across Substack. One query per line.

## `publicationUrls` (type: `array`):

URLs of publications to scrape (e.g. https://stratechery.com or https://noahpinion.substack.com).

## `category` (type: `string`):

Which Substack category leaderboard to browse.

## `maxItemsPerQuery` (type: `integer`):

How many publications to return per search query or category.

## `maxPostsPerPublication` (type: `integer`):

In publication mode, how many recent posts to scrape per publication.

## `includeFullBody` (type: `boolean`):

Fetch each post's full HTML body. Slower and costs more per post. Off = metadata only.

## `includePaywalledPreview` (type: `boolean`):

Include paywalled posts, with their free preview text as the body. Full paid bodies cannot be scraped. Turn OFF to skip paywalled posts entirely (they are not returned at all).

## `minSubscribers` (type: `integer`):

Filter out publications below this subscriber count. Note: ~50% of publications hide their count and will be skipped when this is >0.

## `language` (type: `string`):

Only return publications in this language, as a two-letter code (en, es, fr, de, …). Publications that don't report a language are skipped when this is set. Leave empty to return all languages.

## `proxyConfiguration` (type: `object`):

Datacenter proxy by default (included on every plan). Substack lightly rate-limits by IP — switch to residential proxies for large runs if your plan includes them.

## Actor input object example

```json
{
  "mode": "discovery",
  "searchQueries": [
    "artificial intelligence",
    "machine learning"
  ],
  "publicationUrls": [
    "https://stratechery.com"
  ],
  "category": "Technology",
  "maxItemsPerQuery": 100,
  "maxPostsPerPublication": 50,
  "includeFullBody": false,
  "includePaywalledPreview": true,
  "minSubscribers": 0,
  "language": "",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `results` (type: `string`):

All scraped records. Each row is either a publication or a post — see the record\_type field.

## `resultsCsv` (type: `string`):

The same records in CSV, ready for a spreadsheet.

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "searchQueries": [
        "artificial intelligence",
        "machine learning"
    ],
    "publicationUrls": [
        "https://stratechery.com"
    ]
};

// Run the Actor and wait for it to finish
const run = await client.actor("undivided_alpenglow/substack-intelligence").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "searchQueries": [
        "artificial intelligence",
        "machine learning",
    ],
    "publicationUrls": ["https://stratechery.com"],
}

# Run the Actor and wait for it to finish
run = client.actor("undivided_alpenglow/substack-intelligence").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "searchQueries": [
    "artificial intelligence",
    "machine learning"
  ],
  "publicationUrls": [
    "https://stratechery.com"
  ]
}' |
apify call undivided_alpenglow/substack-intelligence --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,undivided_alpenglow/substack-intelligence"
        }
    }
}

```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/TLYnbh5doKLyeBrFy/builds/cLh7rmGH5MDnuPCzb/openapi.json