# Substack Scraper: Find Authors by Topic (`danthedataman/substack-author-atlas`) Actor

Find Substack authors by topic. Export public profiles, publications and matching post metadata, with language and date filters.

- **URL**: https://apify.com/danthedataman/substack-author-atlas.md
- **Developed by:** [Eli J](https://apify.com/danthedataman) (community)
- **Categories:** Social media, News
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $0.25 / 1,000 authors

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.
Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

This Substack scraper finds authors by topic. Each result is a distinct public author with the profile fields the search exposes, the publications associated with them, and the matching post that put them in the results. Use it for editorial research, newsletter discovery and expert sourcing.

### Quick start

```json
{"queries": ["climate policy"], "maxAuthorsPerQuery": 10}
```

Open the dataset after the run and export JSON, CSV or Excel. The input form is prefilled with this example. API callers must send the input object explicitly. An empty object, missing topics or cap, an empty topic list, or a zero cap returns no rows, logs guidance and writes no `COVERAGE` or `ERRORS` record. No topics are inserted automatically.

### Input

| Field | Type | What it does |
| --- | --- | --- |
| `queries` | array | Topics or keywords, as strings. Required for work. Duplicate topics are searched once. |
| `maxAuthorsPerQuery` | integer | New distinct authors per topic, from 0 to 10000. Required for work; 0 exports nothing. |
| `language` | string | Optional lowercase two-letter code; matches the source post's public language. Missing language is excluded when filtering. |
| `dateFrom` | string | Optional inclusive source-post calendar date in YYYY-MM-DD form. |
| `dateTo` | string | Optional inclusive source-post calendar date, on or after dateFrom. |

For example, find English-language authors whose matching post was published in September 2026:

```json
{"queries": ["climate policy"], "maxAuthorsPerQuery": 10, "language": "en", "dateFrom": "2026-09-01", "dateTo": "2026-09-30"}
```

Filters are applied to returned public posts before writing or billing. Restrictive filters can require more search pages and return fewer authors. Date bounds use the calendar date in the source timestamp.

### Output

This excerpt uses public search data captured on 2026-10-07. It shows selected fields; the scrape timestamp illustrates its UTC format.

```json
{
  "recordType": "author",
  "topic": "climate policy",
  "authorId": "19668611",
  "authorName": "Oren Cass",
  "authorProfileUrl": "https://substack.com/@orencass",
  "subscriberCount": null,
  "publications": [
    {
      "id": "2454085",
      "name": "Commonplace",
      "url": "https://mag.commonplace.org"
    }
  ],
  "sourcePost": {
    "id": "216216777",
    "url": "https://mag.commonplace.org/p/a-new-right-climate-policy-with-chris",
    "title": "A New Right Climate Policy with Chris Barnard and Taylor Myers",
    "publishedAt": "2026-09-18T13:15:08.962Z",
    "language": "en"
  },
  "scrapedAt": "2026-10-07T04:55:25.123456Z"
}
```

| Output key | Meaning |
| --- | --- |
| `recordType`, `topic` | `author` and the first topic that emitted this author. |
| `authorId`, `authorName`, `authorHandle` | Public byline identity; IDs are opaque decimal strings. |
| `authorBio`, `authorPhotoUrl`, `authorProfileUrl` | Public biography, photo and Substack profile link. Optional source values are null when absent or of the wrong type. |
| `publicLinks` | Public URL strings explicitly exposed in byline links, website or Twitter URL fields; no private contacts are inferred. |
| `subscriberCount` | Exact public author subscriber count when exposed, otherwise null. Subscription and bestseller tiers are never treated as counts. |
| `publications` | Associated publication objects: `id`, `name`, `url`, `description`, `language`, `logoUrl`, `subscriberCount`, `freeSubscriberCountText`, `freeSubscriberCountDisplay`, `freeSubscriberCountLowerBound`, `paymentsEnabled`, `role`. Includes the source publication when returned. Exact counts are individual subscribers, null when hidden or not exact. Role is null when not exposed. |
| `sourcePost` | One matching post: `id`, `url`, `title`, `subtitle`, `publishedAt`, `postType`, `language`, `reactionCount`, `commentCount`, `restackCount`, `wordCount`, `coverImageUrl`. Engagement values count individual actions, wordCount counts words. Optional values are null when absent or of the wrong type. Posts without a valid ID, public URL, nonempty title or timezone-qualified publication date are rejected as source errors. |
| `sourceUrl`, `scrapedAt` | Exact search endpoint and UTC time its response was processed. |

Publication `freeSubscriberCountText` and `freeSubscriberCountDisplay` keep Substack's public free-subscriber displays as shown, for example `"11,000"` and `"Over 11,000 subscribers"`. `freeSubscriberCountLowerBound` parses a digits-and-commas display into an integer, 11000 here, and is a lower bound rather than an exact count. Abbreviated displays such as `"11K+"` are not estimated; absent source values are null. These fields are separate from `subscriberCount` and paid-subscriber tiers.

### Coverage and deduplication

Search returns Substack's ranked public post matches. The Actor follows the live continuation flag, starts at page zero, and stops at the author cap or search exhaustion. Repeated pages are reported as an unresolved source. Results are a ranked sample, not a census of everyone writing on the topic, and the cap is an upper bound.

Within each fetched page, the Actor selects the first byline of each qualifying ranked post before remaining coauthors. The cap counts distinct authors, so a small cap can omit coauthors while covering more source publications. Each author ID is emitted once across the run. The earliest-ranked qualifying source post and first topic are retained even when selection order changes. Publication associations exposed for that author on posts that pass the filters within the fetched page are combined. Later pages never bill an emitted author again. The Actor does not promise an independent publication-directory search or an exhaustive author graph.

Some valid public posts name no author. A missing or empty byline array on an otherwise valid post is skipped without retries, browser fallback or a result charge. Other authors on the page and later pages are still processed. `postsWithoutAuthors` counts these posts per topic in `COVERAGE`.

Delivered IDs and counts use the SDK's saved run state. On a migration notice, the Actor stops new work, waits for a pending batch and checkpoints its delivered prefix before requesting a reboot. It waits for the platform to replace the process, so migration cannot finish an incomplete run as a successful job. A resumed run restores deduplication and per-topic caps before fetching more pages.

After a restart, `COVERAGE` lists topics reached by the resumed process. Its pages, posts without authors, requests, bytes, fallback counts and stop reasons describe that process; `authorsEmitted` for each listed topic includes earlier deliveries. Topics the process never reaches keep their delivered dataset rows and have no `COVERAGE` entry.

`ERRORS` retains earlier failures for unfinished topics. A new failure replaces the earlier failure for that topic; a resolved failure is removed. If no failures remain, an earlier error list is replaced with an empty list.

No-work and invalid-input runs do not create `COVERAGE`. No-work guidance appears in the log; invalid-input details are recorded in `ERRORS`. For runs that search, the `COVERAGE` key-value record shows pages read, authors emitted, posts without authors and the stop reason per topic, plus browser fallback requests and decoded response bytes. Its `requests` list records each page's source URL, final route, fallback trigger and HTTP/browser attempt counts. Its `egressRoutes` array names each attempt's direct, datacenter or browser route. `ERRORS` records failures still unresolved in this run, with classified reasons; a clean run with no earlier errors has no ERRORS record. Invalid dates, language codes and caps write an `ERRORS` record naming the input and stop before search. Date format and calendar errors refer to both date fields. Error records never enter the dataset. Partial successes retain useful rows. A valid zero-result search succeeds with guidance. A run with no delivered rows is marked failed when every attempted query fails. Runs with partial results can succeed while recording unresolved queries in `ERRORS`.

### How requests run

The first search request uses anonymous browserless HTTP with direct egress; no account or login is used. A blocked, challenged or malformed response is retried twice with fresh Apify datacenter identities. The rotated identity is retained for later pages. If those attempts fail, the default fallback opens headless Chromium, navigates to the same public JSON URL, and tries up to three times. Fallback pages use the same row contract and filters and are counted in `COVERAGE`.

Use 512 MB memory for both paths. Browser fallback launches Chromium, which adds time and compute per page; `browserFallbackRequests` in `COVERAGE` shows how often it ran. Decoded response bytes include accepted and rejected search responses. Compute and storage charges appear in the run's platform usage record; the Actor uses no residential proxy.

### Billing

Each successfully emitted distinct author is one dataset item and one billable result. Duplicate posts, skipped authors, filtered posts and error records create no result charge. Each page's eligible authors are deduplicated and capped before a batch write. The platform clips a binding batch to its chargeable prefix; the Actor records only that delivered prefix and stops before another write, page or query. Each delivered author costs USD 0.001 on the Free plan, USD 0.00085 on Bronze, USD 0.00065 on Silver and USD 0.00025 on Gold, Platinum and Diamond. That is USD 1.00, 0.85, 0.65 and 0.25 per 1,000 delivered authors. Each run also charges USD 0.00005 per Actor Start event: one event per GB of allocated memory, at least one. A run that returns no authors still pays it.

### Limits and troubleshooting

If results are empty, broaden the topic or remove language/date filters. Missing counts or links mean the anonymous response did not expose them. Ranked results can change between runs. For an unresolved request, inspect `ERRORS` and `COVERAGE`; share the run ID and input through this Actor's Issues tab.

This Actor covers topic-to-author research. Full articles, newsletter archive crawls, standalone profile scraping, publication-directory results, Notes, nested comments, contact extraction and historical growth are outside its coverage. It reads public metadata and does not retrieve subscriber-only bodies.

Substack's [Terms of Use](https://substack.com/tos), captured 2026-10-07, prohibit crawling or scraping its pages and data. Consider those terms when deciding whether to use the exported information. This Actor is independently maintained and is not affiliated with Substack.

# Changelog

This Actor's version history is a separate document: https://apify.com/danthedataman/substack-author-atlas/changelog.md

# Actor input Schema

## `queries` (type: `array`):

Public post searches to find authors. Required to do work; omitted or empty means no search.

## `maxAuthorsPerQuery` (type: `integer`):

Maximum new distinct authors for each topic. Required to do work; 0 means no search. Authors already emitted for another topic are skipped.

## `language` (type: `string`):

Optional lowercase two-letter language code, applied to source posts. Posts with no exposed language do not match.

## `dateFrom` (type: `string`):

Optional inclusive source-post calendar date, YYYY-MM-DD.

## `dateTo` (type: `string`):

Optional inclusive source-post calendar date, YYYY-MM-DD. Must be on or after dateFrom.

## Actor input object example

```json
{
  "queries": [
    "climate policy"
  ],
  "maxAuthorsPerQuery": 10
}
```

# Actor output Schema

## `results` (type: `string`):

No description

## `errors` (type: `string`):

No description

## `coverage` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "queries": [
        "climate policy"
    ],
    "maxAuthorsPerQuery": 10
};

// Run the Actor and wait for it to finish
const run = await client.actor("danthedataman/substack-author-atlas").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "queries": ["climate policy"],
    "maxAuthorsPerQuery": 10,
}

# Run the Actor and wait for it to finish
run = client.actor("danthedataman/substack-author-atlas").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "queries": [
    "climate policy"
  ],
  "maxAuthorsPerQuery": 10
}' |
apify call danthedataman/substack-author-atlas --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,danthedataman/substack-author-atlas"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/Pdkypc0RbHoIeklGH/builds/ajRvec3MEE1Y0Co89/openapi.json