Substack Scraper: Find Authors by Topic

This Substack scraper finds authors by topic. Each result is a distinct public author with the profile fields the search exposes, the publications associated with them, and the matching post that put them in the results. Use it for editorial research, newsletter discovery and expert sourcing.

Quick start

{"queries": ["climate policy"], "maxAuthorsPerQuery": 10}

Open the dataset after the run and export JSON, CSV or Excel. The input form is prefilled with this example. API callers must send the input object explicitly. An empty object, missing topics or cap, an empty topic list, or a zero cap returns no rows, logs guidance and writes no COVERAGE or ERRORS record. No topics are inserted automatically.

Input

Field	Type	What it does
queries	array	Topics or keywords, as strings. Required for work. Duplicate topics are searched once.
maxAuthorsPerQuery	integer	New distinct authors per topic, from 0 to 10000. Required for work; 0 exports nothing.
language	string	Optional lowercase two-letter code; matches the source post's public language. Missing language is excluded when filtering.
dateFrom	string	Optional inclusive source-post calendar date in YYYY-MM-DD form.
dateTo	string	Optional inclusive source-post calendar date, on or after dateFrom.

For example, find English-language authors whose matching post was published in September 2026:

{"queries": ["climate policy"], "maxAuthorsPerQuery": 10, "language": "en", "dateFrom": "2026-09-01", "dateTo": "2026-09-30"}

Filters are applied to returned public posts before writing or billing. Restrictive filters can require more search pages and return fewer authors. Date bounds use the calendar date in the source timestamp.

Output

This excerpt uses public search data captured on 2026-10-07. It shows selected fields; the scrape timestamp illustrates its UTC format.

{
  "recordType": "author",
  "topic": "climate policy",
  "authorId": "19668611",
  "authorName": "Oren Cass",
  "authorProfileUrl": "https://substack.com/@orencass",
  "subscriberCount": null,
  "publications": [
    {
      "id": "2454085",
      "name": "Commonplace",
      "url": "https://mag.commonplace.org"
    }
  ],
  "sourcePost": {
    "id": "216216777",
    "url": "https://mag.commonplace.org/p/a-new-right-climate-policy-with-chris",
    "title": "A New Right Climate Policy with Chris Barnard and Taylor Myers",
    "publishedAt": "2026-09-18T13:15:08.962Z",
    "language": "en"
  },
  "scrapedAt": "2026-10-07T04:55:25.123456Z"
}

Output key	Meaning
recordType, topic	author and the first topic that emitted this author.
authorId, authorName, authorHandle	Public byline identity; IDs are opaque decimal strings.
authorBio, authorPhotoUrl, authorProfileUrl	Public biography, photo and Substack profile link. Optional source values are null when absent or of the wrong type.
publicLinks	Public URL strings explicitly exposed in byline links, website or Twitter URL fields; no private contacts are inferred.
subscriberCount	Exact public author subscriber count when exposed, otherwise null. Subscription and bestseller tiers are never treated as counts.
publications	Associated publication objects: id, name, url, description, language, logoUrl, subscriberCount, freeSubscriberCountText, freeSubscriberCountDisplay, freeSubscriberCountLowerBound, paymentsEnabled, role. Includes the source publication when returned. Exact counts are individual subscribers, null when hidden or not exact. Role is null when not exposed.
sourcePost	One matching post: id, url, title, subtitle, publishedAt, postType, language, reactionCount, commentCount, restackCount, wordCount, coverImageUrl. Engagement values count individual actions, wordCount counts words. Optional values are null when absent or of the wrong type. Posts without a valid ID, public URL, nonempty title or timezone-qualified publication date are rejected as source errors.
sourceUrl, scrapedAt	Exact search endpoint and UTC time its response was processed.

Publication freeSubscriberCountText and freeSubscriberCountDisplay keep Substack's public free-subscriber displays as shown, for example "11,000" and "Over 11,000 subscribers". freeSubscriberCountLowerBound parses a digits-and-commas display into an integer, 11000 here, and is a lower bound rather than an exact count. Abbreviated displays such as "11K+" are not estimated; absent source values are null. These fields are separate from subscriberCount and paid-subscriber tiers.

Coverage and deduplication

Search returns Substack's ranked public post matches. The Actor follows the live continuation flag, starts at page zero, and stops at the author cap or search exhaustion. Repeated pages are reported as an unresolved source. Results are a ranked sample, not a census of everyone writing on the topic, and the cap is an upper bound.

Within each fetched page, the Actor selects the first byline of each qualifying ranked post before remaining coauthors. The cap counts distinct authors, so a small cap can omit coauthors while covering more source publications. Each author ID is emitted once across the run. The earliest-ranked qualifying source post and first topic are retained even when selection order changes. Publication associations exposed for that author on posts that pass the filters within the fetched page are combined. Later pages never bill an emitted author again. The Actor does not promise an independent publication-directory search or an exhaustive author graph.

Some valid public posts name no author. A missing or empty byline array on an otherwise valid post is skipped without retries, browser fallback or a result charge. Other authors on the page and later pages are still processed. postsWithoutAuthors counts these posts per topic in COVERAGE.

Delivered IDs and counts use the SDK's saved run state. On a migration notice, the Actor stops new work, waits for a pending batch and checkpoints its delivered prefix before requesting a reboot. It waits for the platform to replace the process, so migration cannot finish an incomplete run as a successful job. A resumed run restores deduplication and per-topic caps before fetching more pages.

After a restart, COVERAGE lists topics reached by the resumed process. Its pages, posts without authors, requests, bytes, fallback counts and stop reasons describe that process; authorsEmitted for each listed topic includes earlier deliveries. Topics the process never reaches keep their delivered dataset rows and have no COVERAGE entry.

ERRORS retains earlier failures for unfinished topics. A new failure replaces the earlier failure for that topic; a resolved failure is removed. If no failures remain, an earlier error list is replaced with an empty list.

No-work and invalid-input runs do not create COVERAGE. No-work guidance appears in the log; invalid-input details are recorded in ERRORS. For runs that search, the COVERAGE key-value record shows pages read, authors emitted, posts without authors and the stop reason per topic, plus browser fallback requests and decoded response bytes. Its requests list records each page's source URL, final route, fallback trigger and HTTP/browser attempt counts. Its egressRoutes array names each attempt's direct, datacenter or browser route. ERRORS records failures still unresolved in this run, with classified reasons; a clean run with no earlier errors has no ERRORS record. Invalid dates, language codes and caps write an ERRORS record naming the input and stop before search. Date format and calendar errors refer to both date fields. Error records never enter the dataset. Partial successes retain useful rows. A valid zero-result search succeeds with guidance. A run with no delivered rows is marked failed when every attempted query fails. Runs with partial results can succeed while recording unresolved queries in ERRORS.

How requests run

The first search request uses anonymous browserless HTTP with direct egress; no account or login is used. A blocked, challenged or malformed response is retried twice with fresh Apify datacenter identities. The rotated identity is retained for later pages. If those attempts fail, the default fallback opens headless Chromium, navigates to the same public JSON URL, and tries up to three times. Fallback pages use the same row contract and filters and are counted in COVERAGE.

Use 512 MB memory for both paths. Browser fallback launches Chromium, which adds time and compute per page; browserFallbackRequests in COVERAGE shows how often it ran. Decoded response bytes include accepted and rejected search responses. Compute and storage charges appear in the run's platform usage record; the Actor uses no residential proxy.

Billing

Each successfully emitted distinct author is one dataset item and one billable result. Duplicate posts, skipped authors, filtered posts and error records create no result charge. Each page's eligible authors are deduplicated and capped before a batch write. The platform clips a binding batch to its chargeable prefix; the Actor records only that delivered prefix and stops before another write, page or query. Each delivered author costs USD 0.001 on the Free plan, USD 0.00085 on Bronze, USD 0.00065 on Silver and USD 0.00025 on Gold, Platinum and Diamond. That is USD 1.00, 0.85, 0.65 and 0.25 per 1,000 delivered authors. Each run also charges USD 0.00005 per Actor Start event: one event per GB of allocated memory, at least one. A run that returns no authors still pays it.

Limits and troubleshooting

If results are empty, broaden the topic or remove language/date filters. Missing counts or links mean the anonymous response did not expose them. Ranked results can change between runs. For an unresolved request, inspect ERRORS and COVERAGE; share the run ID and input through this Actor's Issues tab.

This Actor covers topic-to-author research. Full articles, newsletter archive crawls, standalone profile scraping, publication-directory results, Notes, nested comments, contact extraction and historical growth are outside its coverage. It reads public metadata and does not retrieve subscriber-only bodies.

Substack's Terms of Use, captured 2026-10-07, prohibit crawling or scraping its pages and data. Consider those terms when deciding whether to use the exported information. This Actor is independently maintained and is not affiliated with Substack.