Substack Scraper (Posts, Publications, Comments)
Pricing
from $1.00 / 1,000 results
Posts from any Substack newsletter, including custom domains: title, authors, date, paywall flag, reactions, comments, restacks and optional full text. Find newsletters by search or category. Bounded retries; failed publications are never charged.
One newsletter per line: its Substack name (the part before .substack.com, e.g. example), a name.substack.com address, a custom-domain URL, or any post URL on it.
Keywords to find newsletters with Substack's publication search. Each match is scraped like a publication you listed. The number of matches per query is capped by 'Publications per search or category'.
Substack category ids whose top publications to scrape, e.g. 4 (Technology), 62 (Business), 153 (Finance), 134 (Science), 96 (Culture). Capped by 'Publications per search or category'.
How many publications each search query or category adds to the run.
Stop after this many posts for each publication. Set 0 with 'Add a publication row' to collect publication info only.
Newest first, or the publication's own 'Top' ranking (most engagement).
Only posts published on or after this date (YYYY-MM-DD, UTC). With 'Newest' the run stops paging as soon as it passes the date; with 'Top' it scans at most 10x the post cap (at least 500 posts).
One extra request per post. Adds bodyHtml and bodyText. Paid-only posts return just the free preview Substack shows to logged-out readers; those rows have isPreviewOnly set.
One extra request per post that has comments. Adds a row per comment (type: comment), replies included, up to 'Maximum comments per post'.
Cap on comment rows per post, counting replies, in Substack's 'best first' order.
Adds one row per publication (type: publication) with its name, description, author, subscriber tier as Substack displays it, language and dates.
Hard cap on dataset rows (posts, comments and publication rows together) for the whole run. Leave empty for no cap.
Publications fetched at the same time. Higher is faster but more likely to be rate-limited.
Each attempt uses a fresh proxy session and resumes where the last one stopped. Publications that still fail are listed in FAILED_PUBLICATIONS and are not charged. Unknown publications fail at once.
How long to wait for a single HTTP response before the attempt counts as failed and is retried on a fresh session.
