Substack Scraper

Substack Scraper collects newsletter posts, public author profiles, publication information, and comments for newsletter research, creator discovery, and content monitoring. It works as a no-code Substack API alternative: you get structured rows for Substack newsletters, writers, and discussions without building your own integration. Start with a newsletter URL, a direct article link, an author profile, or a keyword. Export the resulting dataset to JSON, CSV, or a spreadsheet and join records using the stable identifiers returned with each row.

What Substack Scraper collects

A run can contain three kinds of records. Post records describe articles with their title, subtitle, publication date, author, newsletter, engagement counts, and whether the article is available to everyone or intended for paying subscribers. Profile records describe either a person or a publication, including the public biography, external links, and visible audience estimates. Comment records contain individual comments and replies, with identifiers connecting each reply to its parent and each comment to its post.

This structure lets you compare newsletters without placing comment threads inside article cells. Filter the dataset by record_type to work with articles, people, publications, or discussions separately. Every row also carries _type, input, and fetched_at, so you can identify its kind, the input that produced it, and when it was collected.

Substack Scraper use cases

• Newsletter research. Pull the archive of several Substack newsletters, then compare publishing frequency, likes, comments, and restacks per post to see which topics and formats resonate.
• Creator outreach and partnerships. Search for writers by keyword, collect their public bios, links, and audience estimates, and build a shortlist of Substack writers for sponsorships, guest posts, or podcast invitations.
• Content monitoring. Schedule a run with a recent date range to track new posts from competitors, industry analysts, or your own portfolio of newsletters, and feed them into a spreadsheet, Slack alert, or database.
• Audience and sentiment analysis. Collect comment threads as separate rows with their parent identifiers to study how readers react to a topic, which questions they ask, and which posts start discussions.
• Text analysis and AI pipelines. Enable public article text to build a corpus of free Substack posts for summarisation, topic modelling, or retrieval, with paid previews clearly labelled.

Input

Provide at least one value in urls or keywords. The urls list accepts newsletter homepages, Substack subdomains, bare subdomain names, custom newsletter domains, direct /p/ article links, public author handles prefixed with @, and profile URLs on Substack. Links in the form open.substack.com/pub/.../p/... and substack.com/home/post/p-... are also supported. An unrecognized entry produces a warning while valid entries continue. A completely invalid list returns an input error.

Set maxPostsPerNewsletter to control the number of matching posts collected from each newsletter. The default is 20, with a maximum of 5,000. Choose sort: "new" for recent posts or sort: "top" for the order Substack supplies as its top archive. Use startDate and endDate in YYYY-MM-DD form to select an inclusive UTC date range. Set onlyFree: true to exclude posts for paying subscribers. contentType accepts all, newsletter, podcast, or thread; availability depends on what the publication has posted.

Keyword searches use searchType: "posts" by default. Choose profiles to find people, or publications to discover newsletters associated with matching author profiles. Publication discovery is derived from those author matches, rather than an exhaustive catalogue of newsletters containing the search term. Set maxSearchResultsPerKeyword to limit primary results from each keyword; the default is 20. Additional publication profiles are separate records.

Optional article text is disabled by default. Enable includeContent to receive body_html and plain text. Enable includeComments to collect comments as separate rows, and use maxCommentsPerPost to set the limit, including replies. The default comment limit is 20. includePublicationInfo defaults to true and adds one profile per distinct publication encountered during the run.

Output

Posts use post_id and post_url as stable join keys. They include title, subtitle, posted_at, author_id, author_handle, publication_id, publication_name, publication_url, likes, comments_count, shares, tags, and public content when requested. shares reports Substack restacks. Archive posts and direct article links carry the newsletter's name and URL, including articles by guest authors and when includePublicationInfo is off. That option controls the additional publication profile record. If publication information cannot be retrieved, a warning explains the missing information while available post metadata is retained. Without requested content, text, body_html, and content_status are null; a short excerpt is stored separately and is never presented as full article text.

For content requests, content_status: "full" means the public response included the article text for a free article. content_status: "preview" means a paid article returned only its publicly accessible preview. This actor does not obtain subscriber-only text. A missing content response generates a warning and keeps the available article metadata.

Comments include comment_id, post_id, parent_comment_id, text, posted_at, author information, likes, and depth. A parent id lets you reconstruct a discussion tree. Deleted comments retain their identifier and may still have replies, but their text is null. Comments are limited in tree order, including descendants before the next root discussion. Some posts, including paid articles, show a comment count while their comments are not publicly available; the actor reports this in a warning when no comments can be returned.

Profiles have profile_kind: "user" or "publication", profile_url, profile_id, name, bio, links, and visible audience information. User followers and publication subscriber_count are different measurements. Publication followers repeats its subscriber estimate because newsletters have no separate follower count. Subscriber figures are rounded public estimates supplied by Substack; hidden counts remain null. They are suitable for audience comparisons, not precise subscriber accounting.

Example output

A post record from a real run with urls: ["https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit"] and includeContent: true, shortened to the main fields (the full record also contains body_html, text, bylines, cover_image, and more):

{
  "record_type": "post",
  "post_id": "215694124",
  "post_url": "https://www.lennysnewsletter.com/p/all-of-the-lenny-and-friends-summit",
  "title": "All of the Lenny & Friends Summit talks are now online!",
  "subtitle": "Plus, some reflections and takeaways from the day",
  "posted_at": "2026-09-29T13:15:57.512Z",
  "audience": "everyone",
  "is_paid": false,
  "wordcount": 1276,
  "likes": 277,
  "comments_count": 6,
  "shares": 4,
  "author_name": "Lenny Rachitsky",
  "author_handle": "lenny",
  "publication_name": "Lenny's Newsletter",
  "publication_url": "https://www.lennysnewsletter.com",
  "content_status": "full"
}

A paid article written by guest authors, from a run with a direct article link, includeContent: true, and includePublicationInfo: false. The newsletter's name and URL are still filled, and content_status shows that only the public preview was returned:

{
  "record_type": "post",
  "post_id": "216168140",
  "post_url": "https://www.lennysnewsletter.com/p/advanced-evals-how-to-find-and-fix",
  "title": "Advanced evals: How to find (and fix) hidden AI failures in your product",
  "posted_at": "2026-09-22T12:45:14.998Z",
  "audience": "only_paid",
  "is_paid": true,
  "likes": 347,
  "comments_count": 4,
  "shares": 20,
  "author_name": "Hamel Husain",
  "author_handle": "hamelhusain",
  "publication_id": "10845",
  "publication_name": "Lenny's Newsletter",
  "publication_url": "https://www.lennysnewsletter.com",
  "content_status": "preview"
}

A publication profile from a run with urls: ["garymarcus"]:

{
  "record_type": "profile",
  "profile_kind": "publication",
  "profile_id": "888615",
  "profile_url": "https://garymarcus.substack.com",
  "handle": "garymarcus",
  "name": "Marcus on AI",
  "subscriber_count": 119000,
  "subscriber_count_text": "Hundreds of thousands of subscribers",
  "paid_subscribers_text": "Thousands of paid subscribers",
  "author_name": "Gary Marcus",
  "created_at": "2022-05-14T14:09:01.902Z",
  "language": "en",
  "payments_enabled": true
}

Substack Scraper compared with other options

• Copying by hand works for one or two newsletters but does not scale to archives of hundreds of posts, and engagement counts change after publication.
• RSS feeds list recent posts of one newsletter, but they carry no like, comment, or restack counts, no author or publication profiles, and no comment threads. This actor returns all of those as structured rows and can go back up to 5,000 posts per newsletter.
• Open-source libraries for the unofficial Substack API need code, hosting, and maintenance whenever the site changes. This actor runs in the cloud, exports to JSON, CSV, or Excel, and plugs into Apify schedules, webhooks, and integrations.
• Other Substack scrapers in the Store differ in scope. This one combines posts, comments, writer profiles, publication profiles, keyword search, date filters, and optional public text in one run, separates each record type with record_type, and marks paid previews explicitly instead of presenting them as full articles.

Substack Scraper pricing

Charges correspond to delivered records, with separate events for posts, comments, profiles, and full public article content. A paid article preview does not count as full content. The run-start event is charged once per actor run, including a run that finds no records. Empty results carry no record or content charges. Consult the actor's current pricing panel for event rates.

Non-paying users receive at most ten dataset rows in total. Posts, profiles, and comments share that allowance. A newsletter profile and its discussion can therefore use part of the allowance before ten posts have been returned. Disable optional comments or publication information when you want the allowance to focus on articles.

Larger collections and monitoring

The actor follows continuation automatically to collect larger requests. If the run time limit interrupts a collection, the output reports that it is partial and includes the continuation needed to resume with the same input. For scheduled monitoring, use a narrow date range and store post_id or comment_id in your destination to detect new records across separate runs. A run removes repeated post and profile identifiers within its own collection; it does not remember your previous runs.

Public URLs can disappear or become unavailable. Warnings identify entries or optional phases that did not complete while other records remain available. A run with no matching public data finishes with an empty result status. The actor returns only public source data and never substitutes fabricated profiles or article text for an unavailable source.

FAQ

Can I collect paid newsletter articles?

You can collect their public metadata and public preview. Subscriber-only text is outside the available output. The is_paid and content_status fields make that distinction explicit in each post record.

Does publication search find every matching newsletter?

No. It derives publication records from matching author profiles. A newsletter whose author does not match the keyword may be absent. Use its direct publication URL when you already know the newsletter you need.

Are subscriber numbers exact?

No. Substack publishes rounded estimates and sometimes hides them. The actor preserves the visible estimate and its descriptive text, and returns null when the count is unavailable.

Can I combine several newsletters and keywords?

Yes. Include multiple entries in the two input lists. Limits apply per publication or keyword, while repeated post and profile records are removed across the whole run. The originating input remains on each delivered row.

Is this the official Substack API?

No. This actor is not affiliated with Substack. It reads the same public pages and data a visitor sees in a browser and returns them as structured records, so it only covers public information and needs no Substack account.

How do I find Substack newsletters about a topic?

Add a keyword and set searchType to publications to discover newsletters through matching writers, or posts to find articles mentioning the term. Then pass the newsletters you want into urls to collect their full archive.

Can I export the results to CSV or Excel?

Yes. Every run stores its rows in an Apify dataset that you can download as JSON, CSV, Excel, or XML, or read through the Apify API and integrations.