Beehiiv Newsletter Discovery Scraper

Discover and scrape newsletters from Beehiiv's public directory — the fastest-growing newsletter platform. Browse the full newsletter catalog, get detailed profiles for individual newsletters by URL or subdomain, or extract recent posts and publication history from any Beehiiv newsletter. No login or API key required.

What It Scrapes

Newsletter profiles:

• Name, subdomain, and canonical URL
• Description / tagline
• Avatar / logo image
• Content category (Technology, Business, Finance, etc.)
• Published post count

Post records:

• Title, URL, excerpt/preview text
• Publication date
• Author name
• Parent newsletter name and URL

Supported Modes

Mode	Description
discover	Browse Beehiiv's public newsletter directory; supports keyword search and category filter
getNewsletter	Fetch profile details for specific newsletters by URL or subdomain
searchPosts	Extract recent posts from one or more newsletters by URL or subdomain

Input Parameters

Field	Type	Description	Default
mode	select	Scraping mode	discover
startUrls	list	Newsletter URLs (e.g. https://milkroad.beehiiv.com)	—
subdomains	list	Beehiiv subdomains without .beehiiv.com (e.g. ["milkroad", "tldr"])	—
searchQuery	string	Keyword for discover mode	—
category	select	Filter by category (Technology, Business, Finance, etc.)	All
sortBy	select	Sort order: popular, newest, trending	popular
maxItems	integer	Maximum total records to collect (1–5000)	20
maxPostsPerNewsletter	integer	Max posts per newsletter in searchPosts mode (1–100)	10

Output Fields

Newsletter Record

Field	Type	Description
name	string	Newsletter name
subdomain	string	Beehiiv subdomain (e.g. milkroad)
url	string	Canonical newsletter URL
description	string	Newsletter description or tagline
category	string	Content category
avatarUrl	string	Newsletter logo or avatar image URL
postCount	integer	Total number of published posts
recordType	string	Always "newsletter"
scrapedAt	string	ISO 8601 scrape timestamp

Post Record

Field	Type	Description
title	string	Post title
url	string	Direct URL to the post
publishedAt	string	ISO 8601 publication date
excerpt	string	Post preview text or subtitle
author	string	Author name
newsletterName	string	Parent newsletter name
newsletterUrl	string	Parent newsletter URL
recordType	string	Always "post"
scrapedAt	string	ISO 8601 scrape timestamp

Example Output

Newsletter record:

{
  "name": "TLDR Newsletter",
  "subdomain": "tldr",
  "url": "https://tldr.beehiiv.com",
  "description": "5-minute daily newsletter covering the biggest stories in tech.",
  "category": "Technology",
  "avatarUrl": "https://cdn.beehiiv.com/tldr-logo.png",
  "postCount": 1250,
  "recordType": "newsletter",
  "scrapedAt": "2026-05-17T10:00:00+00:00"
}

Post record:

{
  "title": "OpenAI Releases GPT-5",
  "url": "https://tldr.beehiiv.com/p/openai-releases-gpt5",
  "publishedAt": "2026-04-01T08:00:00+00:00",
  "excerpt": "OpenAI announced GPT-5, its most capable model yet, featuring...",
  "author": "Dan Ni",
  "newsletterName": "TLDR Newsletter",
  "newsletterUrl": "https://tldr.beehiiv.com",
  "recordType": "post",
  "scrapedAt": "2026-05-17T10:00:00+00:00"
}

Frequently Asked Questions

Do I need a Beehiiv account or API key? No. This scraper uses Beehiiv's public discover page and publicly accessible newsletter pages. No authentication is required.

What newsletter categories are available? Technology, Business, Finance, Marketing, Health, Sports, Entertainment, Politics, Science, Food, Travel, Lifestyle, Education, and Other.

How do I scrape a specific newsletter? Use mode=getNewsletter and provide the newsletter URL in startUrls (e.g. https://milkroad.beehiiv.com), or just the subdomain in subdomains (e.g. milkroad).

Can I get all posts from a newsletter? Use mode=searchPosts with the newsletter URL or subdomain. Posts are discovered via the newsletter's sitemap.xml and post listing page. Set maxPostsPerNewsletter up to 100.

Does this work for newsletters on custom domains? Yes — if you know the full URL of a newsletter on a custom domain, provide it in startUrls. The scraper will fetch the page and extract metadata.

Why might some newsletters not have a description or post count? Not all newsletter homepages expose this information in their HTML meta tags. When a field cannot be reliably populated, it is omitted from the output rather than filled with a placeholder.

Limitations

• Beehiiv's discover page is Cloudflare-protected; the scraper uses curl_cffi with Chrome impersonation to bypass this.
• Newsletter subscriber counts are not publicly exposed — Beehiiv does not display them without login.
• Post content (full article body) is not scraped — only title, excerpt, URL, and date are collected.
• The discover page structure can change; the scraper uses multiple fallback strategies (Next.js data, inline JSON, href extraction) for maximum resilience.
• Rate limits may apply during heavy scraping; the scraper includes built-in retry logic and delays between requests.