# Beehiiv Newsletter Discovery Scraper (`crawlerbros/beehiiv-newsletter-scraper`) Actor Discover and scrape newsletters from Beehiiv's public directory. Browse the full newsletter catalog, get detailed newsletter profiles by URL or subdomain, or extract recent posts from any Beehiiv newsletter. No login required - **URL**: https://apify.com/crawlerbros/beehiiv-newsletter-scraper.md - **Developed by:** [Crawler Bros](https://apify.com/crawlerbros) (community) - **Categories:** News, Developer tools, Automation - **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 7 bookmarks - **User rating**: 5.00 out of 5 stars ## Pricing from $3.00 / 1,000 results This Actor is paid per event and usage. You are charged both the fixed price for specific events and for Apify platform usage. Since this Actor supports Apify Store discounts, the price gets lower the higher subscription plan you have. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Beehiiv Newsletter Discovery Scraper Discover and scrape newsletters from [Beehiiv's](https://www.beehiiv.com) public directory — the fastest-growing newsletter platform. Browse the full newsletter catalog, get detailed profiles for individual newsletters by URL or subdomain, or extract recent posts and publication history from any Beehiiv newsletter. No login or API key required. ### What It Scrapes **Newsletter profiles:** - Name, subdomain, and canonical URL - Description / tagline - Avatar / logo image - Content category (Technology, Business, Finance, etc.) - Published post count **Post records:** - Title, URL, excerpt/preview text - Publication date - Author name - Parent newsletter name and URL ### Supported Modes | Mode | Description | |------|-------------| | `discover` | Browse Beehiiv's public newsletter directory; supports keyword search and category filter | | `getNewsletter` | Fetch profile details for specific newsletters by URL or subdomain | | `searchPosts` | Extract recent posts from one or more newsletters by URL or subdomain | ### Input Parameters | Field | Type | Description | Default | |-------|------|-------------|---------| | `mode` | select | Scraping mode | `discover` | | `startUrls` | list | Newsletter URLs (e.g. `https://milkroad.beehiiv.com`) | — | | `subdomains` | list | Beehiiv subdomains without `.beehiiv.com` (e.g. `["milkroad", "tldr"]`) | — | | `searchQuery` | string | Keyword for discover mode | — | | `category` | select | Filter by category (Technology, Business, Finance, etc.) | All | | `sortBy` | select | Sort order: popular, newest, trending | `popular` | | `maxItems` | integer | Maximum total records to collect (1–5000) | `20` | | `maxPostsPerNewsletter` | integer | Max posts per newsletter in searchPosts mode (1–100) | `10` | ### Output Fields #### Newsletter Record | Field | Type | Description | |-------|------|-------------| | `name` | string | Newsletter name | | `subdomain` | string | Beehiiv subdomain (e.g. `milkroad`) | | `url` | string | Canonical newsletter URL | | `description` | string | Newsletter description or tagline | | `category` | string | Content category | | `avatarUrl` | string | Newsletter logo or avatar image URL | | `postCount` | integer | Total number of published posts | | `recordType` | string | Always `"newsletter"` | | `scrapedAt` | string | ISO 8601 scrape timestamp | #### Post Record | Field | Type | Description | |-------|------|-------------| | `title` | string | Post title | | `url` | string | Direct URL to the post | | `publishedAt` | string | ISO 8601 publication date | | `excerpt` | string | Post preview text or subtitle | | `author` | string | Author name | | `newsletterName` | string | Parent newsletter name | | `newsletterUrl` | string | Parent newsletter URL | | `recordType` | string | Always `"post"` | | `scrapedAt` | string | ISO 8601 scrape timestamp | ### Example Output **Newsletter record:** ```json { "name": "TLDR Newsletter", "subdomain": "tldr", "url": "https://tldr.beehiiv.com", "description": "5-minute daily newsletter covering the biggest stories in tech.", "category": "Technology", "avatarUrl": "https://cdn.beehiiv.com/tldr-logo.png", "postCount": 1250, "recordType": "newsletter", "scrapedAt": "2026-05-17T10:00:00+00:00" } ```` **Post record:** ```json { "title": "OpenAI Releases GPT-5", "url": "https://tldr.beehiiv.com/p/openai-releases-gpt5", "publishedAt": "2026-04-01T08:00:00+00:00", "excerpt": "OpenAI announced GPT-5, its most capable model yet, featuring...", "author": "Dan Ni", "newsletterName": "TLDR Newsletter", "newsletterUrl": "https://tldr.beehiiv.com", "recordType": "post", "scrapedAt": "2026-05-17T10:00:00+00:00" } ``` ### Frequently Asked Questions **Do I need a Beehiiv account or API key?** No. This scraper uses Beehiiv's public discover page and publicly accessible newsletter pages. No authentication is required. **What newsletter categories are available?** Technology, Business, Finance, Marketing, Health, Sports, Entertainment, Politics, Science, Food, Travel, Lifestyle, Education, and Other. **How do I scrape a specific newsletter?** Use `mode=getNewsletter` and provide the newsletter URL in `startUrls` (e.g. `https://milkroad.beehiiv.com`), or just the subdomain in `subdomains` (e.g. `milkroad`). **Can I get all posts from a newsletter?** Use `mode=searchPosts` with the newsletter URL or subdomain. Posts are discovered via the newsletter's sitemap.xml and post listing page. Set `maxPostsPerNewsletter` up to 100. **Does this work for newsletters on custom domains?** Yes — if you know the full URL of a newsletter on a custom domain, provide it in `startUrls`. The scraper will fetch the page and extract metadata. **Why might some newsletters not have a description or post count?** Not all newsletter homepages expose this information in their HTML meta tags. When a field cannot be reliably populated, it is omitted from the output rather than filled with a placeholder. ### Limitations - Beehiiv's discover page is Cloudflare-protected; the scraper uses curl\_cffi with Chrome impersonation to bypass this. - Newsletter subscriber counts are not publicly exposed — Beehiiv does not display them without login. - Post content (full article body) is not scraped — only title, excerpt, URL, and date are collected. - The discover page structure can change; the scraper uses multiple fallback strategies (Next.js data, inline JSON, href extraction) for maximum resilience. - Rate limits may apply during heavy scraping; the scraper includes built-in retry logic and delays between requests. # Actor input Schema ## `mode` (type: `string`): What to scrape from Beehiiv. ## `startUrls` (type: `array`): Full URLs of Beehiiv newsletters to scrape, e.g. https://milkroad.beehiiv.com or https://tldrnewsletter.com. ## `subdomains` (type: `array`): Beehiiv newsletter subdomains without .beehiiv.com, e.g. \["milkroad", "tldr"]. Will be resolved to https://subdomain.beehiiv.com. ## `searchQuery` (type: `string`): Keyword to search for in the Beehiiv newsletter directory. ## `category` (type: `string`): Filter newsletters by topic category. ## `sortBy` (type: `string`): How to sort discovered newsletters. ## `maxItems` (type: `integer`): Maximum total records to collect (newsletters or posts). ## `maxPostsPerNewsletter` (type: `integer`): Maximum number of posts to collect from each newsletter. ## Actor input object example ```json { "mode": "discover", "startUrls": [], "subdomains": [], "category": "", "sortBy": "popular", "maxItems": 20, "maxPostsPerNewsletter": 10 } ``` # Actor output Schema ## `newsletters_and_posts` (type: `string`): Dataset containing all scraped Beehiiv newsletter profiles and post records. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "mode": "discover", "startUrls": [], "subdomains": [], "category": "", "sortBy": "popular", "maxItems": 20, "maxPostsPerNewsletter": 10 }; // Run the Actor and wait for it to finish const run = await client.actor("crawlerbros/beehiiv-newsletter-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "mode": "discover", "startUrls": [], "subdomains": [], "category": "", "sortBy": "popular", "maxItems": 20, "maxPostsPerNewsletter": 10, } # Run the Actor and wait for it to finish run = client.actor("crawlerbros/beehiiv-newsletter-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "mode": "discover", "startUrls": [], "subdomains": [], "category": "", "sortBy": "popular", "maxItems": 20, "maxPostsPerNewsletter": 10 }' | apify call crawlerbros/beehiiv-newsletter-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=crawlerbros/beehiiv-newsletter-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Beehiiv Newsletter Discovery Scraper", "description": "Discover and scrape newsletters from Beehiiv's public directory. Browse the full newsletter catalog, get detailed newsletter profiles by URL or subdomain, or extract recent posts from any Beehiiv newsletter. No login required", "version": "1.0", "x-build-id": "ucefyMCcthn1sdk5h" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/crawlerbros~beehiiv-newsletter-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-crawlerbros-beehiiv-newsletter-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/crawlerbros~beehiiv-newsletter-scraper/runs": { "post": { "operationId": "runs-sync-crawlerbros-beehiiv-newsletter-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/crawlerbros~beehiiv-newsletter-scraper/run-sync": { "post": { "operationId": "run-sync-crawlerbros-beehiiv-newsletter-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "required": [ "mode" ], "properties": { "mode": { "title": "Mode", "enum": [ "discover", "getNewsletter", "searchPosts" ], "type": "string", "description": "What to scrape from Beehiiv.", "default": "discover" }, "startUrls": { "title": "Newsletter URLs (mode=getNewsletter or searchPosts)", "type": "array", "description": "Full URLs of Beehiiv newsletters to scrape, e.g. https://milkroad.beehiiv.com or https://tldrnewsletter.com.", "default": [], "items": { "type": "string" } }, "subdomains": { "title": "Beehiiv subdomains (mode=getNewsletter or searchPosts)", "type": "array", "description": "Beehiiv newsletter subdomains without .beehiiv.com, e.g. [\"milkroad\", \"tldr\"]. Will be resolved to https://subdomain.beehiiv.com.", "default": [], "items": { "type": "string" } }, "searchQuery": { "title": "Search query (mode=discover)", "type": "string", "description": "Keyword to search for in the Beehiiv newsletter directory." }, "category": { "title": "Category (mode=discover)", "enum": [ "", "Technology", "Business", "Finance", "Marketing", "Health", "Sports", "Entertainment", "Politics", "Science", "Food", "Travel", "Lifestyle", "Education", "Entrepreneurship", "Investing", "News", "Other" ], "type": "string", "description": "Filter newsletters by topic category.", "default": "" }, "sortBy": { "title": "Sort by (mode=discover)", "enum": [ "popular", "newest", "trending" ], "type": "string", "description": "How to sort discovered newsletters.", "default": "popular" }, "maxItems": { "title": "Max items", "minimum": 1, "maximum": 5000, "type": "integer", "description": "Maximum total records to collect (newsletters or posts).", "default": 20 }, "maxPostsPerNewsletter": { "title": "Max posts per newsletter (mode=searchPosts)", "minimum": 1, "maximum": 100, "type": "integer", "description": "Maximum number of posts to collect from each newsletter.", "default": 10 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```