# Beehiiv Newsletter Scraper - Posts & Authors (`elliotpadfield/beehiiv-newsletter-scraper`) Actor Scrape public Beehiiv newsletters by publication URL, custom domain, sitemap, or post URL. Extract posts, authors, full text, HTML, markdown, images, outbound links, sponsor links, and publication metadata. - **URL**: https://apify.com/elliotpadfield/beehiiv-newsletter-scraper.md - **Developed by:** [Elliot Padfield](https://apify.com/elliotpadfield) (community) - **Categories:** Automation, Lead generation, News - **Stats:** 1 total users, 0 monthly users, 100.0% runs succeeded, NaN bookmarks - **User rating**: No ratings yet ## Pricing from $5.00 / 1,000 results This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events. Learn more: https://docs.apify.com/platform/actors/running/actors-in-store#pay-per-event ## What's an Apify Actor? Actors are a software tools running on the Apify platform, for all kinds of web data extraction and automation use cases. In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours, and optionally produces a well-defined JSON output, datasets with results, or files in key-value store. In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server. Actors are written with capital "A". ## How to integrate an Actor? If asked about integration, you help developers integrate Actors into their projects. You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready. The best way to integrate Actors is as follows. In JavaScript/TypeScript projects, use official [JavaScript/TypeScript client](https://docs.apify.com/api/client/js.md): ```bash npm install apify-client ``` In Python projects, use official [Python client library](https://docs.apify.com/api/client/python.md): ```bash pip install apify-client ``` In shell scripts, use [Apify CLI](https://docs.apify.com/cli/docs.md): ````bash # MacOS / Linux curl -fsSL https://apify.com/install-cli.sh | bash # Windows irm https://apify.com/install-cli.ps1 | iex ```bash In AI frameworks, you might use the [Apify MCP server](https://docs.apify.com/platform/integrations/mcp.md). If your project is in a different language, use the [REST API](https://docs.apify.com/api/v2.md). For usage examples, see the [API](#api) section below. For more details, see Apify documentation as [Markdown index](https://docs.apify.com/llms.txt) and [Markdown full-text](https://docs.apify.com/llms-full.txt). # README ## Beehiiv Newsletter Scraper - Posts & Authors Scrape public Beehiiv newsletters by publication URL, custom domain, sitemap, or direct post URL. This Actor extracts structured Beehiiv post data including titles, descriptions, publish dates, authors, publication metadata, full article text, HTML, markdown-like text, images, outbound links, sponsor-looking links, word count, and reading time. Use it to monitor newsletters, research newsletter sponsorships, build content datasets, track competitor publishing, analyze Beehiiv creators, collect newsletter archives, and export Beehiiv post data to CSV, JSON, Excel, Google Sheets, Make, Zapier, or your own API workflow. ### What can this Beehiiv scraper do? - Scrape any public Beehiiv publication URL or custom domain - Discover posts from Beehiiv publication sitemaps - Scrape direct Beehiiv post URLs - Extract full post content as HTML, clean text, and markdown-like text - Extract author name, author URL, bio, image, and social links when available - Extract publication name, URL, ID, logo, image, and social links when available - Extract primary image and body images - Extract outbound links from the article body - Flag sponsor, affiliate, referral, and campaign-looking URLs - Filter saved posts by keyword - Filter saved posts by published date - Deduplicate posts across publications, sitemaps, and direct URLs - Export structured post data to Apify datasets - Run on schedules for newsletter monitoring - Use Apify residential proxies on every run for production reliability ### What data can you extract from Beehiiv? | Field | Description | |---|---| | `publicationName` | Beehiiv publication name | | `publicationUrl` | Publication URL from structured metadata | | `publicationId` | Beehiiv publication identifier when available | | `publicationLogoUrl` | Publication logo URL | | `publicationSocialUrls` | Publication social profile URLs | | `postId` | Beehiiv post identifier when available | | `postUrl` | URL fetched by the Actor | | `canonicalUrl` | Canonical Beehiiv post URL | | `slug` | Post slug | | `title` | Post title | | `description` | Post description or excerpt | | `authorName` | Author name | | `authorUrl` | Author URL | | `authorDescription` | Author bio or description | | `authorImageUrl` | Author image URL | | `datePublished` | Published timestamp | | `dateModified` | Last modified timestamp | | `isAccessibleForFree` | Public/free flag from structured metadata | | `imageUrl` | Primary post image | | `tags` | Tags detected from Beehiiv tag links | | `html` | Full article body HTML | | `text` | Clean article body text | | `markdown` | Markdown-like article text for AI and analysis workflows | | `imageUrls` | Primary and embedded image URLs | | `outboundUrls` | Links found in the article body | | `sponsorUrls` | Sponsor, affiliate, referral, or campaign-looking links | | `wordCount` | Article word count | | `readingTimeMinutes` | Estimated reading time | | `matchedKeywords` | Keywords that matched the saved post | | `contentFetched` | Whether the Actor found and extracted a full article body | | `scrapedAt` | Timestamp when the row was saved | ### How to scrape Beehiiv newsletters 1. Add one or more Beehiiv publication URLs, archive URLs, custom domains, sitemap URLs, or direct post URLs. 2. Set `maxPosts` to control how many posts to save. 3. Add `keywords`, `dateFrom`, or `dateTo` if you only want matching posts. 4. Keep `includeFullContent`, `includeImages`, and `includeLinks` enabled for the richest dataset. 5. Run the Actor and export the dataset in JSON, CSV, Excel, XML, RSS, or HTML from Apify. ### Input examples #### Scrape a Beehiiv publication ```json { "publicationUrls": ["https://product.beehiiv.com"], "maxPosts": 100, "includeFullContent": true, "includeImages": true, "includeLinks": true } ```` #### Scrape a custom domain and filter by keyword ```json { "publicationUrls": ["https://www.example-newsletter.com"], "keywords": ["AI", "funding", "sponsor"], "dateFrom": "2026-01-01", "maxPosts": 250 } ``` #### Enrich specific Beehiiv post URLs ```json { "postUrls": [ "https://product.beehiiv.com/p/beehiiv-mcp-v2" ], "includeFullContent": true } ``` ### Output example ```json { "sourceType": "publication", "sourceValue": "https://product.beehiiv.com", "publicationName": "beehiiv Product Updates", "publicationUrl": "https://product.beehiiv.com/", "postId": "c5f6f5e5-...", "postUrl": "https://product.beehiiv.com/p/beehiiv-mcp-v2", "canonicalUrl": "https://product.beehiiv.com/p/beehiiv-mcp-v2", "slug": "beehiiv-mcp-v2", "title": "Introducing beehiiv MCP v2", "description": "A product update from beehiiv.", "authorName": "beehiiv", "datePublished": "2026-05-20T12:00:00.000Z", "imageUrl": "https://media.beehiiv.com/...", "text": "Full article text...", "markdown": "Full article text...", "outboundUrls": ["https://www.beehiiv.com/..."], "sponsorUrls": [], "wordCount": 742, "readingTimeMinutes": 4, "scrapedAt": "2026-05-28T10:15:00.000Z" } ``` ### Search methods and filters | Capability | Supported | |---|---| | Publication URL discovery | Yes | | Beehiiv custom domains | Yes | | Direct sitemap URL scraping | Yes | | Direct post URL enrichment | Yes | | Keyword filtering | Yes | | Date range filtering | Yes | | Full HTML extraction | Yes | | Clean text extraction | Yes | | Markdown-like text extraction | Yes | | Author metadata | Yes | | Publication metadata | Yes | | Image extraction | Yes | | Outbound link extraction | Yes | | Sponsor or affiliate link detection | Yes | | Word count and reading time | Yes | | Deduplication across inputs | Yes | | Forced Apify Residential Proxy | Yes | ### Pricing This Actor is designed for pay-per-result pricing. Each saved Beehiiv post is one billable result. | Result type | What counts as one result | |---|---| | Beehiiv post | One saved post row after deduplication, keyword filtering, and date filtering | A typical run can scrape the latest 100 Beehiiv posts from a publication in a few minutes. Failed post fetches, duplicate URLs, and posts filtered out by keyword/date are not saved as dataset items. The Actor stops saving new rows when the Apify pay-per-result charge limit is reached. The Actor always uses Apify residential proxies. For small tests, lower `maxPosts` to 10 or 25. For scheduled monitoring, run daily or weekly with the same publication inputs and deduplicate by `postId` or `canonicalUrl` in your downstream workflow. ### Reliability notes The Actor is fault tolerant across sources and posts. If one sitemap or post URL is blocked, deleted, or returns a Cloudflare challenge, the run logs the failure and continues with the remaining inputs. Article body extraction uses Beehiiv's common content container first, then falls back to broader article and main-content selectors; rows where metadata is available but full article text is not will have `contentFetched: false`. ### Why use this Actor? Beehiiv newsletters are useful for content research, sponsorship intelligence, creator discovery, competitor monitoring, and AI-ready content analysis. This scraper helps answer questions like: - What has a Beehiiv publication published recently? - Which authors write for a newsletter? - Which outbound links, sponsors, and affiliate campaigns appear in posts? - Which newsletters mention a brand, topic, product, or competitor? - How long are posts, and how frequently does a publication publish? - Which Beehiiv posts are best suited for content analysis or lead research? Because it runs on Apify, you also get scheduling, API access, datasets, webhooks, proxy rotation, and integrations without maintaining your own server. # Actor input Schema ## `publicationUrls` (type: `array`): Beehiiv publication homepages, archive pages, or custom domains. The Actor discovers posts through each publication sitemap. ## `urls` (type: `array`): Alias for Publication URLs or Domains. Useful when triggering the Actor from API clients that use a generic urls field. ## `sitemapUrls` (type: `array`): Direct Beehiiv sitemap URLs. Use this when you already know the exact sitemap to process. ## `postUrls` (type: `array`): Public Beehiiv post URLs to enrich directly. ## `keywords` (type: `array`): Only save posts whose title, description, author, publication name, or extracted article text contains one of these terms. ## `dateFrom` (type: `string`): Only save posts published on or after this date. Use YYYY-MM-DD. ## `dateTo` (type: `string`): Only save posts published on or before this date. Use YYYY-MM-DD. ## `maxPosts` (type: `integer`): Maximum number of unique Beehiiv posts to save across all inputs. ## `includeFullContent` (type: `boolean`): Extract article HTML, clean text, markdown-like text, word count, and reading time. ## `includeImages` (type: `boolean`): Extract the primary post image and images embedded in the article body. ## `includeLinks` (type: `boolean`): Extract outbound article links and flag sponsor or affiliate-looking links. ## `maxConcurrency` (type: `integer`): Number of Beehiiv post pages to fetch in parallel. ## `requestDelayMs` (type: `integer`): Delay between retry attempts in milliseconds. Higher values can help on large runs. ## `maxRetries` (type: `integer`): Retry count for transient HTTP failures, proxy failures, and target-side 429/5xx responses. ## Actor input object example ```json { "publicationUrls": [ "https://product.beehiiv.com" ], "dateFrom": "2026-01-01", "dateTo": "2026-05-28", "maxPosts": 100, "includeFullContent": true, "includeImages": true, "includeLinks": true, "maxConcurrency": 8, "requestDelayMs": 250, "maxRetries": 3 } ``` # Actor output Schema ## `results` (type: `string`): Public Beehiiv posts with publication metadata, author details, full article content, images, links, sponsor links, and dates stored in the default dataset. # API You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup. ## JavaScript example ```javascript import { ApifyClient } from 'apify-client'; // Initialize the ApifyClient with your Apify API token // Replace the '' with your token const client = new ApifyClient({ token: '', }); // Prepare Actor input const input = { "publicationUrls": [ "https://product.beehiiv.com" ], "maxPosts": 100, "includeFullContent": true, "includeImages": true, "includeLinks": true, "maxConcurrency": 8, "requestDelayMs": 250, "maxRetries": 3 }; // Run the Actor and wait for it to finish const run = await client.actor("elliotpadfield/beehiiv-newsletter-scraper").call(input); // Fetch and print Actor results from the run's dataset (if any) console.log('Results from dataset'); console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`); const { items } = await client.dataset(run.defaultDatasetId).listItems(); items.forEach((item) => { console.dir(item); }); // 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs ``` ## Python example ```python from apify_client import ApifyClient # Initialize the ApifyClient with your Apify API token # Replace '' with your token. client = ApifyClient("") # Prepare the Actor input run_input = { "publicationUrls": ["https://product.beehiiv.com"], "maxPosts": 100, "includeFullContent": True, "includeImages": True, "includeLinks": True, "maxConcurrency": 8, "requestDelayMs": 250, "maxRetries": 3, } # Run the Actor and wait for it to finish run = client.actor("elliotpadfield/beehiiv-newsletter-scraper").call(run_input=run_input) # Fetch and print Actor results from the run's dataset (if there are any) print("💾 Check your data here: https://console.apify.com/storage/datasets/" + run["defaultDatasetId"]) for item in client.dataset(run["defaultDatasetId"]).iterate_items(): print(item) # 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start ``` ## CLI example ```bash echo '{ "publicationUrls": [ "https://product.beehiiv.com" ], "maxPosts": 100, "includeFullContent": true, "includeImages": true, "includeLinks": true, "maxConcurrency": 8, "requestDelayMs": 250, "maxRetries": 3 }' | apify call elliotpadfield/beehiiv-newsletter-scraper --silent --output-dataset ``` ## MCP server setup ```json { "mcpServers": { "apify": { "command": "npx", "args": [ "mcp-remote", "https://mcp.apify.com/?tools=elliotpadfield/beehiiv-newsletter-scraper", "--header", "Authorization: Bearer " ] } } } ``` ## OpenAPI specification ```json { "openapi": "3.0.1", "info": { "title": "Beehiiv Newsletter Scraper - Posts & Authors", "description": "Scrape public Beehiiv newsletters by publication URL, custom domain, sitemap, or post URL. Extract posts, authors, full text, HTML, markdown, images, outbound links, sponsor links, and publication metadata.", "version": "1.1", "x-build-id": "5FoAKU11rIgyclcmT" }, "servers": [ { "url": "https://api.apify.com/v2" } ], "paths": { "/acts/elliotpadfield~beehiiv-newsletter-scraper/run-sync-get-dataset-items": { "post": { "operationId": "run-sync-get-dataset-items-elliotpadfield-beehiiv-newsletter-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for its completion, and returns Actor's dataset items in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } }, "/acts/elliotpadfield~beehiiv-newsletter-scraper/runs": { "post": { "operationId": "runs-sync-elliotpadfield-beehiiv-newsletter-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor and returns information about the initiated run in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK", "content": { "application/json": { "schema": { "$ref": "#/components/schemas/runsResponseSchema" } } } } } } }, "/acts/elliotpadfield~beehiiv-newsletter-scraper/run-sync": { "post": { "operationId": "run-sync-elliotpadfield-beehiiv-newsletter-scraper", "x-openai-isConsequential": false, "summary": "Executes an Actor, waits for completion, and returns the OUTPUT from Key-value store in response.", "tags": [ "Run Actor" ], "requestBody": { "required": true, "content": { "application/json": { "schema": { "$ref": "#/components/schemas/inputSchema" } } } }, "parameters": [ { "name": "token", "in": "query", "required": true, "schema": { "type": "string" }, "description": "Enter your Apify token here" } ], "responses": { "200": { "description": "OK" } } } } }, "components": { "schemas": { "inputSchema": { "type": "object", "properties": { "publicationUrls": { "title": "Publication URLs or Domains", "uniqueItems": true, "type": "array", "description": "Beehiiv publication homepages, archive pages, or custom domains. The Actor discovers posts through each publication sitemap.", "items": { "type": "string" }, "default": [ "https://product.beehiiv.com" ] }, "urls": { "title": "URLs", "uniqueItems": true, "type": "array", "description": "Alias for Publication URLs or Domains. Useful when triggering the Actor from API clients that use a generic urls field.", "items": { "type": "string" } }, "sitemapUrls": { "title": "Sitemap URLs", "uniqueItems": true, "type": "array", "description": "Direct Beehiiv sitemap URLs. Use this when you already know the exact sitemap to process.", "items": { "type": "string" } }, "postUrls": { "title": "Direct Post URLs", "uniqueItems": true, "type": "array", "description": "Public Beehiiv post URLs to enrich directly.", "items": { "type": "string" } }, "keywords": { "title": "Keyword Filter", "uniqueItems": true, "type": "array", "description": "Only save posts whose title, description, author, publication name, or extracted article text contains one of these terms.", "items": { "type": "string" } }, "dateFrom": { "title": "Start Date", "pattern": "^\\d{4}-\\d{2}-\\d{2}$", "type": "string", "description": "Only save posts published on or after this date. Use YYYY-MM-DD." }, "dateTo": { "title": "End Date", "pattern": "^\\d{4}-\\d{2}-\\d{2}$", "type": "string", "description": "Only save posts published on or before this date. Use YYYY-MM-DD." }, "maxPosts": { "title": "Maximum Posts", "minimum": 1, "maximum": 10000, "type": "integer", "description": "Maximum number of unique Beehiiv posts to save across all inputs.", "default": 100 }, "includeFullContent": { "title": "Include Full Content", "type": "boolean", "description": "Extract article HTML, clean text, markdown-like text, word count, and reading time.", "default": true }, "includeImages": { "title": "Include Images", "type": "boolean", "description": "Extract the primary post image and images embedded in the article body.", "default": true }, "includeLinks": { "title": "Include Links", "type": "boolean", "description": "Extract outbound article links and flag sponsor or affiliate-looking links.", "default": true }, "maxConcurrency": { "title": "Maximum Concurrency", "minimum": 1, "maximum": 30, "type": "integer", "description": "Number of Beehiiv post pages to fetch in parallel.", "default": 8 }, "requestDelayMs": { "title": "Request Delay", "minimum": 0, "maximum": 10000, "type": "integer", "description": "Delay between retry attempts in milliseconds. Higher values can help on large runs.", "default": 250 }, "maxRetries": { "title": "Maximum Retries", "minimum": 0, "maximum": 8, "type": "integer", "description": "Retry count for transient HTTP failures, proxy failures, and target-side 429/5xx responses.", "default": 3 } } }, "runsResponseSchema": { "type": "object", "properties": { "data": { "type": "object", "properties": { "id": { "type": "string" }, "actId": { "type": "string" }, "userId": { "type": "string" }, "startedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "finishedAt": { "type": "string", "format": "date-time", "example": "2025-01-08T00:00:00.000Z" }, "status": { "type": "string", "example": "READY" }, "meta": { "type": "object", "properties": { "origin": { "type": "string", "example": "API" }, "userAgent": { "type": "string" } } }, "stats": { "type": "object", "properties": { "inputBodyLen": { "type": "integer", "example": 2000 }, "rebootCount": { "type": "integer", "example": 0 }, "restartCount": { "type": "integer", "example": 0 }, "resurrectCount": { "type": "integer", "example": 0 }, "computeUnits": { "type": "integer", "example": 0 } } }, "options": { "type": "object", "properties": { "build": { "type": "string", "example": "latest" }, "timeoutSecs": { "type": "integer", "example": 300 }, "memoryMbytes": { "type": "integer", "example": 1024 }, "diskMbytes": { "type": "integer", "example": 2048 } } }, "buildId": { "type": "string" }, "defaultKeyValueStoreId": { "type": "string" }, "defaultDatasetId": { "type": "string" }, "defaultRequestQueueId": { "type": "string" }, "buildNumber": { "type": "string", "example": "1.0.0" }, "containerUrl": { "type": "string" }, "usage": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "integer", "example": 1 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } }, "usageTotalUsd": { "type": "number", "example": 0.00005 }, "usageUsd": { "type": "object", "properties": { "ACTOR_COMPUTE_UNITS": { "type": "integer", "example": 0 }, "DATASET_READS": { "type": "integer", "example": 0 }, "DATASET_WRITES": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_READS": { "type": "integer", "example": 0 }, "KEY_VALUE_STORE_WRITES": { "type": "number", "example": 0.00005 }, "KEY_VALUE_STORE_LISTS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_READS": { "type": "integer", "example": 0 }, "REQUEST_QUEUE_WRITES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_INTERNAL_GBYTES": { "type": "integer", "example": 0 }, "DATA_TRANSFER_EXTERNAL_GBYTES": { "type": "integer", "example": 0 }, "PROXY_RESIDENTIAL_TRANSFER_GBYTES": { "type": "integer", "example": 0 }, "PROXY_SERPS": { "type": "integer", "example": 0 } } } } } } } } } } ```