Beehiiv Newsletter Scraper

Extract public Beehiiv newsletter posts from publication homepages, archives, sitemaps, and direct post URLs. Get titles, authors, dates, access status, reading time, full public text, optional HTML, images, tags, and outbound links without a Beehiiv API key.

Use the Actor for scheduled newsletter monitoring, competitive intelligence, sponsorship research, content analysis, and retrieval-augmented generation (RAG) datasets. It uses fast HTTP requests and Beehiiv's public sitemap and server-rendered post data rather than launching a browser for every page.

Premium posts are never unlocked. When Beehiiv exposes public preview metadata for a gated post, the Actor records that metadata and marks the post as paywalled without returning its protected body.

What does Beehiiv Newsletter Scraper do?

Beehiiv Newsletter Scraper turns one or many Beehiiv-powered sites into a structured post dataset. It accepts regular *.beehiiv.com publications and Beehiiv sites hosted on custom domains.

The Actor can:

• 📰 discover post URLs through a publication sitemap;
• 🎯 process a direct /p/<slug> post URL without crawling an archive;
• ✍️ extract one or multiple authors and author profile URLs;
• 📅 filter posts with inclusive publication-date boundaries;
• 🔎 match keywords against titles, subtitles, and accessible article text;
• 📄 return clean public article text and an estimated word count;
• 🧩 include article HTML when your downstream pipeline needs formatting;
• 🔗 collect unique external links cited in each post;
• 🔒 identify free and premium content without bypassing access controls;
• ⚡ process multiple publication domains in one run.

The easiest test is the prefilled https://www.therundown.ai publication with a three-post limit.

Who is Beehiiv Newsletter Scraper for?

Newsletter competitive-intelligence teams

• Monitor how often competing publishers release content.
• Compare titles, themes, authors, reading time, and cited sources.
• Schedule weekly ingestion across a stable publication watchlist.

Sponsorship and media buyers

• Build a reviewable catalog of recent posts before outreach.
• Identify recurring topics and outbound brands in public editions.
• Export newsletter data to Sheets, a CRM, or an internal scoring model.

Writers, editors, and content strategists

• Research topic coverage and headline patterns.
• Find public articles that mention products, markets, or competitors.
• Compare editorial cadence across multiple Beehiiv publications.

AI and data engineering teams

• Create a public-newsletter corpus for search or RAG.
• Preserve canonical URLs and extraction timestamps for provenance.
• Use text for embeddings and optional HTML for structure-aware parsing.

Why use this Beehiiv scraper?

• No Beehiiv API key: it works with anonymously accessible public pages.
• Custom-domain support: input the actual publication URL, not only a Beehiiv subdomain.
• Low-overhead HTTP extraction: no browser is launched for normal public posts.
• Flexible starts: publication, archive, sitemap, and post URLs share one input field.
• Full public content: receive article text, not only archive-card metadata.
• Paywall-safe behavior: gated bodies stay omitted and are clearly marked.
• Useful filters: reduce exports by publication date and topic keywords.
• Automation-ready: run through the Apify API, schedules, webhooks, Make, or Zapier.
• Standard exports: download JSON, CSV, Excel, XML, or RSS from Apify datasets.

What Beehiiv data can you extract?

Category	Fields
Publication	publicationId, publicationName, publicationDomain, publicationUrl, publicationDescription
Post identity	postId, slug, url
Editorial metadata	title, subtitle, excerpt, authors, authorUrls
Dates and access	publishedAt, updatedAt, audience, isPaywalled, isAccessibleForFree
Content	text, optional html, readingTimeMinutes, wordCount
Media and taxonomy	coverImageUrl, tags
Link intelligence	outboundLinks
Public engagement	likeCount, commentCount when the page exposes reliable values
Provenance	scrapedAt

Optional values are omitted when a publication does not expose them. The Actor does not fabricate subscriber counts, engagement figures, dates, or authors.

How much does it cost to scrape Beehiiv newsletters?

This Actor uses pay-per-event pricing. A small start fee covers run initialization, followed by one charge for each Beehiiv post saved to the dataset. Platform compute is included in the event prices.

Current launch pricing is:

Event	Free	Starter	Scale	Business
Run start	$0.005	$0.005	$0.005	$0.005
One post	$0.00038987	$0.00033902	$0.00026444	$0.00020341

For example, a Starter-plan run that saves 100 posts costs about $0.0389: $0.005 + (100 × $0.00033902). Higher Apify plans receive additional volume discounts. Always check the live Pricing tab for the current rate.

How to scrape Beehiiv newsletter posts

1. Open Beehiiv Newsletter Scraper on Apify.
2. Add a publication homepage, archive, sitemap, or direct post URL.
3. Keep Posts per publication small for your first test.
4. Optionally add a publication-date range or keywords.
5. Enable HTML only when the destination needs article markup.
6. Click Start and wait for the run to finish.
7. Open the Dataset tab to review the extracted posts.
8. Export results as JSON, CSV, Excel, XML, or RSS.
9. Create an Apify schedule if you want recurring monitoring.

A minimal publication run:

{
  "startUrls": [{ "url": "https://www.therundown.ai" }],
  "maxPostsPerPublication": 10,
  "maxItems": 10
}

A direct-post extraction:

{
  "startUrls": [
    { "url": "https://www.therundown.ai/p/nvidia-big-ai-day-at-gtc" }
  ],
  "maxItems": 1,
  "includeHtml": true
}

A topic-monitoring run:

{
  "startUrls": [
    { "url": "https://www.therundown.ai" }
  ],
  "maxPostsPerPublication": 50,
  "publishedAfter": "2026-07-01",
  "keywords": ["OpenAI", "Claude", "agent"],
  "includeOutboundLinks": true
}

Input parameters

Parameter	Type	Default	Description
startUrls	array	required	Beehiiv publication, archive, sitemap, or direct post URLs.
maxPostsPerPublication	integer	10	Maximum newest sitemap posts fetched per publication.
maxItems	integer	50	Total matching posts saved across the run.
publishedAfter	string	empty	Inclusive lower publication-date bound in ISO format.
publishedBefore	string	empty	Inclusive upper publication-date bound in ISO format.
keywords	string array	[]	Keep a post when any keyword matches title, subtitle, or public text.
includeHtml	boolean	false	Include cleaned accessible article HTML.
includeOutboundLinks	boolean	true	Extract unique external links from accessible bodies.
proxyConfiguration	object	proxy off	Optional Apify Proxy retry for protected custom domains.

publishedAfter must be before or equal to publishedBefore. Invalid dates and unsupported URL protocols fail closed with a clear error.

Output example

A free post can look like this:

{
  "publicationId": "4d03390d-2481-4299-b949-ffd8b38b4c38",
  "publicationName": "The Rundown AI",
  "publicationDomain": "www.therundown.ai",
  "publicationUrl": "https://www.therundown.ai/",
  "postId": "bfe910b3-6e6f-4899-968b-9079f10aebf3",
  "slug": "nvidia-big-ai-day-at-gtc",
  "url": "https://www.therundown.ai/p/nvidia-big-ai-day-at-gtc",
  "title": "Nvidia's big AI day at GTC",
  "subtitle": "PLUS: How to use Grok for free automated research",
  "authors": ["Zach Mink", "Joey Liu", "Rowan Cheung"],
  "publishedAt": "2026-03-17T09:00:00.000Z",
  "updatedAt": "2026-03-17T10:02:53Z",
  "audience": "free",
  "isPaywalled": false,
  "isAccessibleForFree": true,
  "readingTimeMinutes": 13,
  "wordCount": 1640,
  "text": "Public article text...",
  "coverImageUrl": "https://beehiiv-images-production.s3.amazonaws.com/...",
  "tags": [],
  "outboundLinks": ["https://example.com/source"],
  "scrapedAt": "2026-07-20T00:00:00.000Z"
}

The example is abbreviated. Actual author lists, word counts, links, and optional fields depend on the public source page.

How sitemap and direct-post discovery work

For a publication or archive input, the Actor requests the site's public sitemap.xml. It follows a bounded number of sitemap-index links, keeps canonical /p/<slug> URLs, and prioritizes entries with newer lastmod values.

For a direct post URL, sitemap discovery is skipped. The post still passes through the same access, date, keyword, output, and billing rules. This prevents direct URLs from bypassing filters or product scope.

How paywalled Beehiiv posts are handled

Beehiiv may expose a premium post's public title, subtitle, author, image, publication date, and canonical URL. The Actor may save that public metadata when it is available.

For premium or inaccessible posts:

• isPaywalled is true;
• isAccessibleForFree is false;
• text and html are omitted;
• no login, subscription, cookie theft, or paywall bypass is attempted.

This behavior is appropriate for archive monitoring while respecting access controls.

Keyword and date filtering

Keyword matching is case-insensitive. A post is retained when any supplied keyword appears in its title, subtitle, or accessible body text.

Date filters are inclusive:

• publishedAfter: "2026-07-01" keeps posts published on or after that date/time.
• publishedBefore: "2026-07-31" keeps posts published on or before that date/time.

If a date filter is active and the source exposes no parseable publication date, that post is excluded. This fail-closed behavior avoids silently returning records outside a requested window.

Tips for best results

• 🧪 Start with 3–10 posts before increasing the limit.
• 🎯 Use direct post URLs for one-off extraction and debugging.
• 📅 Use date windows for scheduled monitors to reduce repeated output.
• 🔎 Keep keyword lists focused; any keyword can match.
• 📄 Leave HTML disabled when plain text is enough to reduce dataset size.
• 🔗 Disable outbound links when you do not need citation analysis.
• 🌐 Enter the publication's real custom domain when it has one.
• 🛡️ Enable Apify Proxy only if a custom domain blocks direct requests.
• 🧾 Deduplicate downstream records by url or postId across scheduled runs.

Scheduling a Beehiiv newsletter monitor

Apify schedules can run this Actor hourly, daily, weekly, or with a custom cron expression. A common workflow is:

1. Store a stable list of competitor publication URLs in the Actor input.
2. Set publishedAfter to a recent rolling date in your orchestration layer.
3. Keep maxPostsPerPublication high enough to cover the interval.
4. Send the resulting dataset to a webhook or automation platform.
5. Deduplicate on canonical url before alerting analysts.

The Actor itself does not persist a hidden cross-run cursor. That keeps each run reproducible from its explicit input.

Beehiiv scraper integrations

Beehiiv Newsletter Scraper → Google Sheets

Export recent posts for editors or media buyers to review titles, authors, dates, and source links. Use an Apify integration or Make scenario to append only unseen URLs.

Beehiiv Newsletter Scraper → Slack or Discord

Schedule a keyword-filtered run and post a webhook message when competitors mention a product, company, or market. Include the canonical URL and subtitle in the alert.

Beehiiv Newsletter Scraper → Make or Zapier

Trigger downstream enrichment after each run. For example, route outbound domains to a sponsorship-prospect table or send public text to a classifier.

Beehiiv Newsletter Scraper → vector database

Chunk the text field, keep publicationName, publishedAt, and url as metadata, then write embeddings to Pinecone, Weaviate, Qdrant, or another vector store.

Beehiiv Newsletter Scraper → data warehouse

Load JSON or CSV into BigQuery, Snowflake, or PostgreSQL. Use postId where present and canonical url as a fallback key.

Using the Apify API with JavaScript

Install the client:

$npm install apify-client

Run the Actor:

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });
const run = await client.actor('automation-lab/beehiiv-newsletter-scraper').call({
  startUrls: [{ url: 'https://www.therundown.ai' }],
  maxPostsPerPublication: 10,
  maxItems: 10,
  includeOutboundLinks: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
console.log(items);

Using the Apify API with Python

Install the client:

$pip install apify-client

Run the Actor:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ['APIFY_TOKEN'])
run = client.actor('automation-lab/beehiiv-newsletter-scraper').call(run_input={
    'startUrls': [{'url': 'https://www.therundown.ai'}],
    'maxPostsPerPublication': 10,
    'maxItems': 10,
})

items = client.dataset(run['defaultDatasetId']).list_items().items
print(items)

Using the Apify API with cURL

Start a synchronous run and return dataset items:

curl -X POST \
  "https://api.apify.com/v2/acts/automation-lab~beehiiv-newsletter-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "startUrls": [{"url": "https://www.therundown.ai"}],
    "maxPostsPerPublication": 10,
    "maxItems": 10
  }'

For production integrations, send the token in an authorization header or secret manager instead of committing it to source control.

Use Beehiiv Newsletter Scraper with AI agents via MCP

Beehiiv Newsletter Scraper can be called by AI assistants through Apify's hosted Model Context Protocol integration.

For Claude Code:

$claude mcp add --transport http apify "https://mcp.apify.com?tools=automation-lab/beehiiv-newsletter-scraper"

For Claude Desktop, Cursor, or VS Code, add this MCP server configuration:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=automation-lab/beehiiv-newsletter-scraper"
    }
  }
}

Your client will ask you to authenticate with Apify.

Example prompts

After setup, try asking:

• “Use automation-lab/beehiiv-newsletter-scraper to extract the latest 10 public posts from The Rundown AI.”
• “Scrape this Beehiiv post and return its authors, publication date, full public text, and outbound sources.”
• “Monitor these Beehiiv publications for posts mentioning Claude or AI agents since July 1.”

Is it legal to scrape Beehiiv newsletters?

Scraping publicly available website data is generally permitted in many jurisdictions, but the rules depend on your location, purpose, and the data involved. You are responsible for complying with applicable laws, Beehiiv and publication terms, copyright, privacy rights, and contractual restrictions.

Use the Actor ethically:

• collect only data you have a lawful reason to process;
• do not attempt to bypass authentication or payment controls;
• avoid redistributing copyrighted article bodies without permission;
• respect deletion, access, and objection rights when personal data is involved;
• use conservative limits and avoid disrupting publication websites.

This documentation is not legal advice. Consult a qualified professional for your specific use case.

Limitations

• The Actor covers Beehiiv-powered sites that expose public sitemaps and post pages.
• A publisher can disable indexing, remove old posts, or change its custom-domain behavior.
• Some custom domains may block direct HTTP and require an Apify Proxy configuration.
• Posts omitted from every public sitemap cannot be discovered from a homepage input; use a direct URL when known.
• Premium content is not extracted beyond public metadata and preview fields.
• Engagement fields are omitted unless the page exposes values that can be interpreted reliably.
• Date filtering requires a parseable public publication date.
• HTML reflects public rendered content and may include publication-specific layout elements.

Troubleshooting

Why did the run say no /p/ post URLs were found?

Confirm that the input is a Beehiiv-powered publication and that its public sitemap is reachable at /sitemap.xml. If you know a post URL, submit that direct /p/<slug> URL instead.

Why was a custom domain blocked?

Some publishers add their own CDN or bot protection. Configure Apify Proxy in the Advanced input section and retry a small run. The Actor retries blocked direct requests once through the configured proxy.

Why are text and html missing?

The post may be premium, inaccessible, or rendered without the expected public content container. Check isPaywalled and isAccessibleForFree first. HTML is also omitted unless includeHtml is enabled.

Why did keyword filtering return no rows?

Keywords use any-match logic but only inspect the public title, subtitle, and accessible text. Check spelling, remove overly narrow phrases, and test without keywords to inspect the available public content.

FAQ

Does this Actor need a Beehiiv account or API key?

No. It reads anonymously accessible publication sitemaps and public post pages.

Can it scrape Beehiiv custom domains?

Yes, when the custom domain exposes standard Beehiiv public surfaces. Provide the complete domain URL in startUrls.

Can it scrape every post in a publication?

It can process posts exposed in the public sitemap up to your configured limits. Set a proportionate limit and check the sitemap if an expected URL is absent.

How fast is the Beehiiv scraper?

Normal runs use direct HTTP and avoid browser startup. Duration depends on publication response times, number of posts, retries, and dataset size.

How does it compare with the Beehiiv API?

Beehiiv's authenticated API serves publication owners and requires credentials. This Actor is for public web posts and works without the publication owner's API key.

Does it bypass newsletter paywalls?

No. Premium posts are marked as paywalled, and protected body text and HTML are omitted.

Can I extract only one post?

Yes. Use the direct /p/<slug> URL and set maxItems to 1.

Can I monitor several publications in one run?

Yes. Add multiple publication URLs and set per-publication and total limits.

Related newsletter and content scrapers

Continue a newsletter or creator-intelligence workflow with these Automation Labs actors:

• Substack Scraper — extract Substack publications and posts.
• Substack Leaderboard Scraper — discover ranked publications by category.
• Substack Comments Scraper — collect public discussion around Substack posts.
• Substack People Scraper — research authors and creator profiles.
• Substack Notes Search & Author Monitor — track shorter-form creator posts.
• Medium Scraper — extract public Medium articles.
• Reddit Scraper — add community discussion to topic research.
• YouTube Transcript — turn public video speech into analysis-ready text.

Support

If a valid public Beehiiv page fails, open an issue from the Actor page and include:

• the source URL;
• a small reproducible input;
• the failed run ID;
• whether proxy use was enabled;
• the expected public field or post.

Do not share private credentials, subscriber cookies, or paid content in an issue.