# Beehiiv Newsletter Scraper (`aurenic/beehiiv-newsletter-scraper`) Actor

Scrape Beehiiv newsletter publications and post archives. Extract titles, authors, publish dates, excerpts, tags, word counts and full text. Keyless, HTTP-only, no login.

- **URL**: https://apify.com/aurenic/beehiiv-newsletter-scraper.md
- **Developed by:** [Aurenic](https://apify.com/aurenic) (community)
- **Security:** Limited permissions
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

from $1.50 / 1,000 results

This Actor is paid per event. You are not charged for the Apify platform usage, but only a fixed price for specific events.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-event

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Beehiiv Newsletter Scraper

Scrape Beehiiv newsletter publications and post archives — keyless, HTTP-only, no login.

### What does Beehiiv Newsletter Scraper do?

Beehiiv is the fastest-growing newsletter platform, hosting publications like The Rundown AI, Milk Road, and The Peak. This actor reads each publication's public sitemap and post pages, extracting titles, authors, publish dates, excerpts, tags, cover images, word counts and optional full text. It also pulls the publication profile: name, description, avatar and subscriber count. No login, no API key, no browser.

### Output fields

| Field | Type | Description |
|---|---|---|
| publication_name | string | Publication display name |
| publication_domain | string | Publication homepage URL |
| post_title | string | Post headline |
| post_subtitle | string | Post excerpt or subtitle |
| author | string | Author name |
| publish_date | string | ISO 8601 publish timestamp |
| post_url | string | Canonical post URL |
| cover_image_url | string | Cover image URL |
| tags | array | Tags extracted from JSON-LD and on-page tags |
| word_count | integer | Estimated word count |
| full_text | string | Full post body text (when `includeFullText` is on) |
| source_keyword | string | Keyword filter applied, if any |
| scraped_at | string | ISO timestamp |

### Who is it for?

- **Content marketers** monitoring competitor newsletters for topic ideas and cadence
- **Market researchers** tracking which stories a niche newsletter covers each week
- **Data analysts** building a structured corpus of newsletter posts for trend analysis
- **AI developers** collecting clean newsletter text for fine-tuning or prompting
- **PR and media relations** monitoring what journalists write about
- **CRM enrichment** filling publication contacts and editorial calendars

### Pricing

**Pay per result: $1.50 per 1,000 posts.**

Charged only for records actually written to the dataset.

### How to use it

1. Open the actor in Apify Console.
2. Paste one or more Beehiiv publication domains or subdomains into **Publication domains**.
3. Optionally cap **Max posts per publication**.
4. Optionally turn on **Include full post text**.
5. Optionally set a **Keyword filter** to only include posts whose URL slug contains the keyword.
6. Click **Start** and export as JSON, CSV or Excel.

### Output example

```json
{
  "record_type": "beehiiv_post",
  "publication_name": "The Peak",
  "publication_domain": "https://www.readthepeak.com",
  "post_title": "Canadian universities are falling behind",
  "post_subtitle": "Canada's post-secondary schools are losing their edge.",
  "author": "Lucas Arender",
  "publish_date": "2026-06-02T10:00:00.000Z",
  "post_url": "https://www.readthepeak.com/p/canadian-universities-are-falling-behind",
  "cover_image_url": "https://beehiiv-images-production.s3.amazonaws.com/...",
  "tags": ["Water Cooler", "Perspectives"],
  "word_count": 291,
  "full_text": null,
  "source_keyword": null,
  "scraped_at": "2026-10-03T12:00:00.000Z"
}
```

### Technical details

- **Stack:** Node.js 24, `apify` SDK, native `fetch`, `cheerio`.
- **Sources:** the publication's `/sitemap.xml`, `/sitemap_index.xml` or `/sitemap-posts.xml`, then each post page's JSON-LD `Article` schema.
- **Fallback discovery:** if no sitemap is found, the actor parses the homepage for `/p/` post links.
- **Custom domains and subdomains:** both `readthepeak.com` and `milkroad.beehiiv.com` work. The actor accepts bare domains, subdomains and full URLs.
- **JSON-LD extraction:** title, description, publish date, author, image, keywords.
- **Retries:** 429 and 5xx responses are retried up to three times with backoff.
- **Proxy:** defaults to the Apify datacenter pool.

### Known limits

- **Bot-protected publications are skipped.** Some high-traffic custom domains sit behind bot protection. Those publications return a warning and are skipped, as noted by competing actors.
- **Free posts only.** Paywalled posts (`isAccessibleForFree: false` in JSON-LD) are automatically skipped.
- **Sitemaps are not always enabled.** Beehiiv does not auto-generate sitemaps on free plans. For publications without a sitemap, the actor falls back to homepage link parsing, which returns fewer posts.
- **Word count is approximate.** It estimates from the rendered article text or the JSON-LD description when full text is off.
- **Subscriber counts are opt-in.** When a publication does not display its subscriber count publicly, `subscriber_count` is null.

### FAQ

**Do I need a Beehiiv API key?**
No. The actor reads public sitemaps and post pages directly.

**Do custom domains work?**
Yes. `readthepeak.com` and `milkroad.beehiiv.com` both work with the same input.

**Can I filter by keyword?**
Yes. Set `keywordFilter` and the actor only includes posts whose URL slug contains the keyword.

**Why are some posts missing?**
Paywalled posts are skipped, and publications behind bot protection return a warning. The actor extracts free, publicly accessible posts.

**How many posts can I get per publication?**
Up to 500 by default, capped by `maxPostsPerPublication`. Set it to 0 for unlimited.

**Does this work for Substack or Ghost?**
No. This actor is Beehiiv-specific. For Substack, use Newsletter Author Contacts. For Ghost, use Ghost Explore Scraper.

### Support

Open an issue on the Actor's page for bugs or feature requests.

### Changelog

#### 0.1 — 2026-10-03

- Initial build. Keyless extraction of Beehiiv newsletter publications and post archives.
- Sitemap discovery (`/sitemap.xml`, `/sitemap_index.xml`, `/sitemap-posts.xml`) with nested sitemap index support.
- Homepage link fallback when no sitemap is available.
- JSON-LD `Article` extraction: title, description, publish date, author, image, keywords.
- Publication profile extraction: name, description, avatar, subscriber count.
- Optional full-text extraction and keyword filtering.
- Paywalled posts automatically skipped.
- Retries on 429 and 5xx with backoff.
- No login, no API key, no browser.

# Actor input Schema

## `publications` (type: `array`):

Beehiiv publication domains, subdomains or full URLs. Example: readthepeak.com, milkroad.beehiiv.com, https://www.therundown.ai

## `maxPostsPerPublication` (type: `integer`):

Maximum posts to extract per publication. Set to 0 for unlimited.

## `includeFullText` (type: `boolean`):

Extract the full post body text. Adds one request per post and increases output size.

## `keywordFilter` (type: `string`):

Only include posts whose URL slug contains this keyword. Leave empty for all posts.

## `maxResultsPerPublication` (type: `integer`):

Cap records written per publication.

## `proxy` (type: `object`):

Defaults to Apify datacenter proxy. Beehiiv publication archives accept datacenter IPs.

## Actor input object example

```json
{
  "publications": [
    "readthepeak.com",
    "www.therundown.ai",
    "www.milkroad.com"
  ],
  "maxPostsPerPublication": 20,
  "includeFullText": false,
  "keywordFilter": "",
  "maxResultsPerPublication": 200,
  "proxy": {
    "useApifyProxy": true
  }
}
```

# Actor output Schema

## `dataset` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {};

// Run the Actor and wait for it to finish
const run = await client.actor("aurenic/beehiiv-newsletter-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {}

# Run the Actor and wait for it to finish
run = client.actor("aurenic/beehiiv-newsletter-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{}' |
apify call aurenic/beehiiv-newsletter-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,aurenic/beehiiv-newsletter-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/f3JravT7gqKLPLflG/builds/1rbrJ2fbB477dU4va/openapi.json