# Blog & Newsletter Scraper: Substack, WordPress, Ghost (`luochengfengqi/blog-newsletter-scraper`) Actor

Extracts articles from Substack, WordPress, Ghost, Blogger, dev.to, Medium and RSS, Atom or JSON feeds through their native JSON APIs. Returns Markdown bodies, authors, tags, dates, paywall status and publication cadence statistics without a browser.

- **URL**: https://apify.com/luochengfengqi/blog-newsletter-scraper.md
- **Developed by:** [luochengfengqi](https://apify.com/luochengfengqi) (community)
- **Stats:** 2 total users, 1 monthly users, 100.0% runs succeeded, 0 bookmarks
- **User rating**: No ratings yet

## Pricing

Pay per usage

This Actor is paid per platform usage. The Actor is free to use, and you only pay for the Apify platform usage, which gets cheaper the higher subscription plan you have.

Learn more: https://docs.apify.com/actors/running/actors-in-store.md#pay-per-usage

## What's an Apify Actor?

An Actor is a serverless cloud program that runs on the Apify platform. It has two run modes.
In Batch mode, an Actor accepts a well-defined JSON input, performs an action which can take anything from a few seconds to a few hours,
and optionally produces a well-defined JSON output, datasets with results, or files in key-value store.
In Standby mode, an Actor provides a web server which can be used as a website, API, or an MCP server.

Apify vocabulary and the platform model are defined once, in the agent quickstart at https://apify.com/agents.md.

## How to integrate an Actor?

If asked about integration, you help developers integrate Actors into their projects.
You adapt to their stack and deliver integrations that are safe, well-documented, and production-ready.

Do not guess an integration path. Every one of them is in the agent quickstart at https://apify.com/agents.md: the Apify MCP server, Agent Skills with the Apify CLI, the JavaScript and Python clients, the REST API, and the account-free path for an agent with no human to sign in. It also carries the rule on stating cost before the first paid run.

For examples already wired to this Actor's own input schema, see the [API](#api) section below.

Each client library has reference documentation the quickstart does not restate: [JavaScript/TypeScript](https://docs.apify.com/api/client/js/docs.md) (`npm install apify-client`) and [Python](https://docs.apify.com/api/client/python/docs.md) (`pip install apify-client`).

# README

## Blog & Newsletter Scraper

Extracts articles from **Substack**, **WordPress**, **Ghost**, **Blogger**, **dev.to**, **Medium** and any **RSS, Atom or JSON Feed** into one normalized JSON schema: full text as Markdown, plain text or cleaned HTML, authors, tags, publication and update dates, paywall status, outbound links, images, and per-publication cadence statistics.

The Actor does not render pages in a browser and needs no proxy. It calls each platform's own public JSON endpoints (Substack archive API, WordPress REST API, Ghost Content API, dev.to articles API, Blogger JSON feed) and parses feeds directly, so a run over the six default publications completes in roughly ten seconds. Platform detection is automatic: pass a domain or URL and the Actor probes the site to find out which publishing system it runs on.

***

### Table of Contents

1. [Supported Platforms and Endpoints](#supported-platforms-and-endpoints)
2. [How to Format Sources](#how-to-format-sources)
3. [Input Parameters](#input-parameters)
4. [Filtering and Execution Order](#filtering-and-execution-order)
5. [Output Schema](#output-schema)
6. [Publication Statistics (`SUMMARY`)](#publication-statistics-summary)
7. [Error and Empty-Result Handling](#error-and-empty-result-handling)
8. [Paywalled Articles](#paywalled-articles)
9. [Pricing](#pricing)
10. [Usage Examples (Console, cURL, Python, Node.js)](#usage-examples)
11. [Limitations and Notes](#limitations-and-notes)

***

### Supported Platforms and Endpoints

| Platform | Underlying Endpoints | Body Source |
| :--- | :--- | :--- |
| **Substack** | `GET /api/v1/archive?sort=new&offset=&limit=25[&search=]`, `GET /api/v1/posts/{slug}`, `GET /feed` (publication metadata), `GET substack.com/api/v1/user/{handle}/public_profile` (for `@handle` input) | `body_html` of the post endpoint |
| **WordPress** (self-hosted and WordPress.com) | `GET /wp-json/wp/v2/posts?_fields=...&per_page=100&page=N` with batched `users`, `categories`, `tags` and `media` lookups by `include=`; `public-api.wordpress.com/wp/v2/sites/{host}/posts` for WordPress.com; `GET /feed/?paged=N` when the REST API is blocked | `content.rendered` |
| **Ghost** | `GET /ghost/api/content/posts/?include=tags,authors&filter=published_at:>'...'` with the public Content API key read from the page source, `GET /ghost/api/content/settings/`, `GET /rss/` fallback | `html` |
| **dev.to** | `GET /api/articles?username=\|tag=&per_page=100&page=N`, `GET /api/articles/{id}` | `body_html` of the article endpoint |
| **Blogger** | `GET /feeds/posts/default?alt=json&max-results=150&start-index=N` with `published-min`, `published-max` and `q` | `content` of the feed entry |
| **Medium** | `GET medium.com/feed/@user`, `/feed/{publication}`, `/feed/tag/{tag}` | `content:encoded` of the feed item |
| **RSS, Atom, JSON Feed** | Any feed URL, including paginated feeds (`rel="next"`, `?paged=N`, `/N/`) | Feed content; summaries are completed from the article page |
| **Single article page** | `page:<url>` | JSON-LD `articleBody` or the main content block of the page |

***

### How to Format Sources

The `sources` array accepts four formats, which can be mixed in the same run.

#### 1. Site URL or Domain (Auto-Detection)

`https://www.lennysnewsletter.com`, `blog.mozilla.org/en`, `https://www.troyhunt.com`. The Actor fetches the home page and checks, in order: Substack markers, Ghost markers, WordPress REST discovery, Blogger markers, `<link rel="alternate">` feeds, and finally common feed paths such as `/feed`, `/rss.xml` and `/atom.xml`. Custom domains work the same way as platform subdomains.

#### 2. Explicit `platform:identifier` Pair

Skips detection and goes straight to the right endpoint.

- `substack:lenny` (publication subdomain) or `substack:@noahpinion` (author handle, resolved to the publication host)
- `devto:ben` (user), `devto:#webdev` (tag), or a `dev.to/...` URL
- `medium:@username`, `medium:netflix-techblog` (publication), `medium:tag/javascript`
- `blogger:android-developers.googleblog.com`
- `wordpress:example.com`, `ghost:example.com`

#### 3. Feed URL

`https://simonwillison.net/atom/everything/` or `feed:https://example.com/index.xml`. RSS 2.0, RDF, Atom and JSON Feed are supported.

#### 4. Single Article

`page:https://example.com/2026/my-post/` returns exactly one record for that page.

A bare word without a dot, such as `ben`, is treated as a Substack publication name. Use `devto:ben` to address a dev.to user.

***

### Input Parameters

| Parameter | Type | Default | Description |
| :--- | :--- | :--- | :--- |
| `sources` | `string[]` | Six example publications | Blogs, newsletters or feeds in any of the formats above. |
| `maxArticlesPerSource` | `integer` | `3` | Articles written to the dataset per source, newest first (1 to 5000). Date and keyword filters are applied before this limit. |
| `bodyFormat` | `string` | `markdown` | `markdown`, `text`, `html` (cleaned, absolute URLs) or `none` (metadata only, body requests are skipped). |
| `publishedAfter` | `string` | empty | Keep articles published on or after this date. Absolute (`2026-01-31`) or relative (`30 days`, `6 months`). |
| `publishedBefore` | `string` | empty | Keep articles published before this date. Same formats as above. |
| `keywords` | `string[]` | `[]` | Case-insensitive keywords matched against title, excerpt, tags and body. An article is kept if it matches any keyword. |
| `skipPaywalled` | `boolean` | `false` | Drop articles that require a paid or member subscription instead of returning their public preview. |
| `fullTextFromPage` | `boolean` | `true` | For feeds that only carry summaries, open the article page and extract the full text. |
| `maxConcurrency` | `integer` | `4` | Article body requests in flight per source (1 to 10). Lower it for sources that answer with HTTP 429. |

Up to three sources are processed in parallel. The defaults are chosen so that a run without any input succeeds and returns data, which is what the Apify Store daily health check requires.

#### Example Input JSON

```json
{
  "sources": [
    "substack:@noahpinion",
    "https://blog.mozilla.org/en",
    "devto:#webdev",
    "https://simonwillison.net/atom/everything/"
  ],
  "maxArticlesPerSource": 25,
  "bodyFormat": "markdown",
  "publishedAfter": "30 days",
  "keywords": ["privacy", "open source"],
  "skipPaywalled": true,
  "maxConcurrency": 4
}
```

***

### Filtering and Execution Order

1. **Resolve.** Each source string is parsed and the platform is detected with a small number of lightweight requests.
2. **List.** The archive is paged newest-first. Filters that the platform supports natively are sent to the server: `search` on Substack, `after`, `before` and `search` on WordPress, an NQL `published_at` filter on Ghost, `published-min`, `published-max` and `q` on Blogger.
3. **Date window.** `publishedAfter` and `publishedBefore` are compared against the platform's UTC publication timestamp. Listing stops once the archive is older than `publishedAfter`.
4. **Keywords.** Matched on the server where the platform offers search, otherwise locally on title, excerpt, tags and body. Multiple keywords use OR logic.
5. **Paywall filter.** `skipPaywalled` is applied while listing on Substack and Ghost, so you still receive up to `maxArticlesPerSource` free articles. Other platforms do not expose paywall status; their articles are always kept.
6. **Cap.** At most `maxArticlesPerSource` articles are kept per source.
7. **Bodies.** Article bodies are fetched with up to `maxConcurrency` requests in flight. Feed summaries are completed from the article page when `fullTextFromPage` is enabled.
8. **Normalize.** HTML is converted to the requested format, emoji are removed from all text fields, word counts, outbound links and images are collected.
9. **Store.** Records are written to the dataset and one `SUMMARY` record per run is written to the key-value store.

***

### Output Schema

Each article is one dataset record. Every field is always present; fields that a platform does not provide are `null` (or an empty array).

#### Sample Dataset Record

The `excerpt`, `body`, `outboundLinks` and `images` values below are shortened for readability. All other values are unmodified output.

```json
{
  "status": "OK",
  "source": "https://www.troyhunt.com",
  "platform": "ghost",
  "publication": "Troy Hunt",
  "publicationUrl": "https://www.troyhunt.com/",
  "articleId": "6ad5ba5a878c8c00019eb8c5",
  "url": "https://www.troyhunt.com/weekly-update-526/",
  "title": "Weekly Update 526: Building HIBP AI/SI First",
  "excerpt": "I have this sense that we've passed the point where \"vibe coding\" is a derogatory term. ...",
  "publishedAt": "2026-10-19T06:47:58.000Z",
  "updatedAt": "2026-10-19T06:47:58.000Z",
  "authors": ["Troy Hunt"],
  "tags": ["Weekly update"],
  "categories": [],
  "isPaywalled": false,
  "coverImage": "https://storage.ghost.io/c/fb/33/fb3391dc-723d-4e74-b95a-d641b5feb38e/content/images/2026/10/image.jpg",
  "wordCount": 132,
  "fullWordCount": null,
  "readingTimeMinutes": 1,
  "reactionCount": null,
  "commentCount": null,
  "bodyStatus": "FULL",
  "body": "I have this sense that we've passed the point where \"vibe coding\" is a derogatory term. ...",
  "outboundLinks": [
    {
      "url": "https://x.com/IntCyberDigest/status/2111592082297270339?ref=troyhunt.com",
      "domain": "x.com",
      "anchorText": "October 17, 2026"
    }
  ],
  "images": [
    "https://storage.ghost.io/c/fb/33/fb3391dc-723d-4e74-b95a-d641b5feb38e/content/images/2018/05/Listen-on-Apple-Podcasts.svg"
  ],
  "scrapedAt": "2026-10-21T03:32:00.817Z"
}
```

#### Field Reference

| Field | Type | Description |
| :--- | :--- | :--- |
| `status` | `string` | `OK` for article records, `NOT_FOUND_OR_ERROR` for diagnostic records. |
| `source` | `string` | The input string that produced the record. |
| `platform` | `string` | `substack`, `wordpress`, `ghost`, `devto`, `blogger`, `medium`, `feed` or `page`. |
| `publication` | `string \| null` | Name of the blog, newsletter or profile. |
| `publicationUrl` | `string \| null` | Home URL of the publication. |
| `articleId` | `string \| null` | Platform-native identifier (post id, entry id or feed GUID). |
| `url` | `string` | Canonical article URL. |
| `title` | `string \| null` | Article title with HTML entities decoded. |
| `excerpt` | `string \| null` | Plain-text summary of at most 300 characters. Taken from the platform when provided, otherwise cut from the body at a word boundary. |
| `publishedAt` | `string \| null` | Publication time, ISO 8601 in UTC. |
| `updatedAt` | `string \| null` | Last modification time, ISO 8601 in UTC, when the platform reports it. |
| `authors` | `string[]` | Author display names. |
| `tags` | `string[]` | Tags or topics assigned by the author. |
| `categories` | `string[]` | Categories (WordPress, Blogger, feed categories). |
| `isPaywalled` | `boolean \| null` | `true` when the article requires a paid or member subscription. `null` when the platform does not expose it. |
| `coverImage` | `string \| null` | URL of the cover or social image. |
| `wordCount` | `integer \| null` | Word count of the returned body. `null` when `bodyFormat` is `none`. |
| `fullWordCount` | `integer \| null` | Word count of the complete article as reported by the platform (Substack). Compare with `wordCount` to see how much of a paywalled post was withheld. |
| `readingTimeMinutes` | `integer \| null` | Reading time at 238 words per minute, based on `fullWordCount` when known, otherwise on `wordCount`. |
| `reactionCount` | `integer \| null` | Likes or reactions (Substack, dev.to). |
| `commentCount` | `integer \| null` | Comments (Substack, dev.to, Blogger). |
| `bodyStatus` | `string` | `FULL`, `PREVIEW`, `EXCERPT`, `EMPTY` or `NOT_REQUESTED`. See below. |
| `body` | `string \| null` | Article text in the format selected by `bodyFormat`. `null` when `bodyFormat` is `none`. |
| `outboundLinks` | `object[]` | Links that leave the publication's own domain, each with `url`, `domain` and `anchorText`. Tracking parameters (`utm_*`, `fbclid`, `gclid` and similar) are removed. At most 100 entries. |
| `images` | `string[]` | Absolute URLs of images inside the body. At most 50 entries. |
| `scrapedAt` | `string` | Extraction time, ISO 8601 in UTC. |
| `message` | `string` | Diagnostic records only: what went wrong. |
| `httpStatus` | `integer \| null` | Diagnostic records only: HTTP status of the failing request when there was one. |

`bodyStatus` values:

| Value | Meaning |
| :--- | :--- |
| `FULL` | The complete article text. For paywalled Substack posts this is set only when `wordCount` reaches 95 percent of `fullWordCount`. |
| `PREVIEW` | The free portion that the platform serves to anonymous visitors. The rest of the article is not available without a subscription. |
| `EXCERPT` | Only the feed summary was available and the article page did not yield clearly more text. |
| `EMPTY` | The platform returned no body content. |
| `NOT_REQUESTED` | `bodyFormat` is `none`. |

***

### Publication Statistics (`SUMMARY`)

Every run writes a `SUMMARY` record to the default key-value store. It lists one entry per source, in input order, with cadence and content statistics computed over the extracted articles:

```json
{
  "source": "https://www.troyhunt.com",
  "platform": "ghost",
  "publication": "Troy Hunt",
  "publicationUrl": "https://www.troyhunt.com/",
  "status": "OK",
  "message": null,
  "totalArticlesOnSource": 1392,
  "articlesStored": 3,
  "articlesExtracted": 3,
  "firstPublishedAt": "2026-10-04T11:52:23.000Z",
  "lastPublishedAt": "2026-10-19T06:47:58.000Z",
  "daysSinceLastPublished": 1.9,
  "spanDays": 14.79,
  "medianGapDays": 7.39,
  "meanGapDays": 7.39,
  "articlesPerMonth": 4.12,
  "articlesLast30Days": 3,
  "articlesLast90Days": 3,
  "averageWordCount": 163,
  "medianWordCount": 169,
  "paywalledShare": 0,
  "topTags": [{ "tag": "Weekly update", "count": 3 }],
  "topAuthors": [{ "author": "Troy Hunt", "count": 3 }]
}
```

The top level of `SUMMARY` also holds `startedAt`, `finishedAt`, `sourcesRequested`, `sourcesSucceeded`, `sourcesFailed`, `articlesStored` and `chargeLimitReached`.

All statistics describe the extracted window, not the complete archive. `totalArticlesOnSource` is the archive size when the platform reports it (WordPress, Ghost, Blogger) and `null` otherwise. To measure cadence over a longer period, raise `maxArticlesPerSource` or widen the date window.

***

### Error and Empty-Result Handling

One failing source never stops the run. Instead the Actor writes a diagnostic record to the dataset and continues with the remaining sources. Diagnostic records are not charged.

Source that does not exist:

```json
{
  "status": "NOT_FOUND_OR_ERROR",
  "source": "substack:zzzz-not-a-real-publication-9917",
  "platform": "substack",
  "message": "zzzz-not-a-real-publication-9917.substack.com does not expose the Substack archive API (HTTP 404)",
  "httpStatus": 404,
  "scrapedAt": "2026-10-21T03:34:44.645Z"
}
```

Source that was reached but where no article matched the filters:

```json
{
  "status": "NOT_FOUND_OR_ERROR",
  "source": "https://www.troyhunt.com",
  "platform": "ghost",
  "message": "Troy Hunt was reached but no articles matched with filters keywords=[\"zzzzqqqq\"]",
  "httpStatus": null,
  "scrapedAt": "2026-10-21T03:34:38.636Z"
}
```

To work only with articles, filter on `status`:

```
GET https://api.apify.com/v2/datasets/{datasetId}/items?clean=true&format=json
```

or select `status == "OK"` in your client. The Console dataset view shows the `status` column first.

***

### Paywalled Articles

The Actor does not log in, does not use cookies and does not bypass paywalls. For paid Substack posts and members-only Ghost posts it returns what the platform serves to anonymous visitors, which is usually an introduction of variable length. Such records have `bodyStatus: "PREVIEW"`, and on Substack `fullWordCount` holds the length of the complete article:

```json
{
  "platform": "substack",
  "title": "How to tell if a startup has a moat",
  "isPaywalled": true,
  "wordCount": 2273,
  "fullWordCount": 4153,
  "bodyStatus": "PREVIEW"
}
```

Set `skipPaywalled` to `true` to exclude such articles entirely.

***

### Pricing

Each article record stored in the dataset is one billable `article-extracted` event. The current price per event is shown on the Actor's Store page. Diagnostic records and the `SUMMARY` record are never billed. If you set a maximum charged results limit for a run, the Actor stops writing once the limit is reached and sets `chargeLimitReached` to `true` in `SUMMARY`. The Actor runs the same way when pay-per-event pricing is not active, in which case no events are charged.

***

### Usage Examples

#### 1. Running in Apify Console (Web UI)

1. Open the Actor page and select **Input**.
2. Paste domains, `platform:identifier` pairs or feed URLs into **Blogs, Newsletters or Feeds**, one per line.
3. Choose `maxArticlesPerSource`, `bodyFormat` and an optional date window or keywords.
4. Select **Start**. The status message shows progress per source.
5. Open the **Output** tab for the `articles` table and the `summary` record, or export the dataset as JSON, CSV, Excel or XML.

#### 2. cURL (Synchronous Run + Dataset JSON Output)

```bash
curl -X POST "https://api.apify.com/v2/acts/luochengfengqi~blog-newsletter-scraper/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "sources": ["substack:@noahpinion", "devto:#webdev"],
    "maxArticlesPerSource": 5,
    "bodyFormat": "markdown",
    "publishedAfter": "14 days"
  }'
```

#### 3. Python (`apify-client`)

```python
from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")

run_input = {
    "sources": ["https://blog.mozilla.org/en", "https://www.troyhunt.com"],
    "maxArticlesPerSource": 20,
    "bodyFormat": "text",
    "keywords": ["security", "privacy"],
}

run = client.actor("luochengfengqi/blog-newsletter-scraper").call(run_input=run_input)

## Articles from the default dataset (diagnostic records have status != "OK")
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["status"] == "OK":
        print(item["publishedAt"], item["title"], item["wordCount"])

## Publication cadence from the key-value store
summary = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("SUMMARY")["value"]
for source in summary["sources"]:
    print(source["publication"], source.get("articlesPerMonth"), source.get("medianGapDays"))
```

#### 4. Node.js (`apify-client`)

```js
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });

const run = await client.actor('luochengfengqi/blog-newsletter-scraper').call({
    sources: ['substack:lenny', 'devto:ben', 'https://simonwillison.net/atom/everything/'],
    maxArticlesPerSource: 10,
    bodyFormat: 'markdown',
    skipPaywalled: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const articles = items.filter((item) => item.status === 'OK');
console.log(`${articles.length} articles`);

const summary = (await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('SUMMARY')).value;
console.log(summary.sources.map((s) => `${s.publication}: ${s.articlesPerMonth ?? 'n/a'} per month`));
```

***

### Limitations and Notes

- **Public data only.** The Actor reads what a site serves to anonymous visitors through its public APIs and feeds. It does not authenticate and does not evaluate `robots.txt`. You are responsible for complying with the terms of the sites you read and with copyright and data protection law for the content you store.
- **Hashnode is not supported.** At the time of writing its public GraphQL API requires a paid plan and its RSS endpoints sit behind a bot challenge.
- **Medium exposes about 10 posts.** Medium's feeds return the most recent items only, and member-only stories carry just the preview in the feed. Raising `maxArticlesPerSource` does not change this.
- **Paywalls.** Paid and member-only posts are returned as previews, see [Paywalled Articles](#paywalled-articles). Substack and Ghost report paywall status; other platforms report `isPaywalled: null`.
- **WordPress REST API disabled.** Some sites block `/wp-json`. The Actor then reads `/feed/?paged=N`, which contains full text only if the site publishes full-text feeds. With `fullTextFromPage` enabled, summaries are completed from the article pages.
- **WordPress.com.** The hosted API sometimes refuses author, category and tag name lookups (HTTP 401 or 429). The affected lists are empty for those articles.
- **Ghost Content API key.** The key is read from the public page source. If a site removes it, the Actor falls back to the RSS feed.
- **Page extraction is heuristic.** For `page:` sources and feed summary completion, the main content is found through JSON-LD and semantic containers, with a text-density fallback. Unusual layouts can include extra elements or miss content. Check `bodyStatus` and `wordCount` when quality matters.
- **Rate limits.** Substack in particular answers with HTTP 429 when many requests arrive in a short time. The Actor honors `Retry-After` (up to 10 seconds) and retries twice before writing a diagnostic record. Lower `maxConcurrency` for large runs.
- **Detection ambiguity.** A bare word is treated as a Substack publication. Use the explicit `devto:`, `medium:` or `blogger:` prefixes for other platforms.
- **Word counts.** `wordCount` is computed from the returned body with Unicode word segmentation. It can differ by a few words from counts reported by a platform.
- **Dates.** All output timestamps are UTC. A plain date in `publishedAfter` or `publishedBefore` such as `2026-01-31` means midnight UTC, and relative values such as `30 days` are counted back from the start of the run. Use a full ISO 8601 timestamp when you need an exact boundary.
- **Text cleaning.** Emoji and pictographs are removed from all text fields, and tracking parameters are removed from outbound links. Other content is left as published.

# Actor input Schema

## `sources` (type: `array`):

One entry per source. Accepted formats: (1) a site URL or domain such as 'https://www.lennysnewsletter.com' or 'blog.mozilla.org/en' (the platform is detected automatically); (2) 'platform:identifier' pairs such as 'substack:lenny', 'substack:@noahpinion', 'devto:ben', 'devto:#webdev', 'medium:@username', 'blogger:android-developers.googleblog.com', 'wordpress:example.com', 'ghost:example.com'; (3) a feed URL, optionally prefixed with 'feed:'; (4) 'page:<url>' for a single article page.

## `maxArticlesPerSource` (type: `integer`):

Maximum number of articles written to the dataset for each source, newest first (1 to 5000). Date and keyword filters are applied before this limit.

## `bodyFormat` (type: `string`):

Format of the article body in the 'body' field. 'markdown' keeps headings, lists, tables, code blocks and links. 'text' returns plain text, 'html' returns cleaned HTML with absolute URLs, and 'none' returns metadata only and skips body requests.

## `publishedAfter` (type: `string`):

Only return articles published on or after this date. Accepts an absolute date such as '2026-01-31' or a relative value such as '30 days' or '6 months'. Leave empty for no lower bound.

## `publishedBefore` (type: `string`):

Only return articles published before this date. Accepts an absolute date such as '2026-06-30' or a relative value such as '7 days'. Leave empty for no upper bound.

## `keywords` (type: `array`):

Optional case-insensitive keywords matched against the title, excerpt, tags and body. An article is kept if it matches ANY keyword. Substack, WordPress and Blogger apply the search on the server, other platforms are filtered locally. Leave empty to keep all articles.

## `skipPaywalled` (type: `boolean`):

When enabled, articles that require a paid or member subscription are not returned. When disabled, the free preview that the platform exposes publicly is returned and marked with bodyStatus 'PREVIEW'.

## `fullTextFromPage` (type: `boolean`):

RSS and Atom feeds often carry only a summary. When enabled, the Actor opens the article page and extracts the full text from JSON-LD or the main content block, and keeps the summary if the page does not hold clearly more text.

## `maxConcurrency` (type: `integer`):

Maximum number of article body requests in flight per source (1 to 10). Lower this value for sources that answer with HTTP 429.

## Actor input object example

```json
{
  "sources": [
    "https://www.lennysnewsletter.com",
    "https://blog.mozilla.org/en",
    "https://www.troyhunt.com",
    "devto:ben",
    "https://android-developers.googleblog.com",
    "https://simonwillison.net/atom/everything/"
  ],
  "maxArticlesPerSource": 3,
  "bodyFormat": "markdown",
  "keywords": [],
  "skipPaywalled": false,
  "fullTextFromPage": true,
  "maxConcurrency": 4
}
```

# Actor output Schema

## `articles` (type: `string`):

No description

## `summary` (type: `string`):

No description

# API

You can run this Actor programmatically using our API. Below are code examples in JavaScript, Python, and CLI, as well as the OpenAPI specification and MCP server setup.

## JavaScript example

```javascript
import { ApifyClient } from 'apify-client';

// Initialize the ApifyClient with your Apify API token
// Replace the '<YOUR_API_TOKEN>' with your token
const client = new ApifyClient({
    token: '<YOUR_API_TOKEN>',
});

// Prepare Actor input
const input = {
    "sources": [
        "https://www.lennysnewsletter.com",
        "https://blog.mozilla.org/en",
        "https://www.troyhunt.com",
        "devto:ben",
        "https://android-developers.googleblog.com",
        "https://simonwillison.net/atom/everything/"
    ],
    "maxArticlesPerSource": 3
};

// Run the Actor and wait for it to finish
const run = await client.actor("luochengfengqi/blog-newsletter-scraper").call(input);

// Fetch and print Actor results from the run's dataset (if any)
console.log('Results from dataset');
console.log(`💾 Check your data here: https://console.apify.com/storage/datasets/${run.defaultDatasetId}`);
const { items } = await client.dataset(run.defaultDatasetId).listItems();
items.forEach((item) => {
    console.dir(item);
});

// 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/js/docs

```

## Python example

```python
from apify_client import ApifyClient

# Initialize the ApifyClient with your Apify API token
# Replace '<YOUR_API_TOKEN>' with your token.
client = ApifyClient("<YOUR_API_TOKEN>")

# Prepare the Actor input
run_input = {
    "sources": [
        "https://www.lennysnewsletter.com",
        "https://blog.mozilla.org/en",
        "https://www.troyhunt.com",
        "devto:ben",
        "https://android-developers.googleblog.com",
        "https://simonwillison.net/atom/everything/",
    ],
    "maxArticlesPerSource": 3,
}

# Run the Actor and wait for it to finish
run = client.actor("luochengfengqi/blog-newsletter-scraper").call(run_input=run_input)

# Fetch and print Actor results from the run's dataset (if there are any)
print(f"💾 Check your data here: https://console.apify.com/storage/datasets/{run.default_dataset_id}")
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item)

# 📚 Want to learn more 📖? Go to → https://docs.apify.com/api/client/python/docs/quick-start

```

## CLI example

```bash
echo '{
  "sources": [
    "https://www.lennysnewsletter.com",
    "https://blog.mozilla.org/en",
    "https://www.troyhunt.com",
    "devto:ben",
    "https://android-developers.googleblog.com",
    "https://simonwillison.net/atom/everything/"
  ],
  "maxArticlesPerSource": 3
}' |
apify call luochengfengqi/blog-newsletter-scraper --silent --output-dataset

```

## MCP server setup

```json
{
    "mcpServers": {
        "apify": {
            "type": "http",
            "url": "https://mcp.apify.com/?tools=fetch-actor-details,luochengfengqi/blog-newsletter-scraper"
        }
    }
}
```

The hosted server signs you in with OAuth on first connect, so no API token belongs in this config. Clients without OAuth support can send an `Authorization: Bearer <APIFY_API_TOKEN>` header instead, using a token from API & Integrations in Apify Console (https://console.apify.com/settings/integrations).

## OpenAPI specification

Download the OpenAPI definition: https://api.apify.com/v2/actors/og19cDgEFHxy8l3qX/builds/W1pFOV6F66iCQYjWb/openapi.json