Blog & Newsletter Scraper

Extracts articles from Substack, WordPress, Ghost, Blogger, dev.to, Medium and any RSS, Atom or JSON Feed into one normalized JSON schema: full text as Markdown, plain text or cleaned HTML, authors, tags, publication and update dates, paywall status, outbound links, images, and per-publication cadence statistics.

The Actor does not render pages in a browser and needs no proxy. It calls each platform's own public JSON endpoints (Substack archive API, WordPress REST API, Ghost Content API, dev.to articles API, Blogger JSON feed) and parses feeds directly, so a run over the six default publications completes in roughly ten seconds. Platform detection is automatic: pass a domain or URL and the Actor probes the site to find out which publishing system it runs on.

---

Table of Contents

1. Supported Platforms and Endpoints
2. How to Format Sources
3. Input Parameters
4. Filtering and Execution Order
5. Output Schema
6. Publication Statistics (SUMMARY)
7. Error and Empty-Result Handling
8. Paywalled Articles
9. Pricing
10. Usage Examples (Console, cURL, Python, Node.js)
11. Limitations and Notes

---

Supported Platforms and Endpoints

Platform	Underlying Endpoints	Body Source
Substack	GET /api/v1/archive?sort=new&offset=&limit=25[&search=], GET /api/v1/posts/{slug}, GET /feed (publication metadata), GET substack.com/api/v1/user/{handle}/public_profile (for @handle input)	body_html of the post endpoint
WordPress (self-hosted and WordPress.com)	GET /wp-json/wp/v2/posts?_fields=...&per_page=100&page=N with batched users, categories, tags and media lookups by include=; public-api.wordpress.com/wp/v2/sites/{host}/posts for WordPress.com; GET /feed/?paged=N when the REST API is blocked	content.rendered
Ghost	GET /ghost/api/content/posts/?include=tags,authors&filter=published_at:>'...' with the public Content API key read from the page source, GET /ghost/api/content/settings/, GET /rss/ fallback	html
dev.to	GET /api/articles?username=|tag=&per_page=100&page=N, GET /api/articles/{id}	body_html of the article endpoint
Blogger	GET /feeds/posts/default?alt=json&max-results=150&start-index=N with published-min, published-max and q	content of the feed entry
Medium	GET medium.com/feed/@user, /feed/{publication}, /feed/tag/{tag}	content:encoded of the feed item
RSS, Atom, JSON Feed	Any feed URL, including paginated feeds (rel="next", ?paged=N, /N/)	Feed content; summaries are completed from the article page
Single article page	page:<url>	JSON-LD articleBody or the main content block of the page

---

How to Format Sources

The sources array accepts four formats, which can be mixed in the same run.

1. Site URL or Domain (Auto-Detection)

https://www.lennysnewsletter.com, blog.mozilla.org/en, https://www.troyhunt.com. The Actor fetches the home page and checks, in order: Substack markers, Ghost markers, WordPress REST discovery, Blogger markers, <link rel="alternate"> feeds, and finally common feed paths such as /feed, /rss.xml and /atom.xml. Custom domains work the same way as platform subdomains.

2. Explicit platform:identifier Pair

Skips detection and goes straight to the right endpoint.

• substack:lenny (publication subdomain) or substack:@noahpinion (author handle, resolved to the publication host)
• devto:ben (user), devto:#webdev (tag), or a dev.to/... URL
• medium:@username, medium:netflix-techblog (publication), medium:tag/javascript
• blogger:android-developers.googleblog.com
• wordpress:example.com, ghost:example.com

3. Feed URL

https://simonwillison.net/atom/everything/ or feed:https://example.com/index.xml. RSS 2.0, RDF, Atom and JSON Feed are supported.

4. Single Article

page:https://example.com/2026/my-post/ returns exactly one record for that page.

A bare word without a dot, such as ben, is treated as a Substack publication name. Use devto:ben to address a dev.to user.

---

Input Parameters

Parameter	Type	Default	Description
sources	string[]	Six example publications	Blogs, newsletters or feeds in any of the formats above.
maxArticlesPerSource	integer	3	Articles written to the dataset per source, newest first (1 to 5000). Date and keyword filters are applied before this limit.
bodyFormat	string	markdown	markdown, text, html (cleaned, absolute URLs) or none (metadata only, body requests are skipped).
publishedAfter	string	empty	Keep articles published on or after this date. Absolute (2026-01-31) or relative (30 days, 6 months).
publishedBefore	string	empty	Keep articles published before this date. Same formats as above.
keywords	string[]	[]	Case-insensitive keywords matched against title, excerpt, tags and body. An article is kept if it matches any keyword.
skipPaywalled	boolean	false	Drop articles that require a paid or member subscription instead of returning their public preview.
fullTextFromPage	boolean	true	For feeds that only carry summaries, open the article page and extract the full text.
maxConcurrency	integer	4	Article body requests in flight per source (1 to 10). Lower it for sources that answer with HTTP 429.

Up to three sources are processed in parallel. The defaults are chosen so that a run without any input succeeds and returns data, which is what the Apify Store daily health check requires.

Example Input JSON

{
  "sources": [
    "substack:@noahpinion",
    "https://blog.mozilla.org/en",
    "devto:#webdev",
    "https://simonwillison.net/atom/everything/"
  ],
  "maxArticlesPerSource": 25,
  "bodyFormat": "markdown",
  "publishedAfter": "30 days",
  "keywords": ["privacy", "open source"],
  "skipPaywalled": true,
  "maxConcurrency": 4
}

---

Filtering and Execution Order

1. Resolve. Each source string is parsed and the platform is detected with a small number of lightweight requests.
2. List. The archive is paged newest-first. Filters that the platform supports natively are sent to the server: search on Substack, after, before and search on WordPress, an NQL published_at filter on Ghost, published-min, published-max and q on Blogger.
3. Date window. publishedAfter and publishedBefore are compared against the platform's UTC publication timestamp. Listing stops once the archive is older than publishedAfter.
4. Keywords. Matched on the server where the platform offers search, otherwise locally on title, excerpt, tags and body. Multiple keywords use OR logic.
5. Paywall filter. skipPaywalled is applied while listing on Substack and Ghost, so you still receive up to maxArticlesPerSource free articles. Other platforms do not expose paywall status; their articles are always kept.
6. Cap. At most maxArticlesPerSource articles are kept per source.
7. Bodies. Article bodies are fetched with up to maxConcurrency requests in flight. Feed summaries are completed from the article page when fullTextFromPage is enabled.
8. Normalize. HTML is converted to the requested format, emoji are removed from all text fields, word counts, outbound links and images are collected.
9. Store. Records are written to the dataset and one SUMMARY record per run is written to the key-value store.

---

Output Schema

Each article is one dataset record. Every field is always present; fields that a platform does not provide are null (or an empty array).

Sample Dataset Record

The excerpt, body, outboundLinks and images values below are shortened for readability. All other values are unmodified output.

{
  "status": "OK",
  "source": "https://www.troyhunt.com",
  "platform": "ghost",
  "publication": "Troy Hunt",
  "publicationUrl": "https://www.troyhunt.com/",
  "articleId": "6ad5ba5a878c8c00019eb8c5",
  "url": "https://www.troyhunt.com/weekly-update-526/",
  "title": "Weekly Update 526: Building HIBP AI/SI First",
  "excerpt": "I have this sense that we've passed the point where \"vibe coding\" is a derogatory term. ...",
  "publishedAt": "2026-10-19T06:47:58.000Z",
  "updatedAt": "2026-10-19T06:47:58.000Z",
  "authors": ["Troy Hunt"],
  "tags": ["Weekly update"],
  "categories": [],
  "isPaywalled": false,
  "coverImage": "https://storage.ghost.io/c/fb/33/fb3391dc-723d-4e74-b95a-d641b5feb38e/content/images/2026/10/image.jpg",
  "wordCount": 132,
  "fullWordCount": null,
  "readingTimeMinutes": 1,
  "reactionCount": null,
  "commentCount": null,
  "bodyStatus": "FULL",
  "body": "I have this sense that we've passed the point where \"vibe coding\" is a derogatory term. ...",
  "outboundLinks": [
    {
      "url": "https://x.com/IntCyberDigest/status/2111592082297270339?ref=troyhunt.com",
      "domain": "x.com",
      "anchorText": "October 17, 2026"
    }
  ],
  "images": [
    "https://storage.ghost.io/c/fb/33/fb3391dc-723d-4e74-b95a-d641b5feb38e/content/images/2018/05/Listen-on-Apple-Podcasts.svg"
  ],
  "scrapedAt": "2026-10-21T03:32:00.817Z"
}

Field Reference

Field	Type	Description
status	string	OK for article records, NOT_FOUND_OR_ERROR for diagnostic records.
source	string	The input string that produced the record.
platform	string	substack, wordpress, ghost, devto, blogger, medium, feed or page.
publication	string | null	Name of the blog, newsletter or profile.
publicationUrl	string | null	Home URL of the publication.
articleId	string | null	Platform-native identifier (post id, entry id or feed GUID).
url	string	Canonical article URL.
title	string | null	Article title with HTML entities decoded.
excerpt	string | null	Plain-text summary of at most 300 characters. Taken from the platform when provided, otherwise cut from the body at a word boundary.
publishedAt	string | null	Publication time, ISO 8601 in UTC.
updatedAt	string | null	Last modification time, ISO 8601 in UTC, when the platform reports it.
authors	string[]	Author display names.
tags	string[]	Tags or topics assigned by the author.
categories	string[]	Categories (WordPress, Blogger, feed categories).
isPaywalled	boolean | null	true when the article requires a paid or member subscription. null when the platform does not expose it.
coverImage	string | null	URL of the cover or social image.
wordCount	integer | null	Word count of the returned body. null when bodyFormat is none.
fullWordCount	integer | null	Word count of the complete article as reported by the platform (Substack). Compare with wordCount to see how much of a paywalled post was withheld.
readingTimeMinutes	integer | null	Reading time at 238 words per minute, based on fullWordCount when known, otherwise on wordCount.
reactionCount	integer | null	Likes or reactions (Substack, dev.to).
commentCount	integer | null	Comments (Substack, dev.to, Blogger).
bodyStatus	string	FULL, PREVIEW, EXCERPT, EMPTY or NOT_REQUESTED. See below.
body	string | null	Article text in the format selected by bodyFormat. null when bodyFormat is none.
outboundLinks	object[]	Links that leave the publication's own domain, each with url, domain and anchorText. Tracking parameters (utm_*, fbclid, gclid and similar) are removed. At most 100 entries.
images	string[]	Absolute URLs of images inside the body. At most 50 entries.
scrapedAt	string	Extraction time, ISO 8601 in UTC.
message	string	Diagnostic records only: what went wrong.
httpStatus	integer | null	Diagnostic records only: HTTP status of the failing request when there was one.

bodyStatus values:

Value	Meaning
FULL	The complete article text. For paywalled Substack posts this is set only when wordCount reaches 95 percent of fullWordCount.
PREVIEW	The free portion that the platform serves to anonymous visitors. The rest of the article is not available without a subscription.
EXCERPT	Only the feed summary was available and the article page did not yield clearly more text.
EMPTY	The platform returned no body content.
NOT_REQUESTED	bodyFormat is none.

---

Publication Statistics (SUMMARY)

Every run writes a SUMMARY record to the default key-value store. It lists one entry per source, in input order, with cadence and content statistics computed over the extracted articles:

{
  "source": "https://www.troyhunt.com",
  "platform": "ghost",
  "publication": "Troy Hunt",
  "publicationUrl": "https://www.troyhunt.com/",
  "status": "OK",
  "message": null,
  "totalArticlesOnSource": 1392,
  "articlesStored": 3,
  "articlesExtracted": 3,
  "firstPublishedAt": "2026-10-04T11:52:23.000Z",
  "lastPublishedAt": "2026-10-19T06:47:58.000Z",
  "daysSinceLastPublished": 1.9,
  "spanDays": 14.79,
  "medianGapDays": 7.39,
  "meanGapDays": 7.39,
  "articlesPerMonth": 4.12,
  "articlesLast30Days": 3,
  "articlesLast90Days": 3,
  "averageWordCount": 163,
  "medianWordCount": 169,
  "paywalledShare": 0,
  "topTags": [{ "tag": "Weekly update", "count": 3 }],
  "topAuthors": [{ "author": "Troy Hunt", "count": 3 }]
}

The top level of SUMMARY also holds startedAt, finishedAt, sourcesRequested, sourcesSucceeded, sourcesFailed, articlesStored and chargeLimitReached.

All statistics describe the extracted window, not the complete archive. totalArticlesOnSource is the archive size when the platform reports it (WordPress, Ghost, Blogger) and null otherwise. To measure cadence over a longer period, raise maxArticlesPerSource or widen the date window.

---

Error and Empty-Result Handling

One failing source never stops the run. Instead the Actor writes a diagnostic record to the dataset and continues with the remaining sources. Diagnostic records are not charged.

Source that does not exist:

{
  "status": "NOT_FOUND_OR_ERROR",
  "source": "substack:zzzz-not-a-real-publication-9917",
  "platform": "substack",
  "message": "zzzz-not-a-real-publication-9917.substack.com does not expose the Substack archive API (HTTP 404)",
  "httpStatus": 404,
  "scrapedAt": "2026-10-21T03:34:44.645Z"
}

Source that was reached but where no article matched the filters:

{
  "status": "NOT_FOUND_OR_ERROR",
  "source": "https://www.troyhunt.com",
  "platform": "ghost",
  "message": "Troy Hunt was reached but no articles matched with filters keywords=[\"zzzzqqqq\"]",
  "httpStatus": null,
  "scrapedAt": "2026-10-21T03:34:38.636Z"
}

To work only with articles, filter on status:

GET https://api.apify.com/v2/datasets/{datasetId}/items?clean=true&format=json

or select status == "OK" in your client. The Console dataset view shows the status column first.

---

Paywalled Articles

The Actor does not log in, does not use cookies and does not bypass paywalls. For paid Substack posts and members-only Ghost posts it returns what the platform serves to anonymous visitors, which is usually an introduction of variable length. Such records have bodyStatus: "PREVIEW", and on Substack fullWordCount holds the length of the complete article:

{
  "platform": "substack",
  "title": "How to tell if a startup has a moat",
  "isPaywalled": true,
  "wordCount": 2273,
  "fullWordCount": 4153,
  "bodyStatus": "PREVIEW"
}

Set skipPaywalled to true to exclude such articles entirely.

---

Pricing

Each article record stored in the dataset is one billable article-extracted event. The current price per event is shown on the Actor's Store page. Diagnostic records and the SUMMARY record are never billed. If you set a maximum charged results limit for a run, the Actor stops writing once the limit is reached and sets chargeLimitReached to true in SUMMARY. The Actor runs the same way when pay-per-event pricing is not active, in which case no events are charged.

---

Usage Examples

1. Running in Apify Console (Web UI)

1. Open the Actor page and select Input.
2. Paste domains, platform:identifier pairs or feed URLs into Blogs, Newsletters or Feeds, one per line.
3. Choose maxArticlesPerSource, bodyFormat and an optional date window or keywords.
4. Select Start. The status message shows progress per source.
5. Open the Output tab for the articles table and the summary record, or export the dataset as JSON, CSV, Excel or XML.

2. cURL (Synchronous Run + Dataset JSON Output)

curl -X POST "https://api.apify.com/v2/acts/luochengfengqi~blog-newsletter-scraper/run-sync-get-dataset-items?token=<YOUR_API_TOKEN>" \
  -H "Content-Type: application/json" \
  -d '{
    "sources": ["substack:@noahpinion", "devto:#webdev"],
    "maxArticlesPerSource": 5,
    "bodyFormat": "markdown",
    "publishedAfter": "14 days"
  }'

3. Python (apify-client)

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")

run_input = {
    "sources": ["https://blog.mozilla.org/en", "https://www.troyhunt.com"],
    "maxArticlesPerSource": 20,
    "bodyFormat": "text",
    "keywords": ["security", "privacy"],
}

run = client.actor("luochengfengqi/blog-newsletter-scraper").call(run_input=run_input)

# Articles from the default dataset (diagnostic records have status != "OK")
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["status"] == "OK":
        print(item["publishedAt"], item["title"], item["wordCount"])

# Publication cadence from the key-value store
summary = client.key_value_store(run["defaultKeyValueStoreId"]).get_record("SUMMARY")["value"]
for source in summary["sources"]:
    print(source["publication"], source.get("articlesPerMonth"), source.get("medianGapDays"))

4. Node.js (apify-client)

import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: '<YOUR_API_TOKEN>' });

const run = await client.actor('luochengfengqi/blog-newsletter-scraper').call({
    sources: ['substack:lenny', 'devto:ben', 'https://simonwillison.net/atom/everything/'],
    maxArticlesPerSource: 10,
    bodyFormat: 'markdown',
    skipPaywalled: true,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
const articles = items.filter((item) => item.status === 'OK');
console.log(`${articles.length} articles`);

const summary = (await client.keyValueStore(run.defaultKeyValueStoreId).getRecord('SUMMARY')).value;
console.log(summary.sources.map((s) => `${s.publication}: ${s.articlesPerMonth ?? 'n/a'} per month`));

---

Limitations and Notes

• Public data only. The Actor reads what a site serves to anonymous visitors through its public APIs and feeds. It does not authenticate and does not evaluate robots.txt. You are responsible for complying with the terms of the sites you read and with copyright and data protection law for the content you store.
• Hashnode is not supported. At the time of writing its public GraphQL API requires a paid plan and its RSS endpoints sit behind a bot challenge.
• Medium exposes about 10 posts. Medium's feeds return the most recent items only, and member-only stories carry just the preview in the feed. Raising maxArticlesPerSource does not change this.
• Paywalls. Paid and member-only posts are returned as previews, see Paywalled Articles. Substack and Ghost report paywall status; other platforms report isPaywalled: null.
• WordPress REST API disabled. Some sites block /wp-json. The Actor then reads /feed/?paged=N, which contains full text only if the site publishes full-text feeds. With fullTextFromPage enabled, summaries are completed from the article pages.
• WordPress.com. The hosted API sometimes refuses author, category and tag name lookups (HTTP 401 or 429). The affected lists are empty for those articles.
• Ghost Content API key. The key is read from the public page source. If a site removes it, the Actor falls back to the RSS feed.
• Page extraction is heuristic. For page: sources and feed summary completion, the main content is found through JSON-LD and semantic containers, with a text-density fallback. Unusual layouts can include extra elements or miss content. Check bodyStatus and wordCount when quality matters.
• Rate limits. Substack in particular answers with HTTP 429 when many requests arrive in a short time. The Actor honors Retry-After (up to 10 seconds) and retries twice before writing a diagnostic record. Lower maxConcurrency for large runs.
• Detection ambiguity. A bare word is treated as a Substack publication. Use the explicit devto:, medium: or blogger: prefixes for other platforms.
• Word counts. wordCount is computed from the returned body with Unicode word segmentation. It can differ by a few words from counts reported by a platform.
• Dates. All output timestamps are UTC. A plain date in publishedAfter or publishedBefore such as 2026-01-31 means midnight UTC, and relative values such as 30 days are counted back from the start of the run. Use a full ISO 8601 timestamp when you need an exact boundary.
• Text cleaning. Emoji and pictographs are removed from all text fields, and tracking parameters are removed from outbound links. Other content is left as published.